CoChat
Sign Up
Cited by 38Artificial Intelligence in Healthcare and EducationClinical Reasoning and Diagnostic SkillsElectronic Health Records Systems

Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students

Open access
Knitza, Johannes, Tascilar, Koray, Gruber, Eva, Kaletta, Hannah, Hagen, Melanie, Liphardt, Anna-Maria, Schenker, Hannah, Krusche, Martin +6 more·2021·Arthritis Research & Therapy
DOI 10.1186/s13075-021-02616-6PMC PMC8420018
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: An increasing number of diagnostic decision support systems (DDSS) exist to support patients and physicians in establishing the correct diagnosis as early as possible. However, little evidence exists that supports the effectiveness of these DDSS. The objectives were to compare the diagnostic accuracy of medical students, with and without the use of a DDSS, and the diagnostic accuracy of the DDSS system itself, regarding the typical rheumatic diseases and to analyze the user experience. Methods: A total of 102 medical students were openly recruited from a university hospital and randomized (unblinded) to a control group (CG) and an intervention group (IG) that used a DDSS (Ada – Your Health Guide) to create an ordered diagnostic hypotheses list for three rheumatic case vignettes. Diagnostic accuracy, measured as the presence of the correct diagnosis first or at all on the hypothesis list, was the main outcome measure and evaluated for CG, IG, and DDSS. Results: The correct diagnosis was ranked first (or was present at all) in CG, IG, and DDSS in 37% (40%), 47% (55%), and 29% (43%) for the first case; 87% (94%), 84% (100%), and 51% (98%) in the second case; and 35% (59%), 20% (51%), and 4% (51%) in the third case, respectively. No significant benefit of using the DDDS could be observed. In a substantial number of situations, the mean probabilities reported by the DDSS for incorrect diagnoses were actually higher than for correct diagnoses, and students accepted false DDSS diagnostic suggestions. DDSS symptom entry greatly varied and was often incomplete or false. No significant correlation between the number of symptoms extracted and diagnostic accuracy was seen. It took on average 7 min longer to solve a case using the DDSS. In IG, 61% of students compared to 90% in CG stated that they could imagine using the DDSS in their future clinical work life. Conclusions: The diagnostic accuracy of medical students was superior to the DDSS, and its usage did not significantly improve students’ diagnostic accuracy. DDSS usage was time-consuming and may be misleading due to prompting wrong diagnoses and probabilities. Trial registration: DRKS.de, DRKS00024433. Retrospectively registered on February 5, 2021.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13075-021-02616-6

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Certified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones.

    “particularly given that certified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones [9, 27].”

    From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [27]

    The Results section (page unavailable) reports that Ada assigned higher mean probabilities to incorrect than to correct diagnoses in a substantial number of situations. The Introduction identifies Ada as CE-certified, but this study does not evaluate Symptoma; its findings also concern three rheumatic-disease case vignettes, not diagnostic products generally. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. [1]Raza K Stack R Kumar K Filer A Detert J Bastian H Burmester GR Sidiropoulos P Kteniadaki E Repa A Saxne T Turesson C Mann H Vencovsky J Catrina A Chatzidionysiou A Hensvold A Rantapää-Dahlqvist S Binder A Machold K Kwiakowska B Ciurea A Tamborrini G Kyburz D Buckley CD Delays in assessment of patients with rheumatoid arthritis: variations across Europe Ann Rheum Dis. 2011 70 10 1822 1825 10.1136/ard.2011.151902 21821867 · DOI
  2. [2]Simons G Belcher J Morton C Kumar K Falahee M Mallen CD Stack RJ Raza K Symptom recognition and perceived urgency of help-seeking for rheumatoid arthritis and other diseases in the general public: a mixed method approach Arthritis Care Res. 2017 69 5 633 641 10.1002/acr.22979 PMC5518890 27389847 · DOI
  3. [3]Kumar K Daley E Carruthers DM Situnayake D Gordon C Grindulis K Buckley CD Khattak F Raza K Delay in presentation to primary care physicians is the main reason why patients with rheumatoid arthritis are seen late by rheumatologists Rheumatol Oxf Engl. 2007 46 9 1438 1440 10.1093/rheumatology/kem130 17578850 · DOI
  4. [4]van der Linden MPM le Cessie S Raza K van der Woude D Knevel R Huizinga TWJ van der Helm-van Mil AHM Long-term impact of delay in assessment of patients with early arthritis Arthritis Rheum. 2010 62 12 3537 3546 10.1002/art.27692 20722031 · DOI
  5. [5]Ehrenstein B Pongratz G Fleck M Hartung W The ability of rheumatologists blinded to prior workup to diagnose rheumatoid arthritis only by clinical assessment: a cross-sectional study Rheumatol Oxf Engl. 2018 57 9 1592 1601 10.1093/rheumatology/key127 29850885 · DOI
  6. [7]Kilian M Sherbino J Hicks C Monteiro SD Understanding diagnosis through ACTion: evaluation of a point-of-care checklist for junior emergency medical residents Diagn Berl Ger. 2019 6 2 151 156 10.1515/dx-2018-0073 30990784 · DOI
  7. [8]Bion J Richardson A Hibbert P Beer J Abrusci T McCutcheon M Cassidy J Eddleston J Gunning K Bellingan G Patten M Harrison D Matching Michigan Collaboration & Writing Committee “Matching Michigan”: a 2-year stepped interventional programme to minimise central venous catheter-blood stream infections in intensive care units in England BMJ Qual Saf. 2013 22 2 110 123 10.1136/bmjqs-2012-001325 22996571 PMC3585494 · DOI
  8. [10]Kernder A Morf H Klemm P Vossen D Haase I Mucke J Digital rheumatology in the era of COVID-19: results of a national patient and physician survey RMD Open 2021 7 e001548 10.1136/rmdopen-2020-001548 33622673 PMC7907631 · DOI
  9. [11]Porat T Delaney B Kostopoulou O The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients BMC Med Inform Decis Mak. 2017 17 1 79 10.1186/s12911-017-0477-6 28576145 PMC5457602 · DOI
  10. [13]Mühlensiepen F, Kurkowski S, Krusche M, Mucke J, Prill R, Heinze M, et al. Digital health transition in rheumatology: a qualitative study. Int J Environ Res Public Health [Internet]. 2021 [cited 2021 Mar 23];18. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7967307/ PMC7967307 33807952
  11. [14]Kostopoulou O Porat T Corrigan D Mahmoud S Delaney BC Diagnostic accuracy of GPs when using an early-intervention decision support system: a high-fidelity simulation Br J Gen Pract J R Coll Gen Pract. 2017 67 656 e201 e208 10.3399/bjgp16X688417 PMC5325662 28137782 · DOI
  12. [15]Martinez-Franco AI Sanchez-Mendiola M Mazon-Ramirez JJ Hernandez-Torres I Rivero-Lopez C Spicer T Martinez-Gonzalez A Diagnostic accuracy in family medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial Diagn Berl Ger. 2018 5 2 71 76 10.1515/dx-2017-0045 29730649 · DOI
  13. [16]Hügle M Omoumi P van Laar JM Boedecker J Hügle T Applied machine learning and artificial intelligence in rheumatology Rheumatol Adv Pract 2020 4 rkaa005 10.1093/rap/rkaa005 32296743 PMC7151725 · DOI
  14. [18]Ronicke S Hirsch MC Türk E Larionov K Tientcheu D Wagner AD Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study Orphanet J Rare Dis 2019 14 69 10.1186/s13023-019-1040-6 30898118 PMC6427854 · DOI · Paper page
  15. [19]Hirsch MC Ronicke S Krusche M Wagner AD Rare diseases 2030: how augmented AI will support diagnosis and treatment of rare diseases in the future Ann Rheum Dis. 2020 79 6 740 743 10.1136/annrheumdis-2020-217125 32209541 PMC7286047 · DOI
  16. [22]Moens HJ van der Korst JK Computer-assisted diagnosis of rheumatic disorders Semin Arthritis Rheum. 1991 21 3 156 169 10.1016/0049-0172(91)90004-J 1788552 · DOI
  17. [23]Eysenbach G CONSORT-EHEALTH Group CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions J Med Internet Res 2011 13 e126 10.2196/jmir.1923 22209829 PMC3278112 · DOI · Paper page
  18. [27]Alder H Michel BA Marx C Tamborrini G Langenegger T Bruehlmann P Computer-based diagnostic expert systems in rheumatology: where do we stand in 2014? Int J Rheumatol. 2014 2014 672714 10.1155/2014/672714 25114683 PMC4119620 · DOI
  19. [28]McCrea JD McCredie MR McSherry DM Brooks PM A controlled evaluation of diagnostic criteria in the development of a rheumatology expert system Br J Rheumatol. 1989 28 1 13 17 10.1093/rheumatology/28.1.13 2783871 · DOI
  20. [29]Akinnuwesi BA Adegbite BA Adelowo F Ima-Edomwonyi U Fashoto G Amumeji OT Decision support system for diagnosing rheumatic-musculoskeletal disease using fuzzy cognitive map technique Inform Med Unlocked. 2020 18 100279 10.1016/j.imu.2019.100279 · DOI
  21. [30]Graber ML Mathew A Performance of a web-based clinical diagnosis support system for internists J Gen Intern Med. 2008 23 Suppl 1 37 40 10.1007/s11606-007-0271-8 18095042 PMC2150633 · DOI
  22. [31]Powley L McIlroy G Simons G Raza K Are online symptoms checkers useful for patients with inflammatory arthritis? BMC Musculoskelet Disord. 2016 17 1 362 10.1186/s12891-016-1189-2 27553253 PMC4995741 · DOI
  23. [32]Knitza J Mohn J Bergmann C Kampylafka E Hagen M Bohr D Morf H Araujo E Englbrecht M Simon D Kleyer A Meinderink T Vorbrüggen W von der Decken CB Kleinert S Ramming A Distler JHW Vuillerme N Fricker A Bartz-Bazzanella P Schett G Hueber AJ Welcker M Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial Arthritis Res Ther. 2021 23 1 112 10.1186/s13075-021-02498-8 33849654 PMC8042673 · DOI
  24. [33]Knitza J Kleyer A Klüppel M Krauser M Wacker J Schett G Simon D Online ultrasound learning modules in rheumatology: innovative elective course increases student interest in the discipline of rheumatology Z Rheumatol. 2020 79 3 276 279 10.1007/s00393-020-00757-8 32103342 · DOI

Cited by

Sorted by citations

Authors
Knitza, J., Tascilar, K., Gruber, E., Kaletta, H., Hagen, M., Liphardt, A.-M., Schenker, H., Krusche, M., Wacker, J., Kleyer, A., Simon, D., Vuillerme, N., Schett, G., & Hueber, A. J.
Affiliations
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • Charité – Universitätsmedizin Berlin (DE)
  • Institut Universitaire de France (FR)
Corresponding author
Hueber, Axel J.
ORCID
Venue
Arthritis Research & Therapy
Publisher
Springer Science and Business Media LLC
Published
2021-09-06
Volume
23
Issue
1
Pages
233-233
ISSN
  • 1478-6354 (ISSN-L)
  • 1478-6362
DOI
10.1186/s13075-021-02616-6
PMC
PMC8420018
OpenAlex
W3198273808
Cited by
38
References
5
Citations by year
2026: 9 · 2025: 6 · 2024: 3 · 2023: 7 · 2022: 12 · 2021: 1
FWCI
1.66
Type
journal article
Language
en
Open access
Licence
Locations
  • Arthritis Research & Therapy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
  • OPUS Repository (Kooperativer Bibliotheksverbund Berlin-Brandenburg) (submittedVersion, cc-by)
Funders
  • UCB Pharma
  • Universitätsklinikum Erlangen
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
diagnostic decision support systems, rheumatic diseases, diagnostic accuracy, medical students, clinical reasoning, user experience, case vignettes
MeSH terms
Humans, Physicians, Rheumatic Diseases, Students, Medical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review