Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 20

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Computerized diagnostic decision support systems–A comparative performance study of Isabel Pro vs. ChatGPT4

Bridges, Joe M.·2024·Diagnosis
DOI 10.1515/dx-2024-0033
Open this paper in CoChat — free account

Abstract

Objectives: Validate the diagnostic accuracy of the Artificial Intelligence Large Language Model ChatGPT4 by comparing diagnosis lists produced by ChatGPT4 to Isabel Pro. Methods: This study used 201 cases, comparing ChatGPT4 to Isabel Pro. Systems inputs were identical. Mean Reciprocal Rank (MRR) compares the correct diagnosis’s rank between systems. Isabel Pro ranks by the frequency with which the symptoms appear in the reference dataset. The mechanism ChatGPT4 uses to rank the diagnoses is unknown. A Wilcoxon Signed Rank Sum test failed to reject the null hypothesis. Results: Both systems produced comprehensive differential diagnosis lists. Isabel Pro’s list appears immediately upon submission, while ChatGPT4 takes several minutes. Isabel Pro produced 175 (87.1 %) correct diagnoses and ChatGPT4 165 (82.1 %). The MRR for ChatGPT4 was 0.428 (rank 2.31), and Isabel Pro was 0.389 (rank 2.57), an average rank of three for each. ChatGPT4 outperformed on Recall at Rank 1, 5, and 10, with Isabel Pro outperforming at 20, 30, and 40. The Wilcoxon Signed Rank Sum Test confirmed that the sample size was inadequate to conclude that the systems are equivalent. ChatGPT4 fabricated citations and DOIs, producing 145 correct references (87.9 %) but only 52 correct DOIs (31.5 %). Conclusions: This study validates the promise of Clinical Diagnostic Decision Support Systems, including the Large Language Model form of artificial intelligence (AI). Until the issue of hallucination of references and, perhaps diagnoses, is resolved in favor of absolute accuracy, clinicians will make cautious use of Large Language Model systems in diagnosis, if at all.

Overview

Integrity signals

Registry

Crossref · 10.1515/dx-2024-0033

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 3 references

Research works printed but not linked: 14

  1. Wachter, RM, Brynjolfsson, E. Will generative artificial intelligence deliver on its promise in health care? JAMA 2024;33:65–9. https://doi.org/10.1001/jama.2023.25054. · DOI
  2. Liu, X, Song, Y, Lin, H, Xu, Y, Chen, C, Yan, C, et al.. Evaluating chatgpt as an adjunct for analyzing challenging case. Blood 2023;142:7273–. https://doi.org/10.1182/blood-2023-181518. · DOI
  3. Horiuchi, D, Tatekawa, H, Shimono, T, Walston, SL, Takita, H, Matsushita, S, et al.. Accuracy of ChatGPT generated diagnosis from patient’s medical history and imaging findings in neuroradiology cases. Neuroradiology 2024;66:73–9. https://doi.org/10.1007/s00234-023-03252-4. · DOI
  4. Dave, T, Athaluri, SA, Singh, S. Chatgpt in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 2023;6:1169595. https://doi.org/10.3389/frai.2023.1169595. · DOI · Paper page
  5. Kanjee, Z, Crowe, B, Rodman, A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA 2023;330:78. https://doi.org/10.1001/jama.2023.8288. · DOI · Paper page
  6. Graber, ML, Mathew, A. Performance of a web-based clinical diagnosis support system for internists. J Gen Intern Med 2008;23:37–40. https://doi.org/10.1007/s11606-007-0271-8. · DOI
  7. Bond, WF, Schwartz, LM, Weaver, KR, Levick, D, Giuliano, M, Graber, ML. Differential diagnosis generators: an evaluation of currently available computer programs. J Gen Intern Med 2012;27:213–9. https://doi.org/10.1007/s11606-011-1804-8. · DOI
  8. Graber, ML. Reaching 95 %: decision support tools are the surest way to improve diagnosis now. BMJ Qual Saf 2022;31:415–8. https://doi.org/10.1136/bmjqs-2021-014033. · DOI
  9. Riches, N, Panagioti, M, Alam, R, Cheraghi-Sohi, S, Campbell, S, Esmail, A, et al.. The effectiveness of electronic differential diagnoses (DDX) generators: a systematic review and meta-analysis. PLoS One 2016;11:e0148991. https://doi.org/10.1371/journal.pone.0148991. · DOI
  10. Friedman, CP, Elstein, AS, Wolf, FM, Murphy, GC, Franz, TM, Heckerling, PS, et al.. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA 1999;282:1851. https://doi.org/10.1001/jama.282.19.1851. · DOI
  11. Sibbald, M, Monteiro, S, Sherbino, J, LoGiudice, A, Friedman, C, Norman, G. Should electronic differential diagnosis support be used early or late in the diagnostic process? A multicentre experimental study of Isabel. BMJ Qual Saf 2021. https://doi.org/10.1136/bmjqs-2021-013493. · DOI
  12. Fritz, P, Kleinhans, A, Raoufi, R, Sediqi, A, Schmid, N, Schricker, S, et al.. Evaluation of medical decision support systems (ddx generators) using real medical cases of varying complexity and origin. BMC Med Inf Decis Mak 2022;22:254. https://doi.org/10.1186/s12911-022-01988-2. · DOI
  13. Graber, ML. Reaching 95 %: decision support tools are the surest way to improve diagnosis now. BMJ Qual Saf 2022;31:415–8. https://doi.org/10.1136/bmjqs-2021-014033. · DOI
  14. Liu, J, Wang, C, Liu, S. Utility of chatgpt in clinical practice. J Med Internet Res 2023;25:e48568. https://doi.org/10.2196/48568. · DOI · Paper page

Cited by

See all 20

Metadata

Authors
Bridges, J. M.
Affiliations
D. Bradley McWilliams School of Biomedical Informatics , University of Texas Health Science Center at Houston , Houston , USA
Venue
Diagnosis
Publisher
Walter de Gruyter GmbH
Published
2024-05-07
Volume
11
Issue
3
Pages
250-258
ISSN
2194-802X
DOI
10.1515/dx-2024-0033
Cited by
20
References
3
Language
en
Licence
creativecommons.org/licenses/by/4.0 (unspecified)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Computerized diagnostic decision support systems–A comparative performance study of Isabel Pro vs. ChatGPT4. https://app.cochat.ai/papers/10.1515/dx-2024-0033
The paper
Bridges, J. M. (2024). Computerized diagnostic decision support systems–A comparative performance study of Isabel Pro vs. ChatGPT4. Diagnosis. https://doi.org/10.1515/dx-2024-0033

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.