CoChat
Sign Up
Cited by 38Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareMachine Learning in Healthcare

Evaluating ChatGPT-4’s Accuracy in Identifying Final Diagnoses Within Differential Diagnoses Compared With Those of Physicians: Experimental Study for Diagnostic Cases

Open access
Hirosawa, Takanobu, Harada, Yukinori, Mizuta, Kazuya, Sakamoto, Tetsu, Tokumasu, Kazuki, Shimizu, Taro·2024·JMIR Formative Research
DOI 10.2196/59267PMC PMC11237772

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Background: The potential of artificial intelligence (AI) chatbots, particularly ChatGPT with GPT-4 (OpenAI), in assisting with medical diagnosis is an emerging research area. However, it is not yet clear how well AI chatbots can evaluate whether the final diagnosis is included in differential diagnosis lists. Objective: This study aims to assess the capability of GPT-4 in identifying the final diagnosis from differential-diagnosis lists and to compare its performance with that of physicians for case report series. Methods: We used a database of differential-diagnosis lists from case reports in the American Journal of Case Reports, corresponding to final diagnoses. These lists were generated by 3 AI systems: GPT-4, Google Bard (currently Google Gemini), and Large Language Models by Meta AI 2 (LLaMA2). The primary outcome was focused on whether GPT-4’s evaluations identified the final diagnosis within these lists. None of these AIs received additional medical training or reinforcement. For comparison, 2 independent physicians also evaluated the lists, with any inconsistencies resolved by another physician. Results: The 3 AIs generated a total of 1176 differential diagnosis lists from 392 case descriptions. GPT-4’s evaluations concurred with those of the physicians in 966 out of 1176 lists (82.1%). The Cohen κ coefficient was 0.63 (95% CI 0.56-0.69), indicating a fair to good agreement between GPT-4 and the physicians’ evaluations. Conclusions: GPT-4 demonstrated a fair to good agreement in identifying the final diagnosis from differential-diagnosis lists, comparable to physicians for case report series. Its ability to compare differential diagnosis lists with final diagnoses suggests its potential to aid clinical decision-making support through diagnostic feedback. While GPT-4 showed a fair to good agreement for evaluation, its application in real-world scenarios and further validation in diverse clinical environments are essential to fully understand its utility in the diagnostic process.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/59267

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GenAI tools can significantly improve clinical accuracy, workflow efficiency, and decision-support capacity when implemented under structured supervision.

    “Studies27,29,30,42 provide empirical evidence that GenAI tools can significantly improve clinical accuracy, workflow efficiency, and decision-support capacity when implemented under structured supervision.”

    From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [29]

    The Results (Overall Evaluation; page not provided) report that GPT-4 agreed with physicians on 82.1% of differential-diagnosis-list evaluations. The Discussion (Conclusions; page not provided) suggests potential for reliable, efficient diagnostic feedback, but does not establish significant improvements in clinical accuracy or workflow efficiency, or evaluate implementation under structured supervision. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Clinician performance is optimized when AI outputs are analyzed rather than uncritically accepted.

    “Empirical evidence27,29,42 indicated that clinician performance is optimized when AI outputs are analyzed rather than uncritically accepted.”

    From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [29]

    The Discussion > Limitations section says AI can assist physicians but cannot replace their judgment, and that overreliance without critical review could lead to diagnostic errors. This supports analyzing rather than uncritically accepting AI outputs, but the paper does not establish through a comparison that doing so optimizes clinician performance. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. [1]Balogh EP Miller BT Ball JR Improving Diagnosis in Health Care 2015 Washington, DC National Academies Press 26803862
  2. [2]Graber M Diagnostic errors in medicine: a case of neglect Jt Comm J Qual Patient Saf 2005 31 2 106 113 10.1016/s1553-7250(05)31015-4 15791770 S1553-7250(05)31015-4 15791770 · DOI · Paper page
  3. [3]Newman-Toker DE Nassery N Schaffer AC Yu-Moe CW Clemens GD Wang Z Zhu Y Saber Tehrani AS Fanai M Hassoon A Siegal D Burden of serious harms from diagnostic error in the USA BMJ Qual Saf 2024 33 2 109 120 10.1136/bmjqs-2021-014130 37460118 bmjqs-2021-014130 PMC10792094 37460118 · DOI
  4. [5]Schiff GD Hasan O Kim S Abrams R Cosby K Lambert BL Elstein AS Hasler S Kabongo ML Krosnjar N Odwazny R Wisniewski MF McNutt RA Diagnostic error in medicine: analysis of 583 physician-reported errors Arch Intern Med 2009 169 20 1881 1887 10.1001/archinternmed.2009.333 19901140 169/20/1881 19901140 · DOI · Paper page
  5. [6]Singh H Connor DM Dhaliwal G Five strategies for clinicians to advance diagnostic excellence BMJ 2022 376 e068044 10.1136/bmj-2021-068044 35172968 35172968 · DOI
  6. [7]Meyer AND Singh H The path to diagnostic excellence includes feedback to calibrate how clinicians think JAMA 2019 321 8 737 738 10.1001/jama.2019.0113 30735239 2724792 30735239 · DOI
  7. [8]Mamede S Schmidt HG Penaforte JC Effects of reflective practice on the accuracy of medical diagnoses Med Educ 2008 42 5 468 475 10.1111/j.1365-2923.2008.03030.x 18412886 MED3030 18412886 · DOI · Paper page
  8. [9]Mamede S Schmidt HG Reflection in medical diagnosis: a literature review Health Prof Educ 2017 3 1 15 25 10.1016/j.hpe.2017.01.003 · DOI
  9. [11]Rubins D McCoy AB Dutta S McEvoy DS Patterson L Miller A Jackson JG Zuccotti G Wright A Real-time user feedback to support clinical decision support system improvement Appl Clin Inform 2022 13 5 1024 1032 10.1055/s-0042-1757923 36288748 36288748 PMC9605820 · DOI
  10. [12]Haug CJ Drazen JM Artificial intelligence and machine learning in clinical medicine, 2023 N Engl J Med 2023 388 13 1201 1208 10.1056/NEJMra2302038 36988595 36988595 · DOI · Paper page
  11. [13]Liu J Wang C Liu S Utility of ChatGPT in clinical practice J Med Internet Res 2023 25 e48568 10.2196/48568 37379067 v25i1e48568 37379067 PMC10365580 · DOI · Paper page
  12. [14]Alowais SA Alghamdi SS Alsuhebany N Alqahtani T Alshaya AI Almohareb SN Aldairem A Alrashed M Bin Saleh K Badreldin HA Al Yami MS Al Harbi S Albekairy AM Revolutionizing healthcare: the role of artificial intelligence in clinical practice BMC Med Educ 2023 23 1 689 10.1186/s12909-023-04698-z 37740191 10.1186/s12909-023-04698-z 37740191 PMC10517477 · DOI · Paper page
  13. [18]Touvron H Martin L Stone K Albert P Almahairi A Babaei Y Llama 2: open foundation and fine-tuned chat models arXiv Preprint posted online July 18, 2023 10.48550/arXiv.2307.09288 · DOI · Paper page
  14. [19]Achiam J Adler S Agarwal S GPT-4 Technical Report arXiv Preprint posted online March 15, 2023 10.48550/arXiv.2303.08774 · DOI · Paper page
  15. [20]Sai S Gaur A Sai R Chamola V Guizani M Rodrigues JJPC Generative AI for transformative healthcare: a comprehensive study of emerging models, applications, case studies, and limitations IEEE Access 2024 12 31078 31106 10.1109/ACCESS.2024.3367715 · DOI · Paper page
  16. [21]Cascella M Montomoli J Bellini V Bignami E Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios J Med Syst 2023 47 1 33 10.1007/s10916-023-01925-4 36869927 10.1007/s10916-023-01925-4 36869927 PMC9985086 · DOI · Paper page
  17. [24]Hirosawa T Mizuta K Harada Y Shimizu T Comparative evaluation of diagnostic accuracy between Google Bard and physicians Am J Med 2023 11 136 11 1119 1123.e18 10.1016/j.amjmed.2023.08.003 37643659 S0002-9343(23)00536-3 37643659 · DOI
  18. [25]Kanjee Z Crowe B Rodman A Accuracy of a generative artificial intelligence model in a complex diagnostic challenge JAMA 2023 330 1 78 80 10.1001/jama.2023.8288 37318797 37318797 PMC10273128 · DOI · Paper page
  19. [26]Price RB Vlahcevic ZR Logical principles in differential diagnosis Ann Intern Med 1971 75 1 89 95 10.7326/0003-4819-75-1-89 4933297 4933297 · DOI
  20. [27]Fernandez Branson C Williams M Chan TM Graber ML Lane KP Grieser S Landis-Lewis Z Cooke J Upadhyay DK Mondoux S Singh H Zwaan L Friedman C Olson APJ Improving diagnostic performance through feedback: the diagnosis learning cycle BMJ Qual Saf 2021 30 12 1002 1009 10.1136/bmjqs-2020-012456 34417335 bmjqs-2020-012456 PMC8606468 34417335 · DOI
  21. [28]Rosner B Zwaan L Olson A Imagining the future of diagnostic performance feedback Diagnosis (Berl) 2023 10 1 31 37 10.1515/dx-2022-0055 36378520 dx-2022-0055 36378520 · DOI
  22. [29]Filiberto A Leeds I Loftus T Editorial: machine learning in clinical decision-making Front Digit Health 2021 3 784495 10.3389/fdgth.2021.784495 34870273 34870273 PMC8636718 · DOI
  23. [30]Mizuta K Hirosawa T Harada Y Shimizu T Can ChatGPT-4 evaluate whether a differential diagnosis list contains the correct diagnosis as accurately as a physician? Diagnosis (Berl) 2024 10.1515/dx-2024-0027 38465399 dx-2024-0027 38465399 · DOI
  24. [31]Graber ML Mathew A Performance of a web-based clinical diagnosis support system for internists J Gen Intern Med 2007 23 S1 37 40 10.1007/s11606-007-0271-8 18095042 18095042 PMC2150633 · DOI
  25. [32]Nori H King N McKinney S Carignan D Horvitz E Capabilities of GPT-4 on medical challenge problems arXiv Preprint posted online March 20, 2024 10.48550/arXiv.2303.13375 · DOI · Paper page
  26. [33]Krupat E Wormwood J Schwartzstein RM Richards JB Avoiding premature closure and reaching diagnostic accuracy: some key predictive factors Med Educ 2017 11 51 11 1127 1137 10.1111/medu.13382 28857266 28857266 · DOI
  27. [34]Fleiss J Levin B Paik M Statistical Methods for Rates and Proportions 2003 Hoboken, NJ John Wiley & Sons
  28. [35]Webber W Moffat A Zobel J A similarity measure for indefinite rankings ACM Trans Inf Syst 2010 11 23 28 4 1 38 10.1145/1852102.1852106 · DOI
  29. [37]Chamberland M Setrakian J St-Onge C Bergeron L Mamede S Schmidt HG Does providing the correct diagnosis as feedback after self-explanation improve medical students diagnostic performance? BMC Med Educ 2019 19 1 194 10.1186/s12909-019-1638-3 31185971 10.1186/s12909-019-1638-3 31185971 PMC6558772 · DOI
  30. [38]Abd-Alrazaq Alaa AlSaad R Alhuwail D Ahmed A Healy PM Latifi S Aziz S Damseh R Alabed Alrazak S Sheikh J Large language models in medical education: opportunities, challenges, and future directions JMIR Med Educ 2023 9 e48291 10.2196/48291 37261894 v9i1e48291 37261894 PMC10273039 · DOI · Paper page
  31. [39]Bond WF Schwartz LM Weaver KR Levick D Giuliano M Graber ML Differential diagnosis generators: an evaluation of currently available computer programs J Gen Intern Med 2012 27 2 213 219 10.1007/s11606-011-1804-8 21789717 21789717 PMC3270234 · DOI
  32. [40]Painter A Hayhoe B Riboli-Sasco E El-Osta A Online symptom checkers: recommendations for a vignette-based cinical evaluation standard J Med Internet Res 2022 24 10 e37408 10.2196/37408 36287594 v24i10e37408 36287594 PMC9647454 · DOI
  33. [41]Wiens J Saria S Sendak M Ghassemi M Liu VX Doshi-Velez F Jung K Heller K Kale D Saeed M Ossorio PN Thadaney-Israni S Goldenberg A Do no harm: a roadmap for responsible machine learning for health care Nat Med 2019 25 9 1337 1340 10.1038/s41591-019-0548-6 31427808 10.1038/s41591-019-0548-6 31427808 · DOI · Paper page
  34. [42]Karches KE Against the iDoctor: why artificial intelligence should not replace physician judgment Theor Med Bioeth 2018 39 2 91 110 10.1007/s11017-018-9442-3 29992371 10.1007/s11017-018-9442-3 29992371 · DOI · Paper page
  35. [45]Passi S Vorvoreanu M Overreliance on AI literature review Microsoft 2022 2024-06-12 https://www.microsoft.com/en-us/research/publication/overreliance-on-ai-literature-review/
  36. [46]Pinar SA Cicekli I Akman V Turing test: 50 years later Minds Mach 2000 10 4 50 10.1023/A:1011288000451 · DOI · Paper page

Cited by

Sorted by citations

Authors
Hirosawa, T., Harada, Y., Mizuta, K., Sakamoto, T., Tokumasu, K., & Shimizu, T.
Affiliations
  • Dokkyo Medical University (JP)
  • Okayama University (JP)
Corresponding author
Hirosawa, Takanobu
ORCID
Venue
JMIR Formative Research
Publisher
JMIR Publications Inc.
Published
2024-06-26
Volume
8
Pages
e59267
ISSN
2561-326X (ISSN-L)
DOI
10.2196/59267
PMC
PMC11237772
OpenAlex
W4396646848
Cited by
38
References
6
Citations by year
2026: 15 · 2025: 21 · 2024: 2
FWCI
3.55
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Dokkyo Medical University
  • Japan Society for the Promotion of Science (22K10421)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
ChatGPT-4, large language models, differential diagnosis, physician comparison, diagnostic accuracy, clinical decision support, AI-assisted diagnosis, Google Bard, Llama 2

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review