CoChat
Sign Up
Cited by 11Artificial Intelligence in Healthcare and EducationArtificial Intelligence in EducationInnovations in Medical Education

Comparative performance of ChatGPT-4o, ChatGPT-5, and gemini 2.5 flash on Persian internal medicine subspecialty board exams

Open access
Sheikhalishahi, Shahab, Haddadi, Alireza, Sadeghipour, Saina, Rafiei, Farzad, Soltani, Hamidreza·2025·Scientific Reports
DOI 10.1038/s41598-025-31251-3PMC PMC12796361

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

This study compared the performance of ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash on the 2025 Iranian internal medicine subspecialty board examinations. A total of 650 multiple-choice questions from six subspecialties were tested, excluding image-based items. Each question was presented in Persian, and responses were evaluated against the official answer key. Accuracy rates were 68.9% for ChatGPT-4o, 74.5% for ChatGPT-5, and 79.9% for Gemini 2.5 Flash, with Gemini performing significantly better than both ChatGPT versions. ChatGPT-5 also showed a significant improvement over ChatGPT-4o, confirming rapid progress in model development. Subspecialty analysis revealed stronger results in rheumatology and respiratory medicine compared to nephrology, while question type and length had no significant impact on outcomes. An artificial neural network that combined the outputs of all three models reached 81.6% accuracy, slightly exceeding Gemini alone. These findings highlight Gemini-2.5 as the most reliable model for this high-stakes internal medicine exam. The results support the growing role of advanced AI systems as assistants in medical education and clinical practice. However, further research is needed to assess their use in multimodal and real-world clinical tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-025-31251-3

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Gemini 2.5 Pro fared relatively better on Taxonomy 3 (clinical reasoning) questions, consistent with reports that newer Gemini models often excel on complex and integrative problem solving tasks.

    “Gemini 2.5 Pro fared relatively better on Taxonomy 3 (clinical reasoning) questions, consistent with reports than newer Gemini models often excel on complex and integrative problem solving tasks (Wielochowska et al. 2025;Sheikhalishahi et al. 2025;Boczkowski et al. 2025).”

    From Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination (2026) · cites this paper as Sheikhalishahi et al. 2025

    The Discussion (page unavailable) reports that Gemini 2.5 Flash outperformed the compared ChatGPT models on Persian internal-medicine board questions and describes an advantage in text-based clinical reasoning. It does not establish that newer Gemini models often excel across complex and integrative problem-solving tasks. The cited study tested Flash, not Gemini 2.5 Pro, and did not analyze Taxonomy 3 questions. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Comparative evaluations of ChatGPT-5 and Gemini 2.5 on internal medicine subspecialty board examinations demonstrate higher overall accuracy for Gemini, while relative performance varies across subspecialties rather than remaining consistent across domains.

    “More recent comparative evaluations of ChatGPT-5 and Gemini 2.5 on internal medicine subspecialty board examinations demonstrate higher overall accuracy for Gemini, while also showing that relative performance varies across subspecialties rather than remaining consistent across domains (Sheikhalishahi et al. 2025).”

    From Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination (2026) · cites this paper as Sheikhalishahi et al. 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [1]Abd-Alrazaq, A. et al. Large Language models in medical education: Opportunities, Challenges, and future directions. JMIR Med. Educ. 9, e48291 (2023). 37261894 PMC10273039 · DOI · Paper page
  2. [2]Elzayyat, M., Mohammad, J. N. & Zaqout, S. Assessing LLM-generated vs. expert-created clinical anatomy mcqs: a student perception-based comparative study in medical education. Med. Educ. Online. 30 (1), 2554678 (2025). 40884796 PMC12404065 · DOI
  3. [3]Ling, Q. et al. Assessing the possibility of using large Language models in ocular surface diseases. Int. J. Ophthalmol. 18 (1), 1–8 (2025). 39829624 PMC11672086 · DOI
  4. [4]Gotta, J. et al. Large Language models (LLMs) in radiology exams for medical students: performance and consequences. Rofo 197 (9), 1057–1067 (2025). 39496293 · DOI
  5. [5]Bahir, D. et al. Gemini AI vs. ChatGPT: A comprehensive examination alongside ophthalmology residents in medical knowledge. Graefes Arch. Clin. Exp. Ophthalmol. 263 (2), 527–536 (2025). 39277830
  6. [6]Al-Thani, S. N. et al. Comparative performance of ChatGPT, Gemini, and final-year emergency medicine clerkship students in answering multiple-choice questions: implications for the use of AI in medical education. Int. J. Emerg. Med. 18 (1), 146 (2025). 40775272 PMC12329864 · DOI
  7. [8]Wu, X. et al. A multi-dimensional performance evaluation of large Language models in dental implantology: comparison of ChatGPT, DeepSeek, Grok, gemini and Qwen across diverse clinical scenarios. BMC Oral Health. 25 (1), 1272 (2025). 40721763 PMC12302792 · DOI
  8. [9]Singal, A. & Goyal, S. Comparative evaluation of AI platforms Google gemini 2.5 Flash, Google gemini 2.0 Flash, deepseek V3 and ChatGPT 4o in solving multiple-choice questions from different subtopics of anatomy. Surg. Radiol. Anat. 47 (1), 193 (2025). 40885861 · DOI
  9. [10]Masalkhi, M., Ong, J., Waisberg, E. & Lee, A. G. Google deepmind’s gemini AI versus chatgpt: a comparative analysis in ophthalmology. Eye (Lond). 38 (8), 1412–1417 (2024). 38355668 PMC11126415 · DOI
  10. [11]Marey, A. et al. Evaluating the accuracy and reliability of AI chatbots in patient education on cardiovascular imaging: a comparative study of ChatGPT, gemini, and copilot. Egypt. J. Radiol. Nuclear Med. 56 (1), 37 (2025). · DOI · Paper page
  11. [12]Bayala, Y. L. T. et al. Performance of the large Language models in African rheumatology: a diagnostic test accuracy study of ChatGPT-4, Gemini, Copilot, and Claude artificial intelligence. BMC Rheumatol. 9 (1), 54 (2025). 40380276 PMC12083132 · DOI
  12. [13]Madrid-García, A. et al. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Sci. Rep. 13 (1), 22129 (2023). 38092821 PMC10719375 · DOI
  13. [14]Meral, G., Ateş, S., Günay, S., Öztürk, A. & Kuşdoğan, M. Comparative analysis of ChatGPT, gemini and emergency medicine specialist in ESI triage assessment. Am. J. Emerg. Med. 81, 146–150 (2024). 38728938 · DOI · Paper page
  14. [15]Lee, J. T. et al. Evaluation of performance of generative large Language models for stroke care. Npj Digit. Med. 8 (1), 481 (2025). 40730644 PMC12307639 · DOI
  15. [16]Bahir, D. et al. Gemini AI vs. ChatGPT: A comprehensive examination alongside ophthalmology residents in medical knowledge. Graefe’s Archive Clin. Experimental Ophthalmol. 263 (2), 527–536 (2025). 39277830
  16. [17]Sabaner, M. C. & Yozgat, Z. Performance of ChatGPT-4 omni and gemini 1.5 pro on Ophthalmology-Related questions in the Turkish medical specialty exam. Turk. J. Ophthalmol. 55 (4), 177–185 (2025). 40838476 PMC12372544 · DOI
  17. [18]Hernández-Flores, L. A. et al. Assessment of challenging oncologic cases: A comparative analysis between ChatGPT, Gemini, and a multidisciplinary tumor board. J. Surg. Oncol. 131 (8), 1562–1570 (2025). 39936586 · DOI
  18. [20]Wang, T. et al. Evaluating the performance of State-of-the-Art artificial intelligence chatbots based on the WHO global guidelines for the prevention of surgical site infection: Cross-Sectional study. J. Med. Internet Res. 27, e75567 (2025). 40744114 PMC12313333 · DOI
  19. [21]Li, H. et al. ChatGPT-4o outperforms gemini advanced in assisting multidisciplinary decision-making for advanced gastric cancer. Eur. J. Surg. Oncol. 51 (8), 110096 (2025). 40294561 · DOI
  20. [22]Lin, C. R. et al. Multiple large Language models versus clinical guidelines for postmenopausal osteoporosis: a comparative study of ChatGPT-3.5, ChatGPT-4.0, ChatGPT-4o, Google gemini, Google gemini Advanced, and Microsoft copilot. Arch. Osteoporos. 20 (1), 120 (2025). 40921907 · DOI
  21. [23]Liu, R., Liu, J., Yang, J., Sun, Z. & Yan, H. Comparative analysis of ChatGPT-4o mini, ChatGPT-4o and gemini advanced in the treatment of postmenopausal osteoporosis. BMC Musculoskelet. Disord. 26 (1), 369 (2025). 40241048 PMC12001388 · DOI
  22. [24]Muluk, E. A comparative analysis of artificial intelligence platforms: ChatGPT-4o and Google gemini in answering questions about birth control methods. Cureus 17 (1), e76745 (2025). 39897238 PMC11785371
  23. [25]McNulty, A. M. et al. Performance evaluation of ChatGPT-4.0 and gemini on image-based neurosurgery board practice questions: A comparative analysis. J. Clin. Neurosci. 134, 111097 (2025). 39938135 · DOI
  24. [26]Sau, S. et al. Accuracy and quality of ChatGPT-4o and Google gemini performance on image-based neurosurgery board questions. Neurosurg. Rev. 48 (1), 320 (2025). 40131528 · DOI
  25. [27]Ali, R. et al. Performance of ChatGPT and GPT-4 on neurosurgery written board examinations. Neurosurgery 93 (6), 1353–1365 (2023). 37581444 · DOI
  26. [28]Huang, K. A., Choudhary, H. K., Hardin, W. M. & Prakash, N. Comparative analysis of ChatGPT-4o and gemini advanced performance on diagnostic radiology In-Training exams. Cureus 17 (3), e80874 (2025). 40255788 PMC12009162
  27. [29]Clark, K. R. Comparative analysis of llms’ performance on a practice radiography certification exam. Radiol. Technol. 96 (5), 334–342 (2025). 40880380 · DOI
  28. [30]Suh, P. S. et al. Comparing diagnostic accuracy of radiologists versus GPT-4V and gemini pro vision using image inputs from diagnosis please cases. Radiology 312 (1), e240273 (2024). 38980179 · DOI
  29. [31]Jain, S., Chakraborty, B., Agarwal, A. & Sharma, R. Performance of large Language models (ChatGPT and gemini Advanced) in Gastrointestinal pathology and clinical review of applications in gastroenterology. Cureus 17 (4), e81618 (2025). 40322390 PMC12048130
  30. [32]Khan, A. A. et al. Artificial intelligence for anesthesiology Board-Style examination questions: role of large Language models. J. Cardiothorac. Vasc Anesth. 38 (5), 1251–1259 (2024). 38423884 · DOI
  31. [33]Passby, L., Jenko, N. & Wernham, A. Performance of ChatGPT on specialty certificate examination in dermatology multiple-choice questions. Clin. Exp. Dermatol. 49 (7), 722–727 (2024). 37264670 · DOI
  32. [34]Dhanvijay, A. K. D. et al. Performance of large Language models (ChatGPT, Bing Search, and Google Bard) in solving case vignettes in physiology. Cureus 15 (8), e42972 (2023). 37671207 PMC10475852
  33. [35]Kumari, A. et al. Large Language models in hematology case solving: A comparative study of ChatGPT-3.5, Google Bard, and Microsoft Bing. Cureus 15 (8), e43861 (2023). 37736448 PMC10511207
  34. [36]Cheong, R. C. T. et al. Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google bard. Eur. Arch. Otorhinolaryngol. 281 (4), 2137–2143 (2024). 38117307 · DOI
  35. [37]Makrygiannakis, M. A., Giannakopoulos, K. & Kaklamanos, E. G. Evidence-based potential of generative artificial intelligence large Language models in orthodontics: a comparative study of ChatGPT, Google Bard, and Microsoft Bing. Eur. J. Orthod. (2024). PMC12810200 38613510
  36. [38]Yamaguchi, S. et al. Evaluating the efficacy of leading large Language models in the Japanese National dental hygienist examination: A comparative analysis of chatGPT, Bard, and Bing chat. J. Dent. Sci. 19 (4), 2262–2267 (2024). 39347065 PMC11437298 · DOI
  37. [39]Fukuda, H. et al. Evaluating the image recognition capabilities of GPT-4V and gemini pro in the Japanese National dental examination. J. Dent. Sci. 20 (1), 368–372 (2025). 39873040 PMC11762652 · DOI
  38. [40]Khan, M. P. & O’Sullivan, E. D. A comparison of the diagnostic ability of large Language models in challenging clinical cases. Front. Artif. Intell. 7, 1379297 (2024). 39161790 PMC11330891 · DOI
  39. [41]Khan, A. A. et al. Assessing the performance of ChatGPT in medical ethical decision-making: a comparative study with USMLE-based scenarios. J. Med. Ethics (2025). 39863417
  40. [42]Lee, Y. et al. Harnessing artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and bard in generating clinician-level bariatric surgery recommendations. Surg. Obes. Relat. Dis. 20 (7), 603–608 (2024). 38644078 · DOI
  41. [43]Lee, Y. et al. Performance of artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and bard in the American society for metabolic and bariatric surgery textbook of bariatric surgery questions. Surg. Obes. Relat. Dis. 20 (7), 609–613 (2024). 38782611 · DOI
  42. [44]Anvari, S., Lee, Y., Jin, D. S., Malone, S. & Collins, M. Artificial intelligence in hepatology: a comparative analysis of ChatGPT-4, Bing, and bard at answering clinical questions. J. Can. Assoc. Gastroenterol. 8 (2), 58–62 (2025). 40224573 PMC11991870 · DOI
  43. [45]Zare, S. et al. Comparing the performance of ChatGPT-3.5-Turbo, ChatGPT-4, and Google bard with Iranian students in pre-internship comprehensive exams. Sci. Rep. 14 (1), 28456 (2024). 39557926 PMC11574257 · DOI
  44. [46]Roos, J., Martin, R. & Kaczmarczyk, R. Evaluating bard gemini pro and GPT-4 vision against student performance in medical visual question answering: comparative case study. JMIR Form. Res. 8, e57592 (2024). 39714199 PMC11683658 · DOI
  45. [47]Meo, S. A., Abukhalaf, F. A., ElToukhy, R. A. & Sattar, K. Exploring the role of DeepSeek-R1, ChatGPT-4, and Google gemini in medical education: how valid and reliable are they? Pak J. Med. Sci. 41 (7), 1887–1892 (2025). 40735572 PMC12302124 · DOI

Cited by

Sorted by citations

Authors
Sheikhalishahi, S., Haddadi, A., Sadeghipour, S., Rafiei, F., & Soltani, H.
Affiliations
Shahid Sadoughi University of Medical Sciences and Health Services (IR)
Corresponding author
  • Rafiei, Farzad
  • Soltani, Hamidreza
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2025-12-03
Volume
16
Issue
1
Pages
1371-1371
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-025-31251-3
PMC
PMC12796361
OpenAlex
W4417035903
Cited by
11
References
4
Citations by year
2026: 11
FWCI
2.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Keywords
large language models, GPT-4o, ChatGPT-5, Gemini 2.5 Flash, Persian language, multiple-choice question accuracy, rheumatology, respiratory medicine, nephrology, neural network ensembles, medical education
MeSH terms
Generative Artificial Intelligence, Educational Measurement, Humans, Internal Medicine, Iran, Specialty Boards, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review