CoChat
Sign Up
Cited by 50Artificial Intelligence in Healthcare and EducationInnovations in Medical EducationArtificial Intelligence in Education

Performance of ChatGPT and Bard on the medical licensing examinations varies across different cultures: a comparison study

Open access
Chen, Yikai, Huang, Xiujie, Yang, Fangjie, Lin, Haiming, Lin, Haoyu, Zheng, Zhuoqun, Liang, Qifeng, Zhang, Jinhai +1 more·2024·BMC Medical Education
DOI 10.1186/s12909-024-06309-xPMC PMC11590336
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: This study aimed to evaluate the performance of GPT-3.5, GPT-4, GPT-4o and Google Bard on the United States Medical Licensing Examination (USMLE), the Professional and Linguistic Assessments Board (PLAB), the Hong Kong Medical Licensing Examination (HKMLE) and the National Medical Licensing Examination (NMLE). METHODS: This study was conducted in June 2023. Four LLMs (Large Language Models) (GPT-3.5, GPT-4, GPT-4o and Google Bard) were applied to four medical standardized tests (USMLE, PLAB, HKMLE and NMLE). All questions are multiple-choice questions and were sourced from the question banks of these examinations. RESULTS: In USMLE step 1, step 2CK and Step 3, there are accuracy rates of 91.5%, 94.2% and 92.7% provided from GPT-4o, 93.2%, 95.0% and 92.0% provided from GPT-4, 65.6%, 71.6% and 68.5% provided from GPT-3.5, and 64.3%, 55.6%, 58.1% from Google Bard, respectively. In PLAB, HKMLE and NMLE, GPT-4o scored 93.3%, 91.7% and 84.9%, GPT-4 scored 86.7%, 89.6% and 69.8%, GPT-3.5 scored 80.0%, 68.1% and 60.4%, and Google Bard scored 54.2%, 71.7% and 61.3%. There was significant difference in the accuracy rates of four LLMs in the four medical licensing examinations. CONCLUSION: GPT-4o performed better in the medical licensing examinations than other three LLMs. The performance of the four models in the NMLE examination needs further improvement. CLINICAL TRIAL NUMBER: Not applicable.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12909-024-06309-x

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Subsequent evaluations of GPT-4 and GPT-4 Omni documented substantially higher accuracy, frequently exceeding 88-90% correct responses, and outperforming earlier large language models on tasks requiring both factual recall and clinical reasoning.

    “Subsequent evaluations of GPT-4 and GPT-4 Omni documented substantially higher accuracy, frequently exceeding 88-90% correct responses, and, most notably, outperforming earlier LLMs on tasks requiring both factual recall and clinical reasoning (Bicknell et al. 2024;Chen et al. 2024).”

    From Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination (2026) · cites this paper as Chen et al. 2024

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 38 · Cited by 50

Research works printed but not linked: 35

  1. [1]Shaheen A, Azam F, Amir M. ChatGPT and the future of Medical Education: opportunities and challenges. J Coll Physicians Surg Pak. 2023;33(10):1207. 37804033
  2. [2]Feng S, Shen Y. ChatGPT and the future of Medical Education. Acad Med. 2023;98(8):867–8. 37162219 · DOI · Paper page
  3. [3]Extance A. ChatGPT has entered the classroom: how LLMs could transform education. Nature. 2023;623(7987):474–7.37968523 · DOI
  4. [4]Daungsupawong H, Wiwanitkit V. ChatGPT and the future of Medical Education: correspondence. J Coll Physicians Surg Pak. 2024;34(2):244–5. 38342882 · DOI
  5. [5]Seetharaman R. Revolutionizing Medical Education: can ChatGPT boost subjective learning and expression? J Med Syst. 2023;47(1):61. 37160568 · DOI
  6. [8]Cheong RCT, Pang KP, Unadkat S, McNeillis V, Williamson A, Joseph J, et al. Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google Bard. Eur Arch Otorhinolaryngol. 2024;281(4):2137–43. 38117307
  7. [9]Ohta K, Ohta S. The performance of GPT-3.5, GPT-4, and Bard on the Japanese national dentist examination: a comparison study. Cureus. 2023;15(12):e50369. 38213361 PMC10782219
  8. [10]Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova AM, Arteaga-Cisneros KF, Chalco XCB, Ordoñez MAB, et al. Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical examination: a cross-sectional study. J Educ Eval Health Prof. 2023;20:30. 37981579 PMC11009012 · DOI
  9. [11]Thibaut G, Dabbagh A, Liverneaux P. Does Google’s Bard Chatbot perform better than ChatGPT on the European hand surgery exam? Int Orthop. 2024;48(1):151–8. 37968408 · DOI
  10. [13]Giannos P, Delardas O. Performance of ChatGPT on UK standardized admission tests: insights from the BMAT, TMUA, LNAT, and TSA examinations. JMIR Med Educ. 2023;9:e47737. 37099373 PMC10173042 · DOI
  11. [14]Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States Medical Licensing examination? The implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023;9:e45312. 36753318 PMC9947764 · DOI · Paper page
  12. [15]Shang L, Xue M, Hou Y, Tang B. Can ChatGPT pass China’s national medical licensing examination? Asian J Surg. 2023;46(12):6112–3. 37775381 · DOI
  13. [16]Wang X, Gong Z, Wang G, Jia J, Xu Y, Zhao J, et al. ChatGPT performs on the Chinese National Medical Licensing Examination. J Med Syst. 2023;47(1):86. 37581690 · DOI
  14. [17]Besler MS, Oleaga L, Junquero V, Merino C. Evaluating GPT-4o’s performance in the official European board of radiology exam: a comprehensive assessment. Acad Radiol. 2024;31(11):4365–71. 39294055
  15. [18]Ebel S, Ehrengut C, Denecke T, Gossmann H, Beeskow AB. GPT-4o’s competency in answering the simulated written European Board of Interventional Radiology exam compared to a medical student and experts in Germany and its ability to generate exam items on interventional radiology: a descriptive study. J Educ Eval Health Prof. 2024;21:21. 39161266 PMC11894030 · DOI
  16. [19]Liu M, Okuhara T, Dai Z, Huang W, Okada H, Furukawa E et al. Performance of Advanced Large Language Models (GPT-4o, GPT-4, Gemini 1.5 Pro, Claude 3 Opus) on Japanese Medical Licensing Examination: A Comparative Study. medRxiv. 2024:2024.07.09.24310129.
  17. [20]Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. 36812645 PMC9931230 · DOI · Paper page
  18. [23]Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. J Esthet Restor Dent. 2023;35(7):1098–102. 37017291 · DOI · Paper page
  19. [24]Gan RK, Ogbodo JC, Wee YZ, Gan AZ, Gonzalez PA. Performance of Google bard and ChatGPT in mass casualty incidents triage. Am J Emerg Med. 2024;75:72–8. 37967485 · DOI
  20. [25]Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H, et al. Performance of ChatGPT Across different versions in Medical Licensing examinations Worldwide: systematic review and Meta-analysis. J Med Internet Res. 2024;26:e60807. 39052324 PMC11310649 · DOI
  21. [26]Wang H, Wu W, Dou Z, He L, Yang L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inf. 2023;177:105173. 37549499 · DOI
  22. [27]Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing examination: comparison study. JMIR Med Educ. 2023;9:e48002. 37384388 PMC10365615 · DOI · Paper page
  23. [28]Lee YT, Kleinbach R, Peng HPC, Chen ZZ. Cross-cultural research on euthanasia and abortion. J Soc Issues. 1996;52(2):131–48. 15156867 · DOI
  24. [30]Alkaissi H, McFarlane SI. Artificial Hallucinations in ChatGPT: implications in Scientific writing. Cureus. 2023;15(2):e35179. 36811129 PMC9939079
  25. [31]Jin Q, Chen F, Zhou Y, Xu Z, Cheung JM, Chen R et al. Hidden Flaws Behind Expert-Level Accuracy of Multimodal GPT-4 Vision in Medicine. ArXiv. 2024. PMC11266508 39043988
  26. [33]Currie G, Barry K. ChatGPT in Nuclear Medicine Education. J Nucl Med Technol. 2023;51(3):247–54. 37433676 · DOI · Paper page
  27. [34]Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in Healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33. 36869927 PMC9985086 · DOI · Paper page
  28. [35]Kuang YR, Zou MX, Niu HQ, Zheng BY, Zhang TL, Zheng BW. ChatGPT encounters multiple opportunities and challenges in neurosurgery. Int J Surg. 2023;109(10):2886–91. 37352529 PMC10583932
  29. [36]Massey PA, Montgomery C, Zhang AS. Comparison of ChatGPT-3.5, ChatGPT-4, and Orthopaedic Resident performance on Orthopaedic Assessment examinations. J Am Acad Orthop Surg. 2023;31(23):1173–9. 37671415 PMC10627532
  30. [37]Giannos P. Evaluating the limits of AI in medical specialisation: ChatGPT’s performance on the UK Neurology Specialty Certificate Examination. BMJ Neurol Open. 2023;5(1):e000451. PMC10277081 37337531 · DOI
  31. [38]Pushpanathan K, Lim ZW, Er Yew SM, Chen DZ, Hui’En Lin HA, Lin Goh JH, et al. Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries. iScience. 2023;26(11):108163. 37915603 PMC10616302 · DOI
  32. [40]Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inf Assoc. 2023;30(7):1237–45. PMC10280357 37087108 · DOI · Paper page
  33. [41]Xie Y, Seth I, Hunter-Smith DJ, Rozen WM, Ross R, Lee M. Aesthetic surgery advice and counseling from Artificial Intelligence: a Rhinoplasty Consultation with ChatGPT. Aesthetic Plast Surg. 2023;47(5):1985–93. 37095384 PMC10581928 · DOI · Paper page
  34. [42]Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023;29(3):721–32. 36946005 PMC10366809 · DOI
  35. [43]Lahat A, Shachar E, Avidan B, Shatz Z, Glicksberg BS, Klang E. Evaluating the use of large language model in identifying top research questions in gastroenterology. Sci Rep. 2023;13(1):4164. 36914821 PMC10011374 · DOI · Paper page

Authors
Chen, Y., Huang, X., Yang, F., Lin, H., Lin, H., Zheng, Z., Liang, Q., Zhang, J., & Li, X.
Affiliations
  • First Affiliated Hospital of Shantou University Medical College (CN)
  • University of Alberta (CA)
Corresponding author
  • Zhang, Jinhai
  • Li, Xinxin
ORCID
Venue
BMC Medical Education
Publisher
Springer Science and Business Media LLC
Published
2024-11-26
Volume
24
Issue
1
Pages
1372-1372
ISSN
1472-6920 (ISSN-L)
DOI
10.1186/s12909-024-06309-x
PMC
PMC11590336
OpenAlex
W4404717881
Cited by
50
References
38
Citations by year
2026: 24 · 2025: 26
FWCI
4.67
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Education (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Guangdong Province undergraduate higher education teaching reform project (No.583)
  • Guangdong Province clinical teaching base teaching reform project (No.128, 2023 NO.32)
  • Ministry of Education industry-university cooperative education project (No.230805236050915)
  • Guangdong Province “New medical” teaching committee teaching reform project (2023 NO.122)
  • Shantou University Medical School undergraduate teaching quality and teaching reform project (2023 No.25)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
large language models, ChatGPT, Google Bard, GPT-4, GPT-4o, medical licensing examinations, USMLE, PLAB, multiple-choice questions, cross-cultural comparison, model accuracy
MeSH terms
Cross-Cultural Comparison, Educational Measurement, Hong Kong, Humans, Licensure, Medical, United States

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review