Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Open accessPublished version of a preprint
Abstract
Background: The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied. Objective: This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages. Methods: This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions. Results: The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages. Conclusions: GPT-4 could become a valuable tool for medical education and clinical support in non–English-speaking regions, such as Japan.
AI snapshot
Written from the paper’s full text.
TL;DR
GPT-4 outperformed GPT-3.5 on the 117th Japanese Medical Licensing Examination (JMLE), reaching the authors’ stated passing criteria while GPT-3.5 did not. The study suggests GPT-4 has improved non-English medical knowledge and clinical reasoning, though limitations include a March 2023 snapshot and exclusion of image/table questions.
Key findings
- Overall, GPT-4 answered 79.9% (95% CI 75.0-84.9) of the 254 analyzed JMLE questions correctly versus 50.8% (95% CI 44.6-57.0) for GPT-3.5, a 29.1% improvement (P<.001).
- By question category, GPT-4 scored 87.2% on essential knowledge, 73.3% on general clinical, and 81.7% on specific disease questions; GPT-3.5 scored 55.1%, 43.8%, and 56.3%, respectively (all P<.001).
- By question type, GPT-4 scored 79.1% on general, 79.6% on clinical, and 86.3% on clinical sentence questions, compared with approximately 50% for GPT-3.5 across these types (all P<.05).
- By difficulty, GPT-4 scored 87.8% on easy, 77.7% on normal, and 73.3% on hard questions; GPT-3.5 scored 69.5%, 46.2%, and 33.3%, respectively. For hard questions, GPT-4’s rate was 17% higher than the examinees’ average correct response rate.
Methods
- The study used the 117th JMLE from 2023, with 254 questions included after excluding 5 officially excluded questions and questions containing tables (n=7), images (n=125), or underlining (n=9).
- GPT-3.5 (default ChatGPT) and GPT-4 (ChatGPT Plus) were queried in Japanese with original JMLE questions and multiple-choice answers; responses were collected between March 16 and 18, 2023.
- Questions were categorized by type (general, clinical, clinical sentence) and difficulty (easy, normal, hard) based on medu4 correct-response rates; correct-response rates were compared using the McNemar test in Stata, with 2-tailed significance at P<.05.
Limitations
- The paper states that results reflect ChatGPT capabilities as of March 17 and 18, 2023, and may change as models improve.
- The paper states that image and table questions were excluded to allow accurate comparison with GPT-3.5, and that the JMLE supplementary assessment for absolute contraindication answers was not evaluated because failed applicants’ scores were not published.
- The paper states that only ChatGPT was investigated, not other LLMs such as Bard or LLaMA; evident from the text, the clinical sentence question subset was small (n=22) with wide confidence intervals, and the difficulty-level counts differ between the Methods text and Table 3.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 11 cited sources checked · 0 supported · 0 flagged
- Findingown data
GPT-4 achieved a 27.6% increase over GPT-3.5 in correct response rate for general questions (P<.001).
“However, GPT-4 achieved a 27.6% increase for general questions (P<.001) and a 29.6% increase for clinical questions (P<.001) compared to GPT-3.5.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved a 29.6% increase over GPT-3.5 in correct response rate for clinical questions (P<.001).
“However, GPT-4 achieved a 27.6% increase for general questions (P<.001) and a 29.6% increase for clinical questions (P<.001) compared to GPT-3.5.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved a 31.5% increase over GPT-3.5 in correct response rate for normal-level questions (P<.001).
“However, GPT-4 exhibited improved performance, with a 40% increase for hard-level questions (P<.001), a 31.5% increase for normal-level questions (P<.001), and an 18.3% increase for easy-level questions (P<.001).”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved a 40% increase over GPT-3.5 in correct response rate for hard-level questions (P<.001).
“However, GPT-4 exhibited improved performance, with a 40% increase for hard-level questions (P<.001), a 31.5% increase for normal-level questions (P<.001), and an 18.3% increase for easy-level questions (P<.001).”
Quote in the paperSubstantial claim - Findingown data
None of GPT-4's correct response rates on the 117th JMLE exceeded the total percentage of correct answers by examinees.
“However, none of these rates exceeded the total percentage of correct answers by examinees.”
Quote in the paperNot a substantial claim
- Findingown data
GPT-4 achieved a 36.3% increase over GPT-3.5 in the number of correct responses to clinical sentence questions.
“Notably, a 36.3% increase was observed in the number of correct responses to clinical sentence questions, with a significant improvement in all question types (all P<.05).”
Quote in the paperNot a substantial claim - Findingown data
For hard-level questions, GPT-4's correct response rate was 17% higher than the examinees' average correct response rate.
“In particular, for hard-level questions, the correct response rate of GPT-4 was 17% higher than the examinees’ average correct response rate.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved an 18.3% increase over GPT-3.5 in correct response rate for easy-level questions (P<.001).
“However, GPT-4 exhibited improved performance, with a 40% increase for hard-level questions (P<.001), a 31.5% increase for normal-level questions (P<.001), and an 18.3% increase for easy-level questions (P<.001).”
Quote in the paperSubstantial claim - Findingown data
GPT-4 significantly outperformed GPT-3.5 in all question formats in terms of correct response rates (all P<.05).
“Finally, GPT-4 significantly outperformed GPT-3.5 in all formats in terms of correct response rates (all P<.05).”
Quote in the paperSubstantial claim - Findingown data
On the 117th JMLE, GPT-4 significantly outperformed GPT-3.5 overall by 29.1 percentage points (P<.001).
“Overall, GPT-4 significantly outperformed GPT-3.5 by 29.1% (P<.001).”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved a 29.5% increase over GPT-3.5 in correct response rate on general clinical questions of the 117th JMLE.
“Similarly, a 29.5% increase was observed for general clinical questions, and a 25.4% increase was observed for specific disease questions.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 achieved a 25.4% increase over GPT-3.5 in correct response rate on specific disease questions of the 117th JMLE.
“Similarly, a 29.5% increase was observed for general clinical questions, and a 25.4% increase was observed for specific disease questions.”
Quote in the paperSubstantial claim - Premisecitation
OpenAI's assessment of GPT-4's performance in non-English languages yielded higher proficiencies in 24 out of 26 languages compared to the previous models' proficiency in English.
“OpenAI assessed GPT-4’s performance in non-English languages, which yielded higher proficiencies in 24 out of 26 languages as compared to the previous models’ proficiency in English [13].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [13] source
Not checked in the free run.
- Premisecitation
A performance disparity between English and other languages in large language models is ubiquitous in NLP.
“A performance disparity between English and other languages in LLMs is ubiquitous in NLP [19].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [19] BenderEMThe #BenderRule: on naming the languages we study and why it mattersThe Gradient20099142023-03-06https://thegradient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters/
Not checked in the free run.
- Interpretationcitation
GPT-4's superior correct response rate on hard-level questions compared with the average correct response rate of general examinees indicates the potential of language models such as GPT-4 to surpass human performance in highly specialized fields.
“Notably, GPT-4’s superior correct response rate on the challenging hard-level questions, compared with the average correct response rate of general examinees, indicates the potential of language models such as GPT-4 to surpass human performance in highly specialized fields [13].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [13] source
Not checked in the free run.
- Interpretationreasoning
The improvement in GPT-4's correct response rates on the JMLE is attributable to augmented NLP capabilities in non-English languages.
“First, we ascribe this enhancement to the augmented NLP capabilities in non-English languages.”
Quote in the paperSubstantial claim - Premisecitation
GPT-3.5 exhibits a decline in NLP proficiency in non-English languages relative to English.
“Additionally, GPT-3.5 exhibits a decline in NLP proficiency in non-English languages relative to English [20].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [20] source
Not checked in the free run.
- Interpretationown data
GPT-4's achievement of the JMLE passing criteria indicates its reliability for clinical reasoning and medical knowledge in non-English languages.
“Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages.”
Quote in the paperSubstantial claim- Rests on an assumption: The Japanese Medical Licensing Examination is a valid benchmark for clinical reasoning and medical knowledge in Japanese.
- Interpretationreasoning
GPT-4's satisfaction of the JMLE passing criteria demonstrates a significant advancement in NLP capabilities specifically in Japanese.
“In contrast, GPT-4 satisfied the JMLE passing criteria, demonstrating a significant advancement in NLP capabilities, specifically in Japanese.”
Quote in the paperSubstantial claim - Interpretationown data
The results of this research validate OpenAI's assertion that GPT-4 has higher proficiency in non-English languages.
“Although OpenAI did not disclose the precise methodologies used to obtain these outcomes, the results of this research validate their assertion.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [13] source
Not checked in the free run.
- May combine two claims. 6
- The quote only partly states this claim. 1
- Classified as probably not a substantial claim. 7
May combine two claims.
Findingown dataGPT-4 achieved an 87.2% correct response rate on essential knowledge questions of the 117th JMLE, a 32.1% improvement over GPT-3.5's 55.1%.
“In terms of the correct response rate for individual questions, the examinees’ rate for essential knowledge questions was 89.2% compared to 87.2% for GPT-4. Notably, this represents a considerable 32.1% improvement over GPT-3.5, which had a 55.1% correct response rate.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-3.5 achieved a 69.5% correct response rate for easy-level questions, 46.2% for normal-level questions, and 33.3% for hard-level questions on the 117th JMLE.
“GPT-3.5 only achieved a 69.5% correct response rate for easy-level questions, 46.2% for normal-level questions, and 33.3% for hard-level questions.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataGPT-4 achieved the JMLE passing rates in all question categories (essential knowledge, general clinical, and specific disease).
“In all cases, GPT-4 achieved the passing rates for the JMLE.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-3.5 achieved correct response rates of approximately 50% across question types on the 117th JMLE, none of which are passing scores.
“Table 2 presents the correct response rates according to the question type, with GPT-3.5 achieving correct response rates of approximately 50%—none of which are passing scores.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4 passed the 117th JMLE, whereas GPT-3.5 failed the examination.
“GPT-4 passed the 117th JMLE, whereas GPT-3.5 failed the examination.”
- Also stated in: § Principal Findings · plain
“GPT-3.5 did not satisfy the passing criteria, whereas GPT-4 achieved the required scores.”
Quote in the paperSubstantial claim- Also stated in: § Principal Findings · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationChatGPT achieved the passing criteria for the US Medical Licensing Examination.
“In the medical domain, ChatGPT achieved the passing criteria for the US Medical Licensing Examination (USMLE) [5,6].”
Quote in the paperNot a substantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationEven in English, in a real clinical setting, GPT-3.5 cannot answer questions at a level acceptable to fully qualified primary care physicians.
“A previous study suggests that even in English, in a real clinical setting, GPT-3.5 cannot answer questions at a level acceptable to fully qualified primary care physicians [10].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [10] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationCompared with GPT-3.5, GPT-4 is more reliable, creative, and able to handle many more nuanced instructions.
“Compared with its predecessor GPT-3.5, GPT-4 is “more reliable, creative, and able to handle many more nuanced instructions” [12].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [12] GPT-4 is OpenAI’s most advanced system, producing safer and more useful responsesOpenAI2023-03-19https://openai.com/product/gpt-4
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningQuestions announced as excluded by the MHLW and questions containing tables, images, or underlining were excluded from the analysis, leaving 254 questions.
“The exclusion criteria included questions for which the Ministry of Health, Labour and Welfare (MHLW) announced as being excluded (n=5), as well as questions containing tables (n=7), images (n=125), and underlining (n=9), which are not recognized by ChatGPT. In total, 254 questions were used in the final analysis.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationChatGPT based on GPT-3.5 performed poorly for 77 out of 79 medical students on a South Korean parasitology examination.
“However, in a previous study, ChatGPT, based on GPT-3.5 (Generative Pre-trained Transformer), performed poorly for 77 out of 79 medical students on a South Korean parasitology examination, which resulted in questions about its ability to provide medically accurate responses in non-English languages [11].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [11] source
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningGPT-4 could become a valuable tool for medical education and clinical support in non-English-speaking regions such as Japan.
“GPT-4 could become a valuable tool for medical education and clinical support in non–English-speaking regions, such as Japan.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOpenAI reported that hallucinations have been mitigated in GPT-4 compared with GPT-3.5.
“OpenAI reported that hallucinations have been mitigated in GPT-4 compared with GPT-3.5 [21].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [21] GPT-4 system cardOpenAI20233232023-03-21https://cdn.openai.com/papers/gpt-4-system-card.pdf
Not checked in the free run.
The paper relies on this without support.
The Japanese Medical Licensing Examination is a valid benchmark for clinical reasoning and medical knowledge in Japanese.
“The JMLE is a mandatory exam for certifying medical practitioners in Japan, necessitating a comprehensive knowledge base and strong clinical reasoning skills.”
The paper relies on this without support.
The 254 included questions are representative of the full 117th JMLE for comparing GPT-3.5 and GPT-4.
“In total, 254 questions were used in the final analysis.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Across all models, performance tended to decline with increased difficulty, especially for higher-level questions requiring multistep reasoning, where accuracy dropped markedly.
“Across all models, performance tended to decline with increased difficulty, especially for higher-level questions that required multistep reasoning, where accuracy dropped markedly [39-45].”
From Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review (2025) · cites this paper as [43]
Results, Table 3 shows that accuracy declined from easy to hard questions for both GPT-3.5 and GPT-4. The paper classifies difficulty by examinee correct-response rates, not by whether questions require multistep reasoning, and it tests only these two models. It therefore does not establish the claimed drop specifically for multistep-reasoning questions or across all models beyond those tested. (LLM, conf=0.88)
Experimental — accuracy not yet measuredGPT-4 achieved a score that met the passing standard of the JMLE, demonstrating robust reliability in clinical reasoning and medical knowledge within the Japanese context.
“Furthermore, GPT-4 achieved a score that met the passing standard of the JMLE, demonstrating its robust reliability in clinical reasoning and medical knowledge within the Japanese context.”
From Large language models in medical and healthcare fields: applications, advances, and challenges (2024) · cites this paper as Takagi et al. 2023
GPT-4 outperformed ChatGPT in general clinical questions, complex questions, and specific disease-related queries in JMLE.
“The results revealed that GPT-4 outperformed ChatGPT in general clinical questions, complex questions, and specific disease-related queries.”
From Large language models in medical and healthcare fields: applications, advances, and challenges (2024) · cites this paper as Takagi et al. 2023
GAI is capable of passing various certification examinations, including medical licensure examinations, at least for questions without images.
“For example, GAI is now capable of passing various certification examinations, including those for medical licensure in at least questions without images [11,12].”
From Developing Medical Education Curriculum Reform Strategies to Address the Impact of Generative AI: Qualitative Study (2023) · cites this paper as [12]
GPT-4 passed a Japanese medical examination.
“Still, with assistance and adaptations, GPT-3.5 nearly passed the United States Medical Licensing Examination [13,14], and GPT-4 passed a Japanese medical examination [15].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [15]
GPT-4 showed a nearly 30% performance increase over GPT-3.5.
“Moreover, the nearly 30% performance increase from GPT-3.5 to GPT-4, as indicated in this study and supported by a Japanese study, which suggests a similar language distribution within the GPT-3.5 and GPT-4 data sets [15].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [15]
GPT-3.5's and GPT-4's performance fell short compared to that of medical students in a Japanese medical examination.
“Despite these improvements, GPT-3.5’s or GPT-4’s performance still fell short in comparison to that of medical students in a Japanese medical examination according to the study by Takagi et al [15].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Easy and hard question counts are reversed
MinorThe Methods report 82 hard and 60 easy questions, but Table 3 reports 60 hard and 82 easy questions.
“We evaluated the difficulty level of each question and categorized them as hard (n=82), normal (n=112), and easy (n=60)”
“| Easy | 82 (32.3) | 98.7 | 69.5 (59.3-79.7) | 87.8 (80.6-95.0) | .001 | | Normal | 112 (44.1) | 90.2 | 46.2 (37.0-55.8) | 77.7 (69.8-85.5) | <.001 | | Hard | 60 (23.6) | 56.3 | 33.3% (21.1-45.6) | 73.3 (61.8-84.8) | <.001 |”
Difficulty thresholds differ at their boundaries
MinorThe Methods include a 79.9% response rate in the hard category and a 97% rate in the easy category. Table 3's note uses strict inequalities that exclude both boundary values.
“Questions with a correct response rate of 79.9% or below were classified as hard, those with a rate between 80% and 96.9% were classified as normal, and those with a rate of 97% or higher were classified as easy.”
“easy, >97%; normal, 80% to 96.9%; and hard, <79.9%.”
Difficulty-group GPT-3.5 rates do not match the overall rate
MinorWeighting Table 3's GPT-3.5 rates by its question counts gives approximately 50.7%, not the 50.8% reported for all 254 questions in Table 1; the gap exceeds rounding to one decimal place. The normal-group rate of 46.2% also cannot result from an integer number correct out of 112 questions.
“| Easy | 82 (32.3) | 98.7 | 69.5 (59.3-79.7) | 87.8 (80.6-95.0) | .001 | | Normal | 112 (44.1) | 90.2 | 46.2 (37.0-55.8) | 77.7 (69.8-85.5) | <.001 | | Hard | 60 (23.6) | 56.3 | 33.3% (21.1-45.6) | 73.3 (61.8-84.8) | <.001 |”
“| All questions | 254 (100) | 84.9 | 50.8 (44.6-57.0) | 79.9 (75.0-84.9) | <.001 |”
Cites
Rosenbloom, L. (2019) · Charleston Advisor · doi:10.5260/chara.21.2.8 · Cited by 16,663 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Role of Chat GPT in public health
Biswas, S. S. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03172-7 · Cited by 673 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
ChatGPT in Healthcare: A Taxonomy and Systematic ReviewPreprint
Li, J., Dada, A., Puladi, B., et al. (2023) · medRxiv · doi:10.1101/2023.03.30.23287899 · Cited by 411 · see citing works
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 344 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.1 · Cited by 278 · see citing works
Huh, S. (2023) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2023.20.01 · Cited by 199 · see citing works
GPT Takes the Bar ExamPreprint
Bommarito, M., & Katz, D. (2022) · Social Science Research Network · doi:10.48550/arxiv.2212.14402 · Cited by 195 · see citing works
Thirunavukarasu, A. J., Hassan, R., Mahmood, S., Sanghera, R., Barzangi, K., El Mukashfi, M., & Shah, S. (2023) · JMIR Medical Education · doi:10.2196/46599 · Cited by 187 · see citing works
Bommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181 · see citing works
ChatGPT Performs on the Chinese National Medical Licensing Examination
Wang, X., Gong, Z., Wang, G., Jia, J., Xu, Y., Zhao, J., Fan, Q., Wu, S., Hu, W., & Li, X. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01961-0 · Cited by 162 · see citing works
ChatGPT: not all languages are equal
Seghier, M. L. (2023) · Nature · doi:10.1038/d41586-023-00680-3 · Cited by 108 · see citing works
Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, J. A. M., & Katz, D. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322372 · Cited by 68 · see citing works
Insights from teaching artificial intelligence to medical students in Canada
Hu, R., Fan, K. Y., Pandey, P., Hu, Z., Yau, O., Teng, M., Wang, P., Li, T., Ashraf, M., & Singla, R. (2022) · Communications Medicine · doi:10.1038/s43856-022-00125-4 · Cited by 58 · see citing works
Boßelmann, C. M., Leu, C., & Lal, D. (2023) · Epilepsia · doi:10.1111/epi.17570 · Cited by 52 · see citing works
ChatGPT Performs on the Chinese National Medical Licensing ExaminationPreprint
Wang, X., Gong, Z., Wang, G., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2584079/v1 · Cited by 44 · see citing works
Observations on LLMs for Telecom Domain: Capabilities and Limitations
Soman, S., & Ranjani, H. G. (2023) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3639856.3639892 · Cited by 34 · see citing works
GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, D. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.04408 · Cited by 15 · see citing works
(2004) · CRC Press eBooks · doi:10.1201/9781482269390-98 · Cited by 3 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 2
- 2021, Stata 17 Base Reference Manual
- 10.48550/arXiv.2305.13102 · DOI
Cited by
Sorted by citations
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Large language models in medical and healthcare fields: applications, advances, and challenges
Wang, D., & Zhang, S. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10921-0 · Cited by 246 · see citing works
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234 · see citing works
“Students will increasingly explore this innovative AI-based technology, with an already growing literature highlighting its significance in health care education through personalized learning with immediate feedback and impressive performance in medical exams [9,18,40,55-60].”
An evaluation framework for clinical use of large language models in patient interaction tasks
Johri, S., Jeong, J., Tran, B. A., Schlessinger, D. I., Wongvibulsin, S., Barnes, L. A., Zhou, H.-Y., Cai, Z. R., Van Allen, E. M., Kim, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03328-5 · Cited by 201 · see citing works
Meyer, A., Riese, J., & Streichert, T. (2024) · JMIR Medical Education · doi:10.2196/50965 · Cited by 110 · see citing works
Tangadulrat, P., Sono, S., & Tangtrakulwanich, B. (2023) · JMIR Medical Education · doi:10.2196/50658 · Cited by 99 · see citing works
Jin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85 · see citing works
Shimizu, I., Kasai, H., Shikino, K., Araki, N., Takahashi, Z., Onodera, M., Kimura, Y., Tsukamoto, T., Yamauchi, K., Asahina, M., et al. (2023) · JMIR Medical Education · doi:10.2196/53466 · Cited by 81 · see citing works
Liu, M., Okuhara, T., Dai, Z., Huang, W., Gu, L., Okada, H., Furukawa, E., & Kiuchi, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105673 · Cited by 73 · see citing works
ChatGPT’s Response Consistency: A Study on Repeated Queries of Medical Examination Questions
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Bashiri Dezfouli, A., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024) · European Journal of Investigation in Health, Psychology and Education · doi:10.3390/ejihpe14030043 · Cited by 69 · see citing works
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41 · see citing works
Salman, I. M., Ameer, O. Z., Khanfar, M. A., & Hsieh, Y.-H. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1495378 · Cited by 36 · see citing works
Huang, S., Wen, C., Bai, X., Li, S., Wang, S., Wang, X., & Yang, D. (2025) · Journal of Medical Internet Research · doi:10.2196/68538 · Cited by 33 · see citing works
“0 would be more advantageous in aiding skill instruction [55].”
Morjaria, L., Burns, L., Bracken, K., Levinson, A. J., Ngo, Q. N., Lee, M., & Sibbald, M. (2024) · International Medical Education · doi:10.3390/ime3010004 · Cited by 33 · see citing works
Lin, Y., Luo, Z., Ye, Z., Zhong, N., Zhao, L., Zhang, L., Li, X., Chen, Z., & Chen, Y. (2025) · JMIR Medical Education · doi:10.2196/71125 · Cited by 31 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
“Some works have focused on specialized medical models (Luo et al., 2022; Singhal et al., 2023; Li et al., 2023; Singhal et al., 2025; Saab et al., 2024), and others on general models (Nori et al., 2023; Gilson et al., 2023; Takagi et al., 2023).”
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25 · see citing works
- Authors
- Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K.
- Affiliations
- Shimane University (JP)
- Shimane University Hospital (JP)
- Corresponding author
- Watari, Takashi
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-06-29
- Volume
- 9
- Pages
- e48002
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC10365615
- OpenAlex
- W4380685958
- Semantic Scholar
- 48948ab1c05260171337730c0e9b26ec7be49b46
- Cited by
- 271
- References
- 25
- Citations by year
- 2026: 27 · 2025: 87 · 2024: 125 · 2023: 31 · 2022: 1
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Shimane University
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- GPT-3.5, GPT-4, large language models, clinical reasoning, medical knowledge, model comparison, medical licensing examinations, medical education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
