Assessing the Performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination
Open accessAbstract
Purpose The purpose of this study was to evaluate the changes in capabilities between the Generative Pre-trained Transformer (GPT)-3.5 and GPT-4 versions of the large-scale language model ChatGPT within a Japanese medical context. Methods The study involved ChatGPT versions 3.5 and 4 responding to questions from the 112th Japanese National Nursing Examination (JNNE). The study comprised three analyses: correct answer rate and score rate calculations, comparisons between GPT-3.5 and GPT-4, and comparisons of correct answer rates for conversation questions. Results ChatGPT versions 3.5 and 4 responded to 237 out of 238 Japanese questions from the 112th JNNE. While GPT-3.5 achieved an overall accuracy rate of 59.9%, failing to meet the passing standards in compulsory and general/scenario-based questions, scoring 58.0% and 58.3%, respectively, GPT-4 had an accuracy rate of 79.7%, satisfying the passing standards by scoring 90.0% and 77.7%, respectively. For each problem type, GPT-4 showed a higher accuracy rate than GPT-3.5. Specifically, the accuracy rates for compulsory questions improved from 58.0% with GPT-3.5 to 90.0% with GPT-4. For general questions, the rates went from 64.6% with GPT-3.5 to 75.6% with GPT-4. In scenario-based questions, the accuracy rates improved substantially from 51.7% with GPT-3.5 to 80.0% with GPT-4. For conversation questions, GPT-3.5 had an accuracy rate of 73.3% and GPT-4 had an accuracy rate of 93.3%. Conclusions The GPT-4 version of ChatGPT displayed performance sufficient to pass the JNNE, significantly improving from GPT-3.5. This suggests specialized medical training could make such models beneficial in Japanese clinical settings, aiding decision-making. However, user awareness and training are crucial, given potential inaccuracies in ChatGPT's responses. Hence, responsible usage with an understanding of its capabilities and limitations is vital to best support healthcare professionals and patients.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Necessary oversight is maintained for complex scenarios when using GenAI.
“while maintaining necessary oversight for complex scenarios (Hallas, 2023; Kaneda et al., 2023; Luo et al., 2023; Ni et al., 2024).”
From Patterns, advances, and gaps in using ChatGPT and similar technologies in nursing education: A PAGER scoping review (2025) · cites this paper as Kaneda et al., 2023
These tools demonstrated value in creating diverse assessment formats across multiple studies.
“these tools demonstrated value in creating diverse assessment formats across mul tiple studies”
From Patterns, advances, and gaps in using ChatGPT and similar technologies in nursing education: A PAGER scoping review (2025) · cites this paper as Kaneda et al., 2023
The Materials and methods section says existing Japanese Nursing Examination questions were entered into GPT-3.5 and GPT-4, which generated answers. I searched for passages about creating assessment formats or generating questions; our copy reports performance on existing question types, not value in creating assessments across studies. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 17 · Cited by 45
Research works printed but not linked: 14
- [1]Ten ways artificial intelligence will transform primary care J Gen Intern Med Lin SY Mahoney MR Sinsky CA 1626 1630 34 2019 31090027 PMC6667610
- [2]Artificial intelligence in healthcare Nat Biomed Eng Yu KH Beam AL Kohane IS 719 731 2 2018 31015651
- [4]Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer NPJ Digit Med Nagpal K Foote D Liu Y 48 2 2019 31304394 PMC6555810
- [5]A deep learning system for differential diagnosis of skin diseases Nat Med Liu Y Jain A Eng C 900 908 26 2020 32424212
- [8]Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models PLOS Digit Health Kung TH Cheatham M Medenilla A 0 2 2023 PMC9931230 36812645
- [9]ChatGPT performs on the Chinese National Medical Licensing Examination Res Sq Wang X Gong Z Wang G 2023 2023 37581690
- [14]Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study JMIR Med Educ Takagi S Watari T Erabi A Sakaguchi K 0 9 2023 PMC10365615 37384388
- [17]Burnout in healthcare workers: prevalence, impact and preventative strategies Local Reg Anesth De Hert S 171 183 13 2020 33149664 PMC7604257
- [18]AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation BMJ Open Sport Exerc Med Anderson N Belavy DL Perle SM Hendricks S Hespanhol L Verhagen E Memon AR 0 9 2023 PMC9936276 36816423
- [20]Are the issues pointed out by ChatGPT can be applied to Japan? Examining the reasons behind high COVID-19 excess deaths in Japan New Microbes New Infect Kaneda Y Tsubokura M Ozaki A Saito H Tanimoto T 101116 53 2023 37025464 PMC10052853
- [21]Learning curves in machine learning Encyclopedia of Machine Learning Perlich C 577 580 Boston, MA Springer 2011 https://link.springer.com/referenceworkentry/10.1007/978-0-387-30164-8_452#citeas · DOI
- [22]In the era of prominent AI, what role will physicians be expected to play? QJM Kaneda Y 2023 37216897
- [24]Is there a place for a holistic approach in surgical training? Arab J Urol Atayoglu T Buchholz N Atayoglu AG Caliskan M 21 24 12 2014 26019916 PMC4434609
- [25]Large language models are human-level prompt engineers arXiv:2211.01910 Y Zhou AI Muresanu Z Han Paster K Pitis S Chan H Ba J 2022 · arXiv
- Authors
- Kaneda, Y., Takahashi, R., Kaneda, U., Akashima, S., Okita, H., Misaki, S., Yamashiro, A., Ozaki, A., & Tanimoto, T.
- Affiliations
- Dokkyo University (JP)
- Corresponding author
- Kaneda, Yudai
- Venue
- Cureus
- Published
- 2023-08-03
- Volume
- 15
- Issue
- 8
- Pages
- e42924-e42924
- ISSN
- 2168-8184 (ISSN-L)
- PMC
- PMC10475149
- OpenAlex
- W4385597618
- Cited by
- 45
- References
- 17
- Citations by year
- 2026: 5 · 2025: 14 · 2024: 21 · 2023: 5
- FWCI
- 1.80
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, GPT-3.5, ChatGPT, large language models, nursing licensure examination, model evaluation, scenario-based questions, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
