Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan
Open accessAbstract
Purpose: This study aims to investigate the effects of language selection and translation quality on Generative Pre-trained Transformer-4 (GPT-4)'s response accuracy to expert-level diagnostic radiology questions. Materials and methods: We analyzed 146 diagnostic radiology questions from the Japan Radiology Board Examination (2020–2022), with consensus answers provided by two board-certified radiologists. The questions, originally in Japanese, were translated into English by GPT-4 and DeepL and into German and Chinese by GPT-4. Responses were generated by GPT-4 five times per question set per language. Response accuracy was compared between languages using one-way ANOVA with Bonferroni correction or the Mann–Whitney U test. Scores on selected English questions translated by a professional service and GPT-4 were also compared. The impact of translation quality on GPT-4’s performance was assessed by linear regression analysis. Results: The median scores (interquartile range) for the 146 questions were 70 (68–72) (Japanese), 89 (84.5–95.5) (GPT-4 English), 64 (55.5–67) (Chinese), and 56 (46.5–67.5) (German). Significant differences were found between Japanese and English (p = 0.002) and between Japanese and German (p = 0.022). The counts of correct responses across five attempts for each question were significantly associated with the quality of translation into English (GPT-4, DeepL) and German (GPT-4). In a subset of 31 questions where English translations yielded fewer correct responses than Japanese originals, professionally translated questions yielded better scores than those translated by GPT-4 (13 versus 8 points, p = 0.0079). Conclusion: GPT-4 exhibits higher accuracy when responding to English-translated questions compared to original Japanese questions, a trend not observed with German or Chinese translations. Accuracy improves with higher-quality English translations, underscoring the importance of high-quality translations in improving GPT-4’s response accuracy to diagnostic radiology questions in non-English languages and aiding non-native English speakers in obtaining accurate answers from large language models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Harigai et al found that GPT-4 achieved significantly higher accuracy on English-translated radiology questions (median 89 points) than on the original Japanese versions (median 70 points).
“Harigai et al [60] found that GPT-4 achieved significantly higher accuracy on English-translated radiology questions (median 89 points) than on the original Japanese versions (median 70 points), attributing this gap to the predominance of English in training data and to translation challenges specific to Japanese, including logographic script ambiguity and structural differences.”
From AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study (2026) · cites this paper as [60]
The Abstract Results report median scores of 89 for English-translated questions and 70 for the Japanese originals, with a significant difference (p = 0.002). The Discussion discusses English-language training data and Japanese translation difficulties as possible explanations, but hedges their causal role rather than establishing that they caused the gap. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLLM performance has been shown to vary substantially across languages.
“All feedback was generated and evaluated in Japanese, and LLM performance has been shown to vary substantially across languages.”
From AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study (2026) · cites this paper as [60]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
Roos, J., Kasapovic, A., Jansen, T., & Kaczmarczyk, R. (2023) · JMIR Medical Education · doi:10.2196/46482 · Cited by 98 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- [1]Alawida M, Mejri S, Mehmood A, et al. A Comprehensive study of chatgpt: advancements, limitations, and ethical considerations in natural language processing and cybersecurity. Information. 2023;14(8):462. 10.3390/info14080462. · DOI
- [4]Achiam J, Adler S, Agarwal S et al. OpenAI, GPT-4 Technical Report. 2023; arXiv:2303.08774. · arXiv
- [6]Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology. 2023;307: e230582. 10.1148/radiol.230582. 37191485 · DOI · Paper page
- [7]Panthier C, Gatinel D. Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: A novel approach to medical knowledge assessment. J Fr Ophtalmol. 2023;46:706–11. 10.1016/j.jfo.2023.05.006. 37537126 · DOI · Paper page
- [9]Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: Comparison Study. JMIR Med Educ. 2023;9:48002. 10.2196/48002. PMC10365615 37384388 · DOI · Paper page
- [10]Toyama Y, Harigai A, Abe M, et al. Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Jpn J Radiol. 2024;42:201–7. 10.1007/s11604-023-01491-2. 37792149 PMC10811006 · DOI
- [11]Wang H, Wu W, Dou Z, He L, Yang L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inform. 2023;177: 105173. 10.1016/j.ijmedinf.2023.105173. 37549499 · DOI
- [12]Wang YM, Shen HW, Chen TJ. Performance of ChatGPT on the pharmacist licensing examination in Taiwan. J Chin Med Assoc. 2023;86:653–8. 10.1097/JCMA.0000000000000942. 37227901 PMC12755457 · DOI · Paper page
- [13]Anderson LW, Krathwohl DR, Airasian PW et al. A Taxonomy for Learning, Teaching, and Assessing: a Revision of Bloom’s Taxonomy of Educational Objectives. Pearson. 2000.
- [16]Chiswick BR, Miller PW. Linguistic distance: a quantitative measure of the distance between English and other languages. J Multiling Multicult Dev. 2005;26:1–11. 10.1080/14790710508668395. · DOI · Paper page
- [17]Licht D, Gao C, Lam J, Guzman F, Diab M, Koehn P. Consistent Human Evaluation of Machine Translation across Language Pairs. Proc 15th Bienn Conf Assoc Mach Transl Am (Vol. 1: Res. Track). 2022; 309–321.
- [18]NLLB Team. Scaling neural machine translation to 200 languages. Nature. 2024;630(8018):841–6. 10.1038/s41586-024-07335-x. 38839963 PMC11208141 · DOI
- [19]Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930–40. 10.1038/s41591-023-02448-8. 37460753 · DOI · Paper page
Cited by
Sorted by citations
Kondo, T., Donkers, J., Nishigori, H., Rovers, S., & Heeneman, S. (2026) · JMIR Medical Education · doi:10.2196/90064 · Cited by 1 · see citing works
- Authors
- Harigai, A., Toyama, Y., Nagano, M., Abe, M., Kawabata, M., Li, L., Yamamura, J., & Takase, K.
- Affiliations
- Tohoku University (JP)
- Tohoku University Hospital (JP)
- Tohoku Medical and Pharmaceutical University Hospital (JP)
- Universität Hamburg (DE)
- Corresponding author
- Toyama, Yoshitaka
- Venue
- Japanese Journal of Radiology
- Published
- 2024-10-28
- Volume
- 43
- Issue
- 2
- Pages
- 319-329
- ISSN
- 1867-1071 (ISSN-L)
- 1867-108X
- PMC
- PMC11790683
- OpenAlex
- W4403827866
- Cited by
- 15
- References
- 3
- Citations by year
- 2026: 10 · 2025: 5
- FWCI
- 1.40
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Japanese Journal of Radiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, large language models, diagnostic radiology, language translation, translation quality, response accuracy, DeepL, non-native English speakers
- MeSH terms
- Clinical Competence, Educational Measurement, Humans, Japan, Language, Surveys and Questionnaires, Radiology, Translating, Translations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
