CoChat

Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination

Open access
Rosoł, Maciej, Gąsior, Jakub S., Łaba, Jonasz, Korzeniewski, Kacper, Młyńczak, Marcel · 2023 · Scientific Reports

Published version of a preprint

Cited by 157

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

The study aimed to evaluate the performance of two Large Language Models (LLMs): ChatGPT (based on GPT-3.5) and GPT-4 with two temperature parameter values, on the Polish Medical Final Examination (MFE). The models were tested on three editions of the MFE from: Spring 2022, Autumn 2022, and Spring 2023 in two language versions—English and Polish. The accuracies of both models were compared and the relationships between the correctness of answers with the answer’s metrics were investigated. The study demonstrated that GPT-4 outperformed GPT-3.5 in all three examinations regardless of the language used. GPT-4 achieved mean accuracies of 79.7% for both Polish and English versions, passing all MFE versions. GPT-3.5 had mean accuracies of 54.8% for Polish and 60.3% for English, passing none and 2 of 3 Polish versions for temperature parameter equal to 0 and 1 respectively while passing all English versions regardless of the temperature parameter value. GPT-4 score was mostly lower than the average score of a medical student. There was a statistically significant correlation between the correctness of the answers and the index of difficulty for both models. The overall accuracy of both models was still suboptimal and worse than the average for medical students. This emphasizes the need for further improvements in LLMs before they can be reliably deployed in medical settings. These findings suggest an increasing potential for the usage of LLMs in terms of medical education.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-023-46995-z

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 20

  1. Montejo-Ráez, A. & Jiménez-Zafra, S. M. Current approaches and applications in natural language processing. Appl. Sci. https://doi.org/10.3390/app12104859 (2022). · DOI
  2. Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y. & Radev, D. Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations (Springer, 2023).
  3. Gencer, A. & Aydin, S. Can ChatGPT pass the thoracic surgery exam?. Am. J. Med. Sci. https://doi.org/10.1016/j.amjms.2023.08.001 (2023). · DOI · Paper page
  4. Beam, K. et al. Performance of a large language model on practice questions for the neonatal board examination. JAMA Pediatr. https://doi.org/10.1001/jamapediatrics.2023.2373 (2023). · DOI · Paper page
  5. Wang, X. et al. ChatGPT performs on the Chinese national medical licensing examination. https://doi.org/10.21203/rs.3.rs-2584079/v1 (2023). · DOI
  6. Fang, C. et al. How does ChatGPT4 preform on non-English national medical licensing examination? An evaluation in Chinese language. MedRxiv https://doi.org/10.1101/2023.05.03.23289443 (2023). · DOI
  7. Rao, A. et al. Evaluating ChatGPT as an adjunct for radiologic decision-making. MedRxiv https://doi.org/10.1101/2023.02.02.23285399 (2023). · DOI · Paper page
  8. BaoZhi, S. & Yuhong, Z. Medical curricula in China and the USA: A comparative study. Med. Teach. 25, 422–427 (2003). · doi:10.1080/0142159031000136725
  9. Schulte, K. L. et al. Credentialing in interventional therapy in Europe: Comparison of curricula including endovascular therapy of arterial diseases. VASA Eur. J. Vasc. Med. https://doi.org/10.1024/0301-1526/a000618 (2017). · doi:10.1024/0301-1526/a000618
  10. Zavlin, D., Jubbal, K. T., Noé, J. G. & Gansbacher, B. A comparison of medical education in Germany and the United States: From applying to medical school to the beginnings of residency. GMS Germ. Med. Sci. 15, 1–10 (2017).
  11. LeCroy, C. W., & Krysik, J. Understanding and Interpreting Effect Size Measures. Soc. Work Res. 31(4), 243–248 (2007). · DOI
  12. Mihalache, A., Popovic, M. M. & Muni, R. H. Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment. JAMA Ophthalmol. https://doi.org/10.1001/jamaophthalmol.2023.1144 (2023). · DOI · Paper page
  13. Kormilitzin, A., Vaci, N., Liu, Q. & Nevado-Holgado, A. Med7: A transferable clinical natural language processing model for electronic health records. Artif. Intell. Med. 118, 102086 (2021). · DOI
  14. Varshney, D., Zafar, A., Behera, N. K. & Ekbal, A. Knowledge graph assisted end-to-end medical dialog generation. Artif. Intell. Med. 139, 102535 (2023). · DOI · Paper page
  15. Ayers, J. W. et al. Evaluating artificial intelligence responses to public health questions. JAMA Netw. Open 6, e2317517 (2023). · DOI · Paper page
  16. Ahn, S. The impending impacts of large language models on medical education. Korean J. Med. Educ. 35, 103 (2023). · DOI · Paper page
  17. Kraljevic, Z. et al. Multi-domain clinical natural language processing with MedCAT: The medical concept annotation toolkit. Artif. Intell. Med. 117, 102083 (2021). · DOI · Paper page
  18. Cascella, M., Montomoli, J., Bellini, V. & Bignami, E. Evaluating the feasibility of ChatGPT in healthcare: An analysis of multiple clinical and research scenarios. J. Med. Syst. 47, 33 (2023). · DOI · Paper page
  19. Short, C. E. & Short, J. C. The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creation. J. Bus. Ventur. Insights 19, e00388 (2023). · DOI · Paper page
  20. Cheng, K. et al. Artificial intelligence in sports medicine: Could GPT-4 make human doctors obsolete?. Ann. Biomed. Eng. https://doi.org/10.1007/s10439-023-03213-1 (2023). · DOI

Cited by

See all 157

Metadata

Authors
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M.
Venue
Scientific Reports
Published
2023-11-22
Volume
13
Issue
1
Pages
20512-20512
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4388896754
Cited by
157
References
25
FWCI
5.40
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. https://app.cochat.ai/papers/10.1038/s41598-023-46995-z
The paper
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023). Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Scientific Reports. https://doi.org/10.1038/s41598-023-46995-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.