This too shall pass: the performance of ChatGPT‐3.5, ChatGPT‐4 and New Bing in an Australian medical licensing examination
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
To the Editor: Following the release of the generative pre-trained transformer (GPT) ChatGPT in November 2022, a wide range of large language models (LLMs), including ChatGPT-3.5 (GPT-3-derived), ChatGPT-4 and New Bing (GPT-4-derived), have been made publicly available. There is suggestion that ChatGPT-4 outperforms ChatGPT-3.5 in answering questions from medical exams,1 but it is unknown whether GPT-4-derived LLMs consistently outperform GPT-3-derived LLMs. Despite being GPT-4-derived, New Bing was fine-tuned independently to ChatGPT-4. Fine-tuning provides additional training to LLMs for a specific task, such as responding to user queries, which can include human feedback on LLM-generated responses.2 New Bing and ChatGPT may differ in performance due to such variations in fine-tuning, and, at the time of testing (16–18 March 2023), only New Bing incorporated basic web searches. We tested ChatGPT-3.5, ChatGPT-4 and New Bing against all 50 publicly available Australian Medical Council licensing examination practice questions.3 The questions had a five-option multiple-choice format, and we copied each one in full to the LLM. We omitted images during testing because only text inputs were accepted. Two medical student investigators (OK and CG) decoded responses into answer options, which the Australian Medical Council website graded. Each algorithm was tested three times in independent sessions. ChatGPT-3.5 and ChatGPT-4 answered 49/50 questions in every session, and New Bing answered 49, 46 and 43 questions in separate sessions (Supporting Information). ChatGPT-4 provided 46/50 answer options that were identical in each session. Conversely, ChatGPT-3.5 gave only 37/50 identical answers, and New Bing, 33/50 (Box). ChatGPT-4 scored the highest mean (39.7; standard deviation [SD], 0.6), and the means for New Bing and ChatGPT-3 were 36.0 (SD, 2.6) and 33.0 (SD, 2.6) respectively. GPT-4-derived LLMs (ChatGPT-4 and New Bing) appear to exceed the medical multiple-choice performance of their GPT-3-derived predecessors (ChatGPT-3.5). Such models have previously been shown to outperform PubMedGPT,4 GPT-3 without fine-tuning, and InstructGPT.5 Despite improvements in accuracy, limitations in consistency remain. These results indicate the potential influence of fine-tuning and web-search access for LLM performance. Caution is advised when choosing whether to use an LLM, and which one to use, for a specific task, as it cannot be assumed that all same-generation LLMs perform equally. We thank Joshua Kovoor for providing editorial and statistical support to this piece. No relevant disclosures. Supplementary table Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Cited by
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103
“Others have reported mixed results when comparing the accuracy of multiple AI chatbots (e.g., ChatGPT-4 vs Microsoft Bing, ChatGPT-4 vs Google Bard) in several medical examinations tests [23–29].”
ResultJin, H. K., Lee, H. E., & Kim, E. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05944-8 · Cited by 85
“Among the studies, two were deemed to be at high risk [33, 34], and 18 at low risk [11, 13, 21, 27, 29–31 ,”
BackgroundChen, J., Zhu, L., Mou, W., Lin, A., Zeng, D., Qi, C., Liu, Z., Jiang, A., Tang, B., Shi, W., et al. (2024) · iMetaOmics · doi:10.1002/imo2.7 · Cited by 33
“Large ‐ language models such as ChatGPT [5], Google Bard [6], and New Bing [7] are perhaps the best ‐ explored generative AI tools in the medical field today.”
Global trends and hotspots of ChatGPT in medical research: a bibliometric and visualized study
Liu, L., Qu, S., Zhao, H., Kong, L., Xie, Z., Jiang, Z., & Zou, P. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1406842 · Cited by 14
Metadata
- Authors
- Kleinig, O., Gao, C., & Bacchi, S.
- Venue
- Medical Journal of Australia
- Publisher
- Wiley
- Published
- 2023-08
- Volume
- 219
- Issue
- 5
- Pages
- 237-237
- ISSN
- 0025-729X (ISSN-L)
- 1326-5377
- OpenAlex
- W4385476456
- Cited by
- 28
- References
- 2
- FWCI
- 0.93
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: This too shall pass: the performance of ChatGPT‐3.5, ChatGPT‐4 and New Bing in an Australian medical licensing examination. https://app.cochat.ai/papers/10.5694/mja2.52061
- The paper
- Kleinig, O., Gao, C., & Bacchi, S. (2023). This too shall pass: the performance of ChatGPT‐3.5, ChatGPT‐4 and New Bing in an Australian medical licensing examination. Medical Journal of Australia. https://doi.org/10.5694/mja2.52061
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.