CoChat

This too shall pass: the performance of ChatGPT‐3.5, ChatGPT‐4 and New Bing in an Australian medical licensing examination

Kleinig, Oliver, Gao, Christina, Bacchi, Stephen · 2023 · Medical Journal of Australia
Cited by 28

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

To the Editor: Following the release of the generative pre-trained transformer (GPT) ChatGPT in November 2022, a wide range of large language models (LLMs), including ChatGPT-3.5 (GPT-3-derived), ChatGPT-4 and New Bing (GPT-4-derived), have been made publicly available. There is suggestion that ChatGPT-4 outperforms ChatGPT-3.5 in answering questions from medical exams,1 but it is unknown whether GPT-4-derived LLMs consistently outperform GPT-3-derived LLMs. Despite being GPT-4-derived, New Bing was fine-tuned independently to ChatGPT-4. Fine-tuning provides additional training to LLMs for a specific task, such as responding to user queries, which can include human feedback on LLM-generated responses.2 New Bing and ChatGPT may differ in performance due to such variations in fine-tuning, and, at the time of testing (16–18 March 2023), only New Bing incorporated basic web searches. We tested ChatGPT-3.5, ChatGPT-4 and New Bing against all 50 publicly available Australian Medical Council licensing examination practice questions.3 The questions had a five-option multiple-choice format, and we copied each one in full to the LLM. We omitted images during testing because only text inputs were accepted. Two medical student investigators (OK and CG) decoded responses into answer options, which the Australian Medical Council website graded. Each algorithm was tested three times in independent sessions. ChatGPT-3.5 and ChatGPT-4 answered 49/50 questions in every session, and New Bing answered 49, 46 and 43 questions in separate sessions (Supporting Information). ChatGPT-4 provided 46/50 answer options that were identical in each session. Conversely, ChatGPT-3.5 gave only 37/50 identical answers, and New Bing, 33/50 (Box). ChatGPT-4 scored the highest mean (39.7; standard deviation [SD], 0.6), and the means for New Bing and ChatGPT-3 were 36.0 (SD, 2.6) and 33.0 (SD, 2.6) respectively. GPT-4-derived LLMs (ChatGPT-4 and New Bing) appear to exceed the medical multiple-choice performance of their GPT-3-derived predecessors (ChatGPT-3.5). Such models have previously been shown to outperform PubMedGPT,4 GPT-3 without fine-tuning, and InstructGPT.5 Despite improvements in accuracy, limitations in consistency remain. These results indicate the potential influence of fine-tuning and web-search access for LLM performance. Caution is advised when choosing whether to use an LLM, and which one to use, for a specific task, as it cannot be assumed that all same-generation LLMs perform equally. We thank Joshua Kovoor for providing editorial and statistical support to this piece. No relevant disclosures. Supplementary table Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.5694/mja2.52061

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Cited by

Metadata

Authors
Kleinig, O., Gao, C., & Bacchi, S.
Venue
Medical Journal of Australia
Publisher
Wiley
Published
2023-08
Volume
219
Issue
5
Pages
237-237
ISSN
  • 0025-729X (ISSN-L)
  • 1326-5377
OpenAlex
W4385476456
Cited by
28
References
2
FWCI
0.93
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.