CoChat

Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis

Open access
Levin, Gabriel, Horesh, Nir, Brezinov, Yoav, Meyer, Raanan · 2024 · BJOG: An International Journal of Obstetrics & Gynaecology
Cited by 90

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

The use of ChatGPT, an artificial intelligence (AI) language model, has been described in various scientific and medical applications.1 With its human-like conversation capacity and large quantity of data, ChatGPT has the potential to become an important medical education tool.2 ChatGPT's performance in different medical knowledge examinations has been recently studied in various medical disciplines. However, reported rates of correct answers vary dramatically across different examinations and medical fields.3, 4 We aimed to conduct a meta-analysis of studies reporting ChatGPT's performance in medical examinations with multiple-choice questions. PubMed, Scopus and Web of Science websites were searched for relevant articles from the inception of these databases to 2 June 2023, by use of the word “ChatGPT” (no equivalent Mesh term exists). We manually reviewed every article title and abstract. In case an abstract was not available, we accessed the abstract on the journal's web site. We included all peer-reviewed articles, assessing the performance (number of right answers/number of questions) of ChatGPT in multiple-choice questions in the field of medicine. Exclusion criteria were: (i) performance of ChatGPT not in English language; (ii) a study evaluating performance of ChatGPT in a setting other than multiple-choice question (e.g. open questions, frequently asked questions); and (iii) a study not reporting ChatGPT version 3.5. All review stages were conducted independently by two reviewers (RM and GL). Disagreements were resolved by discussion with a third reviewer (YB). Data were extracted from each included study without modifications and a database was constructed containing the study field of medicine (e.g. dermatology, plastic surgery etc.), cohort size (number of questions answered), number of correct questions, ChatGPT performance (number of correct answers/numbers of answered questions) and 95% CI for performance rate. We used MedCalc Statistical Software version 19.2.6 (MedCalc Software bv, Ostend, Belgium) and OpenMeta[Analyst] for the analysis. The process of literature search and article selection is presented in the Supplementary material. Finally, a total of 19 articles were included in the analysis. Two articles (11%) studied Plastic Surgery examinations, two (11%) studied the United Stated Medical Licensing Examinations (USMLE) and two (11%) studied anaesthesia examinations. All other publications studied different medical fields (Table 1). The median number of questions per examination was 242, ranging from 20 in medical physiology examinations to 3705 in anaesthesia examinations (mean 524.5 ± 847.2 standard deviation, Figure 1, Table 1). Overall performance of ChatGPT ranged from 40% in the biomedical admission test to 100% in a diabetes knowledge questionnaire. The mean performance of ChatGPT was 61.1% (95% CI 56.1%–66.0%. As the literature regarding the performance of ChatGPT in medical education is mounting, summarising the current performance of ChatGPT in medical examinations can provide an insight into the present and future of AI in medical education. Our meta-analysis suggests that ChatGPT correctly answered the majority of multiple-choice questions in medical examinations and demonstrated a performance of approximately a passing grade. Medical education, test preparation services, and medical examinations form a large industrial market.5 Currently, it seems that the use of ChatGPT for examination preparation should be prudent, and preparations for examinations using this platform should be done with the proper caution. A correct response rate of more than 95% may allow ChatGPT to become a reliable educational tool,6 and it is unknown whether future versions will reach this maturity level, as the training data set is not specifically developed with a focus on medical education. Our limitations are the inclusion of only ChatGPT version 3.5 studies, the heterogeneity of the included studies with unreported numbers of choices per examination, and the possible overestimation of the tool performance. Future meta-analyses may include future versions of AI chatbots to provide updated understanding of their role in medical education. None. This research received no external funding. None declared. No ethical approval was needed by the institutional review board as this study analyzed only published available public data and no human patients data was sused. The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions. Data S1. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/1471-0528.17641

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 90

Metadata

Authors
Levin, G., Horesh, N., Brezinov, Y., & Meyer, R.
Venue
BJOG: An International Journal of Obstetrics & Gynaecology
Publisher
Wiley
Published
2023-08-21
Volume
131
Issue
3
Pages
378-380
ISSN
  • 0306-5456 (ISSN-L)
  • 1470-0328
  • 1471-0528
OpenAlex
W4386023761
Cited by
90
References
4
FWCI
1.47
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.