ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Media coverage suggests that ChatGPT can pass examinations based on multiple choice questions (MCQs), including those used to qualify doctors, lawyers, scientists etc. This poses a potential risk to the integrity of those examinations. We reviewed current research evidence regarding the performance of ChatGPT on MCQ-based examinations in higher education, along with recommendations for how educators might address challenges and benefits arising from these data. 53 studies were included, covering 114 question sets, totalling 49014 MCQs. Free versions of ChatGPT based upon GPT-3/3.5 performed better than random guessing but failed most examinations, performing significantly worse than the average human student. GPT-4 passed most examinations with a performance that was on a par with human subjects. These findings indicate that all summative MCQ-based assessments should be conducted under secure conditions with restricted access to ChatGPT and similar tools, particularly those examinations which assess foundational knowledge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“Students could ask ChatGPT to explain the answers it gives, and generate practice test questions (Bommineni et al. 2023; Cuthbert and Simpson 2023; Friederichs, Friederichs and März 2023; Fijačko et al. 2023; Kung et al. 2023) b. Teaching Assistant .”
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
The role of google scholar in evidence reviews and its applicability to grey literature searching
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138237 · Cited by 2,387
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
The publishing delay in scholarly peer-reviewed journals
Björk, B.-C., & Solomon, D. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.09.001 · Cited by 432 · cited 1×
“The average time to publish an academic paper is over a year (Björk and Solomon 2013) and so the majority of the research on ChatGPT is currently available only as preprints.”
Farazouli, A., Cerratto-Pargman, T., Bolander-Laksov, K., & McGrath, C. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2023.2241676 · Cited by 219
Cited by
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“These concerns around negative trends in academic integrity in online exams were already well advanced when ChatGPT was released, and a new and significant compounding factor was introduced [18–21].”
ChatGPT: The End of Online Exam Integrity?
Susnjak, T., & McIntosh, T. (2024) · Education Sciences · doi:10.3390/educsci14060656 · Cited by 189
Role of activity-based learning and ChatGPT on students' performance in education
Al Shloul, T., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100219 · Cited by 173
Evangelista, E. D. L. (2025) · Contemporary Educational Technology · doi:10.30935/cedtech/15775 · Cited by 124
“Similarly, Newton and Xiromeriti (2024) proposed evidence-based policies to protect the integrity of assessments and exams, ensuring that AI-generated content does not violate academic standards.”
Metadata
- Authors
- Newton, P., & Xiromeriti, M.
- Venue
- Assessment & Evaluation in Higher Education
- Publisher
- Informa UK Limited
- Published
- 2024-01-17
- Volume
- 49
- Issue
- 6
- Pages
- 781-798
- ISSN
- 0260-2938 (ISSN-L)
- 1469-297X
- OpenAlex
- W4390940237
- Cited by
- 70
- References
- 16
- FWCI
- 26.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review. https://app.cochat.ai/papers/10.1080/02602938.2023.2299059
- The paper
- Newton, P., & Xiromeriti, M. (2024). ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2023.2299059
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.