Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA Examinations
Open accessPublished version of a preprint
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Background: Large language models, such as ChatGPT by OpenAI, have demonstrated potential in various applications, including medical education. Previous studies have assessed ChatGPT’s performance in university or professional settings. However, the model’s potential in the context of standardized admission tests remains unexplored. Objective: This study evaluated ChatGPT’s performance on standardized admission tests in the United Kingdom, including the BioMedical Admissions Test (BMAT), Test of Mathematics for University Admission (TMUA), Law National Aptitude Test (LNAT), and Thinking Skills Assessment (TSA), to understand its potential as an innovative tool for education and test preparation. Methods: Recent public resources (2019-2022) were used to compile a data set of 509 questions from the BMAT, TMUA, LNAT, and TSA covering diverse topics in aptitude, scientific knowledge and applications, mathematical thinking and reasoning, critical thinking, problem-solving, reading comprehension, and logical reasoning. This evaluation assessed ChatGPT’s performance using the legacy GPT-3.5 model, focusing on multiple-choice questions for consistency. The model’s performance was analyzed based on question difficulty, the proportion of correct responses when aggregating exams from all years, and a comparison of test scores between papers of the same exam using binomial distribution and paired-sample (2-tailed) t tests. Results: The proportion of correct responses was significantly lower than incorrect ones in BMAT section 2 (P<.001) and TMUA paper 1 (P<.001) and paper 2 (P<.001). No significant differences were observed in BMAT section 1 (P=.2), TSA section 1 (P=.7), or LNAT papers 1 and 2, section A (P=.3). ChatGPT performed better in BMAT section 1 than section 2 (P=.047), with a maximum candidate ranking of 73% compared to a minimum of 1%. In the TMUA, it engaged with questions but had limited accuracy and no performance difference between papers (P=.6), with candidate rankings below 10%. In the LNAT, it demonstrated moderate success, especially in paper 2’s questions; however, student performance data were unavailable. TSA performance varied across years with generally moderate results and fluctuating candidate rankings. Similar trends were observed for easy to moderate difficulty questions (BMAT section 1, P=.3; BMAT section 2, P=.04; TMUA paper 1, P<.001; TMUA paper 2, P=.003; TSA section 1, P=.8; and LNAT papers 1 and 2, section A, P>.99) and hard to challenging ones (BMAT section 1, P=.7; BMAT section 2, P<.001; TMUA paper 1, P=.007; TMUA paper 2, P<.001; TSA section 1, P=.3; and LNAT papers 1 and 2, section A, P=.2). Conclusions: ChatGPT shows promise as a supplementary tool for subject areas and test formats that assess aptitude, problem-solving and critical thinking, and reading comprehension. However, its limitations in areas such as scientific and mathematical knowledge and applications highlight the need for continuous development and integration with conventional learning strategies in order to fully harness its potential.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Natural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823 · cited 1×
“One of the key advancements in this field is the development of large language models that are capable of generating human-like responses to user prompts [1].”
BackgroundChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
The future of medical education and research: Is ChatGPT a blessing or blight in disguise?
Arif, T. B., Munaf, U., & Ul-Haque, I. (2023) · Medical Education Online · doi:10.1080/10872981.2023.2181052 · Cited by 317
ChatGPT in Clinical Toxicology
Sabry Abdel-Messih, M., & Kamel Boulos, M. N. (2023) · JMIR Medical Education · doi:10.2196/46876 · Cited by 95
Research works printed but not linked: 2
- 10.1109/icaccs48705.2020.9074420 · DOI
- 10.48550/arXiv.2212.14882 · DOI · Paper page
Cited by
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
“Students will increasingly explore this innovative AI-based technology, with an already growing literature highlighting its significance in health care education through personalized learning with immediate feedback and impressive performance in medical exams [9,18,40,55-60].”
Kofinas, A. K., Tsay, C. H., & Pike, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13585 · Cited by 207
Miao, J., Thongprayoon, C., Suppadungsuk, S., Garcia Valencia, O. A., Qureshi, F., & Cheungpasitporn, W. (2023) · Clinics and Practice · doi:10.3390/clinpract14010008 · Cited by 111
Rossettini, G., Rodeghiero, L., Corradi, F., Cook, C., Pillastrini, P., Turolla, A., Castellini, G., Chiappinotto, S., Gianola, S., & Palese, A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05630-9 · Cited by 103
“Only two studies [34, 35] have attempted to address these limitations, identifying ChatGPT-3.”
BackgroundResultEvaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment
Lai, U. H., Wu, K. S., Hsu, T.-Y., & Kan, J. K. C. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1240915 · Cited by 67
Metadata
- Authors
- Giannos, P., & Delardas, O.
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2023-04-26
- Volume
- 9
- Pages
- e47737
- ISSN
- 2369-3762 (ISSN-L)
- OpenAlex
- W4367048218
- Cited by
- 95
- References
- 7
- FWCI
- 3.29
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA Examinations. https://app.cochat.ai/papers/10.2196/47737
- The paper
- Giannos, P., & Delardas, O. (2023). Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA Examinations. JMIR Medical Education. https://doi.org/10.2196/47737
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.