ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom)
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Introduction: Large language models, in particular ChatGPT, have showcased remarkable language processing capabilities. Given the substantial workload of university medical staff, this study aims to assess the quality of multiple-choice questions (MCQs) produced by ChatGPT for use in graduate medical examinations, compared to questions written by university professoriate staffs based on standard medical textbooks. Methods: 50 MCQs were generated by ChatGPT with reference to two standard undergraduate medical textbooks (Harrison’s, and Bailey & Love’s). Another 50 MCQs were drafted by two university professoriate staff using the same medical textbooks. All 100 MCQ were individually numbered, randomized and sent to five independent international assessors for MCQ quality assessment using a standardized assessment score on five assessment domains, namely, appropriateness of the question, clarity and specificity, relevance, discriminative power of alternatives, and suitability for medical graduate examination. Results: The total time required for ChatGPT to create the 50 questions was 20 minutes 25 seconds, while it took two human examiners a total of 211 minutes 33 seconds to draft the 50 questions. When a comparison of the mean score was made between the questions constructed by A.I. with those drafted by humans, only in the relevance domain that the A.I. was inferior to humans (A.I.: 7.56 +/- 0.94 vs human: 7.88 +/- 0.52; p = 0.04). There was no significant difference in question quality between questions drafted by A.I. versus humans, in the total assessment score as well as in other domains. Questions generated by A.I. yielded a wider range of scores, while those created by humans were consistent and within a narrower range. Conclusion: ChatGPT has the potential to generate comparable-quality MCQs for medical graduate examinations within a significantly shorter time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, while LLMs can quickly generate MCQs, their quality and difficulty often do not meet expectations.”
From MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback (2025) · cites this paper as Cheung et al. (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This aligns with the previously mentioned finding of LLM-generated questions being less relevant than human-created ones [4].”
From Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine (2024) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Cheung et al. reported that manual creation took 10 times longer than automatic generation [4].”
From Docimological Quality Analysis of LLM-Generated Multiple Choice Questions in Computer Science and Medicine (2024) · cites this paper as [4]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Artificial Intelligence in Education: A Review
Chen, L., Chen, P., & Lin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2988510 · Cited by 4,133
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Haenlein, M., & Kaplan, A. (2019) · California Management Review · doi:10.1177/0008125619864925 · Cited by 2,655
Assessment in medical education
Epstein, R. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmra054784 · Cited by 1,872
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
Harrison's Principles of Internal Medicine
Bigby, J. (1988) · Archives of Dermatology · doi:10.1001/archderm.1988.01670020093028 · Cited by 1,234
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185
Bloom’s taxonomy of cognitive learning objectives
Adams, N. E. (2015) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.103.3.010 · Cited by 1,030
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973
Open artificial intelligence platforms in nursing education: Tools for academic progress or abuse?
O’Connor, S., & ChatGPT (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2022.103537 · Cited by 585
Developing and Validating Multiple-choice Test Items
Haladyna, T. M. (2004) · Routledge eBooks · doi:10.4324/9780203825945 · Cited by 584
Bailey & Love's Short Practice of Surgery
Reed, M. (1997) · PubMed Central · Cited by 438
Bailey & Love's Short Practice of Surgery.
Mwr, R. (1997) · Annals of The Royal College of Surgeons of England · Cited by 404
The impact of site-specific digital histology signatures on deep learning model accuracy and bias
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24698-1 · Cited by 330
Rao, S. K., Kimball, A. B., Lehrhoff, S. R., Hidrue, M. K., Colton, D. G., Ferris, T. G., & Torchiana, D. F. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001461 · Cited by 255
Quality improvement in medical education: current state and future directions
Wong, B. M., Levinson, W., & Shojania, K. G. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04154.x · Cited by 235
Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017) · Review of Educational Research · doi:10.3102/0034654317726529 · Cited by 219
Item analysis of multiple choice questions: A quality assurance test for an assessment tool
Kumar, D., Jaipurkar, R., Shekhar, A., Sikri, G., & Srinivas, V. (2021) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2020.11.007 · Cited by 109
Multiple-Choice Item Distractor Development Using Topic Modeling Approaches
Shin, J., Guo, Q., & Gierl, M. J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00825 · Cited by 79
Burnout among Academic Clinicians as It Correlates with Workload and Demographic Variables
Nassar, A. K., Reid, S. E., Kahnamoui, K., Tuma, F., Waheed, A., & McConnell, M. (2020) · Behavioral Sciences · doi:10.3390/bs10060094 · Cited by 62
Charan, J., Vegada, B., Shukla, A., Khilnani, A., & Desai, C. (2016) · Indian Journal of Pharmacology · doi:10.4103/0253-7613.190757 · Cited by 58
An investigation into the optimal number of distractors in single-best answer exams
Kilgour, J. M., & Tayyaba, S. (2015) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9652-7 · Cited by 51
Scotti, V. (2020) · IEEE Instrumentation & Measurement Magazine · doi:10.1109/mim.2020.9082795 · Cited by 50
The future of medical education
Yeoh, K. (2019) · Singapore Medical Journal · doi:10.11622/smedj.2019003 · Cited by 47
Evaluation of the quality of multiple-choice questions according to the students’ academic level
Iñarrairaegui, M., Fernández-Ros, N., Lucena, F., Landecho, M. F., García, N., Quiroga, J. A., & Herrero, J. I. (2022) · BMC Medical Education · doi:10.1186/s12909-022-03844-3 · Cited by 27
Written-Based Progress Testing: A Scoping Review.
Dion, V., St‐Onge, C., Bartman, I., Touchie, C., & Pugh, D. (2022) · PubMed · doi:10.1097/acm.0000000000004507 · Cited by 16
Davis, R., & Schlamminger, S. (2020) · IEEE Instrumentation & Measurement Magazine · doi:10.1109/mim.2020.9082793 · Cited by 6
Research works printed but not linked: 1
- Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, et al. Emergent abilities of large language models. arXiv preprint arXiv:220607682. 2022.
Cited by
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
A Scoping Review on How Generative Artificial Intelligence Transforms Assessment in Higher Education
Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00468-z · Cited by 493
“Overall, teachers should teach and assess the skills needed in an AI-influenced landscape (Chan, 2023; Cheung et al., 2023; Farazouli et al., 2023; Stutz et al., 2023).”
InfluentialMoulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105474 · Cited by 244
Peláez-Sánchez, I. C., Velarde-Camaqui, D., & Glasserman-Morales, L. D. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1392091 · Cited by 146
ChatGPT in the higher education: A systematic literature review and research challenges
Baig, M. I., & Yadegaridehkordi, E. (2024) · International Journal of Educational Research · doi:10.1016/j.ijer.2024.102411 · Cited by 208
Metadata
- Authors
- Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H.
- Affiliations
- University of Hong Kong (HK)
- University of Edinburgh (GB)
- National University Cancer Institute, Singapore (SG)
- Ollscoil na Gaillimhe – University of Galway (IE)
- Corresponding author
- Co, Michael Tiong-Hong
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-08-29
- Volume
- 18
- Issue
- 8
- Pages
- e0290691
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC10464959
- OpenAlex
- W4386249608
- Cited by
- 195
- References
- 28
- Citations by year
- 2026: 61 · 2025: 71 · 2024: 54 · 2023: 9
- FWCI
- 7.68
- Type
- journal article
- Language
- en
- Update notices
- correction (2025-06-27)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, large language models, multiple-choice questions, medical education, medical textbooks
- MeSH terms
- Artificial Intelligence, Education, Medical, Graduate, Educational Measurement, United Kingdom, Hong Kong, Humans, Ireland, Prospective Studies, Singapore
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


