ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
ChatGPT’s role in creating multiple-choice questions (MCQs) is growing but the validity of these artificial-intelligence-generated questions is unclear. This literature review was conducted to address the urgent need for understanding the application of ChatGPT in generating MCQs for medical education. Following the database search and screening of 1920 studies, we found 23 relevant studies. We extracted the prompts for MCQ generation and assessed the validity evidence of MCQs. The findings showed that prompts varied, including referencing specific exam styles and adopting specific personas, which align with recommended prompt engineering tactics. The validity evidence covered various domains, showing mixed accuracy rates, with some studies indicating comparable quality to human-written questions, and others highlighting differences in difficulty and discrimination levels, alongside a significant reduction in question creation time. Despite its efficiency, we highlight the necessity of careful review and suggest a need for further research to optimize the use of ChatGPT in question generation. Main messages Ensure high-quality outputs by utilizing well-designed prompts; medical educators should prioritize the use of detailed, clear ChatGPT prompts when generating MCQs. Avoid using ChatGPT-generated MCQs directly in examinations without thorough review to prevent inaccuracies and ensure relevance. Leverage ChatGPT’s potential to streamline the test development process, enhancing efficiency without compromising quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G. (2023) · JMIR Medical Education · doi:10.2196/46885 · Cited by 840
Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study
Sallam, M., Salim, N. A., Barakat, M., Al-Mahzoum, K., Al-Tammemi, A. B., Malaeb, D., Hallit, R., & Hallit, S. (2023) · JMIR Medical Education · doi:10.2196/48254 · Cited by 234
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023) · PLOS ONE · doi:10.1371/journal.pone.0290691 · Cited by 195
Weidener, L., & Fischer, M. (2024) · JMIR Medical Education · doi:10.2196/51247 · Cited by 163
Eppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117
Harnessing the potential of large language models in medical education: promise and pitfalls
Benítez, T. M., Xu, Y., Boudreau, J. D., Kow, A. W. C., Bello, F., Van Phuoc, L., Wang, X., Sun, X., Leung, G. K.-K., Lan, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad252 · Cited by 100
E, K., S, P., R, G., R, K. L., A, B., M, G., T, O., S, R., V, R., H, M., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04752-w · Cited by 89
Indran, I. R., Paranthaman, P., Gupta, N., & Mustafa, N. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2294703 · Cited by 89
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2208731 · Cited by 76
Agarwal, M., Sharma, P., & Goswami, A. (2023) · Cureus · doi:10.7759/cureus.40977 · Cited by 72
Research works printed but not linked: 23
- Heng, 2023, The impact of chat generative pre-trained transformer (ChatGPT) on medical education, Postgrad Med J, 99, 1125, 10.1093/postmj/qgad058 · DOI · Paper page
- Sardesai, 2024, Utilizing generative conversational artificial intelligence to create simulated patient encounters: a pilot study for anaesthesia training, Postgrad Med J, 100, 237, 10.1093/postmj/qgad137 · DOI · Paper page
- Pugh, 2019, Plus ça change, plus c’est pareil: making a continued case for the use of MCQs in medical education, Med Teach, 41, 569, 10.1080/0142159X.2018.1505035 · DOI
- Cross, 2023, Transforming medical education: assessing the integration of ChatGPT into faculty workflows at a Caribbean medical school, Cureus, 15, e41399
- Tricco, 2017, Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
- Downing, 2009, Assessment in Health Professions Education, 21, 10.4324/9780203880135-8 · doi:10.4324/9780203880135-8
- Sevgi, 2023, The role of an open artificial intelligence platform in modern neurosurgical education: a preliminary study, Neurosurg Rev, 46, 86, 10.1007/s10143-023-01998-2 · DOI
- Subbaramaiah, 2023, Impact of ChatGPT support on conducting regional anaesthesia examination, Indian J Anaesth, 67, 1025, 10.4103/ija.ija_841_23 · DOI
- Biswas, 2023, Passing is great: can ChatGPT conduct USMLE exams?, Ann Biomed Eng, 51, 1885, 10.1007/s10439-023-03224-y · DOI
- Meşe, 2024, Educating the next generation of radiologists: a comparative report of ChatGPT and e-learning resources, Diagn Interv Radiol, 30, 10.4274/dir.2023.232496 · DOI
- Zuckerman, 2023, ChatGPT for assessment writing, Med Teach., 45, 1224, 10.1080/0142159X.2023.2249239 · DOI · Paper page
- Ayub, 2023, Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis, Cureus, 15, e43717
- Divito, 2023, The tools of the future are the challenges of today: the use of ChatGPT in problem-based learning medical education, Med Teach., 46
- Koga, 2023, The potential of ChatGPT in medical education: focusing on USMLE preparation, Ann Biomed Eng, 51, 2123, 10.1007/s10439-023-03253-7 · DOI
- Hirosawa, Enhancing clinical reasoning with chat generative pre-trained transformer: a practical guide, Diagnosis (Berl), 11
- Kıyak, 2023, A ChatGPT prompt for writing case-based multiple-choice questions, Rev Esp Edu Med, 4, 98, 10.6018/edumed.587451 · DOI · Paper page
- Kıyak, 2024, ChatGPT for generating multiple-choice questions: evidence on the use of artificial intelligence in automatic item generation for a rational pharmacotherapy exam, Eur J Clin Pharmacol, 80, 10.1007/s00228-024-03649-x · DOI
- Han, 2024, An explorative assessment of ChatGPT as an aid in medical education: use it with caution, Med Teach, 46, 10.1080/0142159X.2023.2271159 · DOI · Paper page
- Totlis, 2023, The potential role of ChatGPT and artificial intelligence in anatomy education: a conversation with ChatGPT, Surg Radiol Anat, 45, 1321, 10.1007/s00276-023-03229-1 · DOI · Paper page
- Laupichler, 2024, Large language models in medical education: comparing ChatGPT- to human-generated exam questions, Acad Med, 99, 10.1097/ACM.0000000000005626 · DOI · Paper page
- Ngo, 2024, ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions, Acad Pathol, 11, 100099, 10.1016/j.acpath.2023.100099 · DOI
- Doggett, 2024, Response to: “ChatGPT for assessment writing”, Med Teach, 1
- Gierl, 2021, Advanced Methods in Automatic Item Generation, 10.4324/9781003025634 · DOI
Cited by
Quantum leap in medical mentorship: exploring ChatGPT’s transition from textbooks to terabytes
Chokkakula, S., Chong, S., Yang, B., Jiang, H., Yu, J., Han, R., Attitalla, I. H., Yin, C., & Zhang, S. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1517981 · Cited by 12
“Its ability to rapidly craft consistent, realistic clinical vignettes of varying complexities has garnered significant attention in the field (113, 114).”
Fąferek, J., Kononowicz, A. A., Bogutska, N., Da Silva Domingues, V., Davydova, N., Frankowska, A., Iguacel, I., Mayer, A., Morin, L., Pavlyukovich, N., et al. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08006-9 · Cited by 10
“Former research has shown a promising potential of generative AI, often ChatGPT, in creating educational content, such as multiple-choice questions [55, 56] assessment items [57], feedback to learners [58], and VPs and case vignettes [32, 33].”
Lee, W., Chang, S., & Suh, J. (2025) · Education Sciences · doi:10.3390/educsci15111478 · Cited by 5
“Rapid generation of test questions and materials, reducing instructors’ workload and preparation time (Kıyak & Emekli, 2024; Cheung et al., 2023).”
Medical students' use of large language models: a national survey
Barr, A. A., Rozman, R. C., Liu, K., Pham, M., Klarenbach, Z., Chinna-Meyyappan, A., Hassan, A. Y., Zarychta, M., Ferri, O. E., Al-Khaz’Aly, A., et al. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106589 · Cited by 1
Metadata
- Authors
- Kıyak, Y. S., & Emekli, E.
- Venue
- Postgraduate Medical Journal
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-06-06
- Volume
- 100
- Issue
- 1189
- Pages
- 858-865
- ISSN
- 0032-5473 (ISSN-L)
- 1469-0756
- OpenAlex
- W4399387113
- Cited by
- 137
- References
- 14
- FWCI
- 10.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. https://app.cochat.ai/papers/10.1093/postmj/qgae065
- The paper
- Kıyak, Y. S., & Emekli, E. (2024). ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgraduate Medical Journal. https://doi.org/10.1093/postmj/qgae065
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.