CoChat

ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review

Open access
Kıyak, Yavuz Selim, Emekli, Emre · 2024 · Postgraduate Medical Journal
Cited by 137

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Abstract

ChatGPT’s role in creating multiple-choice questions (MCQs) is growing but the validity of these artificial-intelligence-generated questions is unclear. This literature review was conducted to address the urgent need for understanding the application of ChatGPT in generating MCQs for medical education. Following the database search and screening of 1920 studies, we found 23 relevant studies. We extracted the prompts for MCQ generation and assessed the validity evidence of MCQs. The findings showed that prompts varied, including referencing specific exam styles and adopting specific personas, which align with recommended prompt engineering tactics. The validity evidence covered various domains, showing mixed accuracy rates, with some studies indicating comparable quality to human-written questions, and others highlighting differences in difficulty and discrimination levels, alongside a significant reduction in question creation time. Despite its efficiency, we highlight the necessity of careful review and suggest a need for further research to optimize the use of ChatGPT in question generation. Main messages Ensure high-quality outputs by utilizing well-designed prompts; medical educators should prioritize the use of detailed, clear ChatGPT prompts when generating MCQs. Avoid using ChatGPT-generated MCQs directly in examinations without thorough review to prevent inaccuracies and ensure relevance. Leverage ChatGPT’s potential to streamline the test development process, enhancing efficiency without compromising quality.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/postmj/qgae065

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 14 references

Research works printed but not linked: 23

  1. Heng, 2023, The impact of chat generative pre-trained transformer (ChatGPT) on medical education, Postgrad Med J, 99, 1125, 10.1093/postmj/qgad058 · DOI · Paper page
  2. Sardesai, 2024, Utilizing generative conversational artificial intelligence to create simulated patient encounters: a pilot study for anaesthesia training, Postgrad Med J, 100, 237, 10.1093/postmj/qgad137 · DOI · Paper page
  3. Pugh, 2019, Plus ça change, plus c’est pareil: making a continued case for the use of MCQs in medical education, Med Teach, 41, 569, 10.1080/0142159X.2018.1505035 · DOI
  4. Cross, 2023, Transforming medical education: assessing the integration of ChatGPT into faculty workflows at a Caribbean medical school, Cureus, 15, e41399
  5. Tricco, 2017, Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide
  6. Downing, 2009, Assessment in Health Professions Education, 21, 10.4324/9780203880135-8 · doi:10.4324/9780203880135-8
  7. Sevgi, 2023, The role of an open artificial intelligence platform in modern neurosurgical education: a preliminary study, Neurosurg Rev, 46, 86, 10.1007/s10143-023-01998-2 · DOI
  8. Subbaramaiah, 2023, Impact of ChatGPT support on conducting regional anaesthesia examination, Indian J Anaesth, 67, 1025, 10.4103/ija.ija_841_23 · DOI
  9. Biswas, 2023, Passing is great: can ChatGPT conduct USMLE exams?, Ann Biomed Eng, 51, 1885, 10.1007/s10439-023-03224-y · DOI
  10. Meşe, 2024, Educating the next generation of radiologists: a comparative report of ChatGPT and e-learning resources, Diagn Interv Radiol, 30, 10.4274/dir.2023.232496 · DOI
  11. Zuckerman, 2023, ChatGPT for assessment writing, Med Teach., 45, 1224, 10.1080/0142159X.2023.2249239 · DOI · Paper page
  12. Ayub, 2023, Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis, Cureus, 15, e43717
  13. Divito, 2023, The tools of the future are the challenges of today: the use of ChatGPT in problem-based learning medical education, Med Teach., 46
  14. Koga, 2023, The potential of ChatGPT in medical education: focusing on USMLE preparation, Ann Biomed Eng, 51, 2123, 10.1007/s10439-023-03253-7 · DOI
  15. Hirosawa, Enhancing clinical reasoning with chat generative pre-trained transformer: a practical guide, Diagnosis (Berl), 11
  16. Kıyak, 2023, A ChatGPT prompt for writing case-based multiple-choice questions, Rev Esp Edu Med, 4, 98, 10.6018/edumed.587451 · DOI · Paper page
  17. Kıyak, 2024, ChatGPT for generating multiple-choice questions: evidence on the use of artificial intelligence in automatic item generation for a rational pharmacotherapy exam, Eur J Clin Pharmacol, 80, 10.1007/s00228-024-03649-x · DOI
  18. Han, 2024, An explorative assessment of ChatGPT as an aid in medical education: use it with caution, Med Teach, 46, 10.1080/0142159X.2023.2271159 · DOI · Paper page
  19. Totlis, 2023, The potential role of ChatGPT and artificial intelligence in anatomy education: a conversation with ChatGPT, Surg Radiol Anat, 45, 1321, 10.1007/s00276-023-03229-1 · DOI · Paper page
  20. Laupichler, 2024, Large language models in medical education: comparing ChatGPT- to human-generated exam questions, Acad Med, 99, 10.1097/ACM.0000000000005626 · DOI · Paper page
  21. Ngo, 2024, ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions, Acad Pathol, 11, 100099, 10.1016/j.acpath.2023.100099 · DOI
  22. Doggett, 2024, Response to: “ChatGPT for assessment writing”, Med Teach, 1
  23. Gierl, 2021, Advanced Methods in Automatic Item Generation, 10.4324/9781003025634 · DOI

Cited by

See all 137

Metadata

Authors
Kıyak, Y. S., & Emekli, E.
Venue
Postgraduate Medical Journal
Published
2024-06-06
Volume
100
Issue
1189
Pages
858-865
ISSN
  • 0032-5473 (ISSN-L)
  • 1469-0756
OpenAlex
W4399387113
Cited by
137
References
14
FWCI
10.98
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. https://app.cochat.ai/papers/10.1093/postmj/qgae065
The paper
Kıyak, Y. S., & Emekli, E. (2024). ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgraduate Medical Journal. https://doi.org/10.1093/postmj/qgae065

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.