CoChat
Sign Up
Cited by 32Artificial Intelligence in Healthcare and EducationClinical Reasoning and Diagnostic SkillsInnovations in Medical Education

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Open access
Hudon, Alexandre, Kiepura, Barnabé, Pelletier, Myriam, Phan, Véronique·2024·JMIR Medical Education
DOI 10.2196/54067PMC PMC11007379
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)–based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials. Objective: The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion. Methods: This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs. Results: A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario ( P =.84), clinical questions ( P =.99), and expert opinion ( P =.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content. Conclusions: This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/54067

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Fields such as Biology, Business, and Medicine remain underexplored, largely due to challenges in adapting general-purpose GenAI models to specialized or niche contexts.

    “In contrast, fields such as Biology (Collins et al., 2024), Business (Milić et al., 2024; Netland et al. 2025), and Medicine (Hudon et al., 2024) remain underexplored, largely due to challenges in adapting general-purpose GenAI models to specialized or niche contexts.”

    From Pedagogical Applications of Generative AI in Higher Education: A Systematic Review of the Field (2025) · cites this paper as Hudon et al., 2024

    The Discussion’s “Comparison With Prior Work” says ChatGPT applications in psychiatry medical education are limited, supporting an underexplored niche within medicine. The Introduction and Conclusions identify a gap in research on ChatGPT-generated script concordance tests and note weaknesses in the generated material. The paper does not establish that medicine broadly remains underexplored or that difficulty adapting general-purpose models is the main reason. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 37 · Cited by 32

Research works printed but not linked: 26

  1. [3]Adams NE Bloom’s taxonomy of cognitive learning objectives J Med Libr Assoc 07 2015 103 3 152 153 doi 10.3163/1536-5050.103.3.010 Medline 26213509 PMC4511057 · DOI · Paper page
  2. [4]Heist BS Gonzalo JD Durning S Torre D Elnicki DM Exploring clinical reasoning strategies and test-taking behaviors during clinical vignette style multiple-choice examinations: a mixed methods study J Grad Med Educ Dec 2014 6 4 709 714 doi 10.4300/JGME-D-14-00176.1 Medline 26140123 PMC4477567 · DOI · Paper page
  3. [5]Butler AC Multiple-choice testing in education: are the best practices for assessment also good for learning? J Appl Res Mem Cogn 07 2018 7 3 323 331 doi 10.1016/j.jarmac.2018.07.002 · DOI · Paper page
  4. [6]Charlin B Roy L Brailovsky C Goulet F van der Vleuten C The script concordance test: a tool to assess the reflective clinician Teach Learn Med 2000 12 4 189 195 doi 10.1207/S15328015TLM1204_5 Medline 11273368 · DOI · Paper page
  5. [7]Giet D Massart V Gagnon R Charlin B Le test de concordance de script en 20 questions. Twenty questions on script concordance tests [Article in French] Pédagogie Médicale Feb 4 2013 14 1 39 48 doi 10.1051/pmed/2012026 · DOI
  6. [8]Petrucci AM Nouh T Boutros M Gagnon R Meterissian SH Assessing clinical judgment using the script concordance test: the importance of using specialty-specific experts to develop the scoring key Am J Surg Feb 2013 205 2 137 140 doi 10.1016/j.amjsurg.2012.09.002 Medline 23246286 · DOI
  7. [9]Fournier JP Demeester A Charlin B Script concordance tests: guidelines for construction BMC Med Inform Decis Mak 05 6 2008 8 18 doi 10.1186/1472-6947-8-18 Medline 18460199 PMC2427021 · DOI
  8. [10]Dory V Gagnon R Vanpee D Charlin B How to construct and implement script concordance tests: insights from a systematic review Med Educ Jun 2012 46 6 552 563 doi 10.1111/j.1365-2923.2011.04211.x Medline 22626047 · DOI
  9. [11]Leclerc AA Nguyen LHP Charlin B Lubarsky S Ayad T Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology Can J Surg 05 26 2021 64 3 E317 E323 doi 10.1503/cjs.014919 Medline 34038060 PMC8327993 · DOI
  10. [12]See KC Tan KL Lim TK The script concordance test for clinical reasoning: re-examining its utility and potential weakness Med Educ Nov 2014 48 11 1069 1077 doi 10.1111/medu.12514 Medline 25307634 · DOI
  11. [13]Kazour F Richa S Zoghbi M El-Hage W Haddad FG Using the script concordance test to evaluate clinical reasoning skills in psychiatry Acad Psychiatry Feb 2017 41 1 86 90 doi 10.1007/s40596-016-0539-6 Medline 27178278 · DOI
  12. [14]Charlin B Gagnon R Lubarsky S et al Assessment in the context of uncertainty using the script concordance test: more meaning for scores Teach Learn Med 07 2010 22 3 180 186 doi 10.1080/10401334.2010.488197 Medline 20563937 · DOI
  13. [15]Lineberry M Kreiter CD Bordage G Threats to validity in the use and interpretation of script concordance test scores Med Educ Dec 2013 47 12 1175 1183 doi 10.1111/medu.12283 Medline 24206151 · DOI
  14. [16]Walters M Alonge T Zeller M Impact of COVID-19 on medical education: perspectives from students Acad Med Mar 1 2022 97 3S S40 S48 doi 10.1097/ACM.0000000000004525 Medline 34789656 PMC8855758 · DOI
  15. [17]Saeki S Okada R Shane PY Medical education during the COVID-19: a review of guidelines and policies adapted during the 2020 pandemic Healthcare (Basel) Mar 16 2023 11 6 867 doi 10.3390/healthcare11060867 Medline 36981524 PMC10048016 · DOI
  16. [18]Khan RA Jawaid M Khan AR Sajjad M ChatGPT - reshaping medical education and clinical management Pak J Med Sci 2023 39 2 605 607 doi 10.12669/pjms.39.2.7653 Medline 36950398 PMC10025693 · DOI · Paper page
  17. [22]Kimmerle J Timm J Festl-Wietek T Cress U Herrmann-Werner A Medical students' attitudes toward AI in medicine and their expectations for medical education J Med Educ Curric Dev Dec 6 2023 10 23821205231219346 doi 10.1177/23821205231219346 Medline 38075443 PMC10704950 · DOI · Paper page
  18. [23]Martínez-Mesa J González-Chica DA Bastos JL Bonamigo RR Duquia RP Sample size: how many participants do I need in my research? An Bras Dermatol 2014 89 4 609 615 doi 10.1590/abd1806-4841.20143705 Medline 25054748 PMC4148275 · DOI
  19. [24]Asiamah N Mensah H Oteng-Abayie EF Do larger samples really lead to more precise estimates? a simulation study Am J Educ Res 01 2017 5 1 9 17 doi 10.12691/education-5-1-2 · DOI
  20. [25]Gilson A Safranek CW Huang T et al How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? the implications of large language models for medical education and knowledge assessment JMIR Med Educ Feb 8 2023 9 e45312 doi 10.2196/45312 Medline 36753318 PMC9947764 · DOI · Paper page
  21. [26]Dave T Athaluri SA Singh S ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations Front Artif Intell 05 4 2023 6 1169595 doi 10.3389/frai.2023.1169595 Medline 37215063 PMC10192861 · DOI · Paper page
  22. [30]Safranek CW Sidamon-Eristoff AE Gilson A Chartash D The role of large language models in medical education: applications and implications JMIR Med Educ Aug 14 2023 9 e50945 doi 10.2196/50945 Medline 37578830 PMC10463084 · DOI · Paper page
  23. [33]Ashraf H Ashfaq H The role of ChatGPT in medical research: progress and limitations Ann Biomed Eng Mar 2024 52 3 458 461 doi 10.1007/s10439-023-03311-0 Medline 37452215 · DOI
  24. [35]Luykx JJ Gerritse F Habets PC Vinkers CH The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two-layer assessment World Psychiatry Oct 2023 22 3 479 480 doi 10.1002/wps.21145 Medline 37713576 PMC10503909 · DOI
  25. [36]Karabacak M Ozkara BB Margetis K Wintermark M Bisdas S The advent of generative language models in medical education JMIR Med Educ Jun 6 2023 9 e48163 doi 10.2196/48163 Medline 37279048 PMC10282912 · DOI · Paper page
  26. [38]Andrade C The limitations of online surveys Indian J Psychol Med Oct 13 2020 42 6 575 576 doi 10.1177/0253717620957496 Medline 33354086 PMC7735245 · DOI · Paper page

Authors
Hudon, A., Kiepura, B., Pelletier, M., & Phan, V.
Affiliations
  • Université de Montréal (CA)
  • Université Laval (CA)
Corresponding author
Hudon, Alexandre
ORCID
Venue
JMIR Medical Education
Publisher
JMIR Publications Inc.
Published
2024-04-04
Volume
10
Pages
e54067-e54067
ISSN
2369-3762 (ISSN-L)
DOI
10.2196/54067
PMC
PMC11007379
OpenAlex
W4394689344
Cited by
32
References
37
Citations by year
2026: 9 · 2025: 22 · 2024: 1
FWCI
3.00
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Keywords
ChatGPT, script concordance test, generative AI, psychiatric training, undergraduate medical education, clinical reasoning, AI-generated educational materials, DSM-5 diagnostic criteria
MeSH terms
Artificial Intelligence, Canada, Education, Medical, Undergraduate, Humans, Learning, Psychiatry

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review