Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items

Young, Roger, Courtney, Emily, Kah, Alexander, Wilkerson, Mariah, Chen, Yi-Hsin · 2025 · Teaching of Psychology

Abstract

Background: Multiple-choice item (MCI) assessments are burdensome for instructors to develop. Artificial intelligence (AI, e.g., ChatGPT) can streamline the process without sacrificing quality. The quality of AI-generated MCIs and human experts is comparable. However, whether the quality of AI-generated MCIs is equally good across various domain- and task-specific prompts remains to be determined. Therefore, we ask whether AI can generate high-quality MCIs to assess learning outcomes from a psychology textbook chapter reading. Objective: In an exploratory study, we enlist Item Response Theory analysis and expert reviewers to assess MCIs generated by ChatGPT-4 from a psychology textbook chapter. Method: We submitted a prompt and textbook chapter to ChatGPT-4 requesting 20 MCIs. One hundred ninety undergraduate participants read the chapter before responding to the MCIs. Expert reviewers assessed the MCIs for learning outcome alignment and quality. Results: ChatGPT-4-generated MCIs were low in difficulty and high in discrimination. Expert reviewers found that nearly all items were logically sound, aligned with learning objectives, and met prevailing standards of MCI quality. Conclusion: When carefully prompted, ChatGPT-4 can rapidly generate high-quality MCIs to test comprehension of a psychology textbook chapter. However, due to the uniformly low difficulty of the items, we recommend enlisting ChatGPT-4 to write MCIs for formative, but not summative, assessments.

Overview

Integrity signals

Registry

Crossref · 10.1177/00986283241311220

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 27 references

Research works printed but not linked: 3

  1. Bhandari S. Liu Y. Pardos Z. A. (2023). Evaluating ChatGPT-generated textbook questions using IRT [Paper presentation]. Generative AI for Education Workshop (GAIED) at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) New Orleans LA December 10–16 2023 (pp. 1–9). https://gaied.org/neurips2023/files/44/44_paper.pdf
  2. Brannan D., Mohr C. D. (2024). Love, friendship, and social support. In Biswas-Diener R., Diener E. (Eds.), Noba textbook series: Psychology. DEF Publishers. http://noba.to/s54tmp7k
  3. Yi-Hsin C., Li I. (2015). IA_CTT: A SAS Macro for conducting item analysis based on classical test theory. Southeast SAS Users Group 2015 Proceedings. SAS Institute. https://www.lexjansen.com/sesug/2015/184_Final_PDF.pdf

Cited by

See all 6

Metadata

Authors
Young, R., Courtney, E., Kah, A., Wilkerson, M., & Chen, Y.-H.
Affiliations
University of South Florida (US)
Corresponding author
Young, Roger
Venue
Teaching of Psychology
Published
2025-01-31
Volume
52
Issue
3
Pages
305-313
ISSN
  • 0098-6283 (ISSN-L)
  • 1532-8023
OpenAlex
W4407043322
Cited by
6
References
27
Citations by year
2026: 2 · 2025: 4
FWCI
6.29
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
Teaching of Psychology (publishedVersion)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
  • Social Sciences › Education › Educational Assessment and Pedagogy
Keywords
ChatGPT-4, item generation, item difficulty, item discrimination, prompt engineering, formative assessment, summative assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items. https://app.cochat.ai/papers/10.1177/00986283241311220
The paper
Young, R., Courtney, E., Kah, A., Wilkerson, M., & Chen, Y.-H. (2025). Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items. Teaching of Psychology. https://doi.org/10.1177/00986283241311220

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.