The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items
Abstract
Background: Multiple-choice item (MCI) assessments are burdensome for instructors to develop. Artificial intelligence (AI, e.g., ChatGPT) can streamline the process without sacrificing quality. The quality of AI-generated MCIs and human experts is comparable. However, whether the quality of AI-generated MCIs is equally good across various domain- and task-specific prompts remains to be determined. Therefore, we ask whether AI can generate high-quality MCIs to assess learning outcomes from a psychology textbook chapter reading. Objective: In an exploratory study, we enlist Item Response Theory analysis and expert reviewers to assess MCIs generated by ChatGPT-4 from a psychology textbook chapter. Method: We submitted a prompt and textbook chapter to ChatGPT-4 requesting 20 MCIs. One hundred ninety undergraduate participants read the chapter before responding to the MCIs. Expert reviewers assessed the MCIs for learning outcome alignment and quality. Results: ChatGPT-4-generated MCIs were low in difficulty and high in discrimination. Expert reviewers found that nearly all items were logically sound, aligned with learning objectives, and met prevailing standards of MCI quality. Conclusion: When carefully prompted, ChatGPT-4 can rapidly generate high-quality MCIs to test comprehension of a psychology textbook chapter. However, due to the uniformly low difficulty of the items, we recommend enlisting ChatGPT-4 to write MCIs for formative, but not summative, assessments.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Journal of Education and Practice
(2017) · doi:10.7176/jep · Cited by 2,735
Essentials of Educational Measurement
Garvin, A. D., & Ebel, R. L. (1980) · Educational Researcher · doi:10.2307/1175572 · Cited by 1,481
Methodology Review: Assessing Unidimensionality of Tests and ltenls
Hattie, J. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900204 · Cited by 1,013
The promises and challenges of artificial intelligence for teachers: A systematic review of research
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022) · TechTrends · doi:10.1007/s11528-022-00715-y · Cited by 984 · cited 1×
“However, generative AI and advanced machine learning methods have the potential to be harnessed by educators as a tool for AQG and the ef fi cient creation of MCI assessments, among other critical teaching tasks (Celik et al., 2022).”
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973 · cited 5×
“The current research employs an IRT framework (Embretson & Reise, 2013) and qualitative item-level review drawing from prevailing guidelines for MCI writing (Haladyna et al., 2002).”
InfluentialUnifactor Latent Trait Models Applied to Multifactor Tests: Results and Implications
Reckase, M. D. (1979) · Journal of Educational Statistics · doi:10.2307/1164671 · Cited by 933
Local Dependence Indexes for Item Pairs Using Item Response Theory
Chen, W.-H., & Thissen, D. (1997) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165285 · Cited by 806
Embretson, S. E., & Reise, S. P. (2013) · Psychology Press eBooks · doi:10.4324/9781410605269 · Cited by 761
Scaling Performance Assessments: Strategies for Managing Local Item Dependence
Yen, W. M. (1993) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1993.tb00423.x · Cited by 744
(2011) · Routledge eBooks · doi:10.4324/9780203874776 · Cited by 574
Research works printed but not linked: 3
- Bhandari S. Liu Y. Pardos Z. A. (2023). Evaluating ChatGPT-generated textbook questions using IRT [Paper presentation]. Generative AI for Education Workshop (GAIED) at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) New Orleans LA December 10–16 2023 (pp. 1–9). https://gaied.org/neurips2023/files/44/44_paper.pdf
- Brannan D., Mohr C. D. (2024). Love, friendship, and social support. In Biswas-Diener R., Diener E. (Eds.), Noba textbook series: Psychology. DEF Publishers. http://noba.to/s54tmp7k
- Yi-Hsin C., Li I. (2015). IA_CTT: A SAS Macro for conducting item analysis based on classical test theory. Southeast SAS Users Group 2015 Proceedings. SAS Institute. https://www.lexjansen.com/sesug/2015/184_Final_PDF.pdf
Cited by
Grobelny, J., Szymański, K., & Strozyk, Z. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105813 · Cited by 8
Park, G., & Choi, S. (2025) · Journal of Curriculum and Evaluation · doi:10.29221/jce.2025.28.3.319 · Cited by 2
Introduction to the Special Issue on Artificial Intelligence and the Teaching of Psychology
Boysen, G. A. (2025) · Teaching of Psychology · doi:10.1177/00986283251331220 · Cited by 3
Psychometrics and Linguistics in ChatGPT-Generated Reading Tests Compared with CET-4
Li, B., Qin, Q., Li, W., & Mao, C. (2026) · Journal of Psychoeducational Assessment · doi:10.1177/07342829261444828 · Cited by 3
“However, their use in high-stakes testing should be approached with caution, aligning with prior studies (Bhandari et al., 2024; Young et al., 2025) that recommend using ChatGPT-generated items primarily in low-stakes testing.”
InfluentialAutomatic Question Generation with Large Language Models: A Survey
de Oliveira, R. M., Hernández, A., Garbin, M. C., dos Reis, J. C., & Azevedo, R. (2026) · ACM Computing Surveys · doi:10.1145/3832773 · Cited by 1
Metadata
- Authors
- Young, R., Courtney, E., Kah, A., Wilkerson, M., & Chen, Y.-H.
- Affiliations
- University of South Florida (US)
- Corresponding author
- Young, Roger
- Venue
- Teaching of Psychology
- Publisher
- SAGE Publications
- Published
- 2025-01-31
- Volume
- 52
- Issue
- 3
- Pages
- 305-313
- ISSN
- 0098-6283 (ISSN-L)
- 1532-8023
- OpenAlex
- W4407043322
- Semantic Scholar
- df83db6179d4c25e3ef94088d9d5962943184967
- Cited by
- 6
- References
- 27
- Citations by year
- 2026: 2 · 2025: 4
- FWCI
- 6.29
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Teaching of Psychology (publishedVersion)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Education › Educational Assessment and Pedagogy
- Keywords
- ChatGPT-4, item generation, item difficulty, item discrimination, prompt engineering, formative assessment, summative assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items. https://app.cochat.ai/papers/10.1177/00986283241311220
- The paper
- Young, R., Courtney, E., Kah, A., Wilkerson, M., & Chen, Y.-H. (2025). Content and Item Response Theory Analysis of ChatGPT-4-Generated Multiple-Choice Items. Teaching of Psychology. https://doi.org/10.1177/00986283241311220
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.