Evaluating the Psychometric Characteristics of Generated Multiple-Choice Test Items
Abstract
Item development is a time- and resource-intensive process. Automatic item generation integrates cognitive modeling with computer technology to systematically generate test items. To date, however, items generated using cognitive modeling procedures have received limited use in operational testing situations. As a result, the psychometric characteristics of generated multiple-choice test items are largely unknown and undocumented. We present item analysis results from one of the first empirical studies designed to evaluate the psychometric properties of generated multiple-choice items using the results from a high stakes national medical licensure examination. The item analysis results for the correct option revealed that the generated items measured examinees’ performance across a broad range of ability levels while, at the same time, providing a consistently strong level of discrimination for each item. Results for the incorrect options revealed that the generated items consistently differentiated the low from the high performing examinees.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Items generated through AIGAQG have psychometric properties similar to traditionally constructed [1, 61].
“Items generated through AIGAQG have psychometric properties similar to traditionally constructed [1, 61].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [61]
Cites
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973 · cited 1× · see citing works
“They compared items developed with AIG to those created with traditional item development procedures using eight indicators of item quality, as identified by Haladyna, Downing, and Rodriguez (2002).”
MethodLane, S. (2015) · Routledge eBooks · doi:10.4324/9780203102961 · Cited by 730 · see citing works
(2011) · Routledge eBooks · doi:10.4324/9780203874776 · Cited by 574 · see citing works
van der Linden, W. J., & van der Linden, W. J. (2010) · Springer eBooks · doi:10.1007/978-0-387-85461-8 · Cited by 515 · see citing works
Item Generation for Test Development
(2013) · Routledge eBooks · doi:10.4324/9781410602145 · Cited by 298 · see citing works
Semantic structures and diagnostic thinking of experts and novices
Bordage, G., & Lemieux, M. (1991) · Academic Medicine · doi:10.1097/00001888-199109000-00045 · Cited by 225 · see citing works
Expertise in Medicine and Surgery
Norman, G. R., Eva, K. W., Brooks, L. R., & Hamstra, S. J. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816796.019 · Cited by 186 · see citing works
(2012) · Routledge eBooks · doi:10.4324/9780203803912 · Cited by 152 · see citing works
Using automatic item generation to create multiple‐choice test items
Gierl, M. J., Lai, H., & Turner, S. R. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04289.x · Cited by 139 · see citing works
Self-citationComputer-Assisted Test Assembly Using Optimization Heuristics
Luecht, R. M. (1998) · Applied Psychological Measurement · doi:10.1177/01466216980223003 · Cited by 136 · see citing works
National Council on Measurement in Education
Pedulla, J. J. (1984) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1984.tb00749.x · Cited by 120 · see citing works
Promoting diagnostic problem representation
NENDAZ, M. R., & Bordage, G. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01279.x · Cited by 99 · see citing works
A FEASIBILITY STUDY OF ON‐THE‐FLY ITEM GENERATION IN ADAPTIVE TESTING
Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2002) · ETS Research Report Series · doi:10.1002/j.2333-8504.2002.tb01890.x · Cited by 83 · see citing works
Gierl, M. J., & Lai, H. (2013) · Educational Measurement Issues and Practice · doi:10.1111/emip.12018 · Cited by 73 · see citing works
Self-citationGenerative Testing: From Conception to Implementation
Singley, M. K., & Bennett, R. E. (2013) · Routledge eBooks · doi:10.4324/9781410602145-18 · Cited by 66 · see citing works
Developing a Taxonomy of Item Model Types to Promote Assessment Engineering
Gierl, M. J., Zhou, J., & Alves, C. (2008) · Open Access Journals at BC (Boston College) · Cited by 60 · see citing works
Self-citationItem modelling procedure for constructing content‐equivalent multiple choice questions
LaDuca, A., STAPLES, W. I., Templeton, B., & Holzman, G. B. (1986) · Medical Education · doi:10.1111/j.1365-2923.1986.tb01042.x · Cited by 60 · see citing works
Assessing the semantic content of clinical case presentations
Bordage, G., Connell, K. J., Chang, R. W., Gecht, M. R., & Sinacore, J. (1997) · Academic Medicine · doi:10.1097/00001888-199710001-00013 · Cited by 53 · see citing works
Evaluating the quality of medical multiple‐choice items created with automated processes
Gierl, M. J., & Lai, H. (2013) · Medical Education · doi:10.1111/medu.12202 · Cited by 41 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Automatic item generation: foundations and machine learning-based approaches for assessments
Circi, R., Hicks, J., & Sikali, E. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.858273 · Cited by 52 · see citing works
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36 · see citing works
- Authors
- Gierl, M. J., Lai, H., Pugh, D., Touchie, C., Boulais, A.-P., & De Champlain, A.
- Affiliations
- University of Alberta (CA)
- University of Ottawa (CA)
- Medical Council of Canada (CA)
- Corresponding author
- Gierl, Mark J.
- Venue
- Applied Measurement in Education
- Publisher
- Informa UK Limited
- Published
- 2016-03-30
- Volume
- 29
- Issue
- 3
- Pages
- 196-210
- ISSN
- 0895-7347 (ISSN-L)
- 1532-4818
- OpenAlex
- W2320000683
- Cited by
- 33
- References
- 19
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 7 · 2023: 8 · 2022: 3 · 2021: 1 · 2020: 2 · 2019: 1 · 2018: 3 · 2017: 1 · 2016: 2
- FWCI
- 2.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Applied Measurement in Education (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- automatic item generation, multiple-choice test items, psychometric properties, item analysis, cognitive modeling, medical licensing examinations, item discrimination, item difficulty, distractor analysis, high-stakes testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
