Estimating Item Difficulty With Comparative Judgments
Open accessAbstract
Previous investigations of the ability of content experts and test developers to estimate item difficulty have, for the most part, produced disappointing results. These investigations were based on a noncomparative method of independently rating the difficulty of items. In this article, we argue that, by eliciting comparative judgments of difficulty, judges can more accurately estimate item difficulties. In this study, judges from different backgrounds rank ordered the difficulty of SAT® mathematics items in sets of 7 items. Results showed that judges are reasonably successful in rank ordering several items in terms of difficulty, with little variability across judges and content areas. Simulations of a possible implementation of comparative judgments for difficulty estimation show that it is possible to achieve high correlations between true and estimated difficulties with relatively few comparisons. Implications of these results for the test development process are discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There are variations and contradictions in experts' judgments regarding item difficulty in the literature [58-60].
“There are variations and contradictions in experts’ judgments regarding item difficulty in the literature [58–60].”
From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [60]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
A law of comparative judgment.
Thurstone, L. (1994) · Psychological Review · doi:10.1037/h0070288 · Cited by 5,077 · see citing works
Construct validity: Construct representation versus nomothetic span.
Whitely, S. E. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.1.179 · Cited by 605 · see citing works
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (1995) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429258411-25 · Cited by 553 · see citing works
A cognitive design system approach to generating valid tests: Application to abstract reasoning.
Embretson, S. E. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.3.380 · Cited by 349 · see citing works
CONSTRUCT VALIDITY: CONSTRUCT REPRESENTATION VERSUS NOMOTHETIC SPAN
Embretson, S. E. (1983) · Cited by 322 · see citing works
Item Generation for Test Development
(2013) · Routledge eBooks · doi:10.4324/9781410602145 · Cited by 298 · see citing works
Test theory for a new generation of tests
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5724 · Cited by 279 · see citing works
Construct Validity: A Universal Validity System or Just Another Test Evaluation Procedure?
Embretson, S. E. (2007) · Educational Researcher · doi:10.3102/0013189x07311600 · Cited by 92 · see citing works
Item Diffficulty Modeling of Paragraph Comprehension Items
Gorin, J. S., & Embretson, S. E. (2006) · Applied Psychological Measurement · doi:10.1177/0146621606288554 · Cited by 91 · see citing works
The prediction of TOEFL reading item difficulty: implications for construct validity
Freedle, R. O., & Kostin, I. W. (1993) · Language Testing · doi:10.1177/026553229301000203 · Cited by 85 · see citing works
How to Equate Tests With Little or No Data
Mislevy, R. J., Sheehan, K. M., & Wingersky, M. S. (1993) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1993.tb00422.x · Cited by 78 · see citing works
A GENERATIVE APPROACH TO PSYCHOLOGICAL AND EDUCATIONAL MEASUREMENT
Bejar, I. I. (1991) · ETS Research Report Series · doi:10.1002/j.2333-8504.1991.tb01387.x · Cited by 72 · see citing works
Subject Matter Experts' Assessment of Item Statistics
Bejar, I. I. (1983) · Applied Psychological Measurement · doi:10.1177/014662168300700306 · Cited by 71 · see citing works
Cross, L. H., Impara, J. C., Frary, R. B., & Jaeger, R. M. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb00224.x · Cited by 67 · see citing works
Swaminathan, H., Hambleton, R. K., Sireci, S. G., Xing, D., & Rizavi, S. (2003) · Applied Psychological Measurement · doi:10.1177/0146621602239475 · Cited by 53 · see citing works
The Improvement of Estimates of Test Difficulty
Lorge, I. D., & Kruglov, L. P. (1953) · Educational and Psychological Measurement · doi:10.1177/001316445301300104 · Cited by 38 · see citing works
ITEMS BY DESIGN: THE IMPACT OF SYSTEMATIC FEATURE VARIATION ON ITEM STATISTICAL CHARACTERISTICS
Enright, M. K., Morley, M. E., & Sheehan, K. M. (1999) · ETS Research Report Series · doi:10.1002/j.2333-8504.1999.tb01818.x · Cited by 34 · see citing works
Items by Design: The Impact of Systematic Feature Variation on Item Statistical Characteristics
Enright, M. K., Morley, M. E., & Sheehan, K. M. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1501_04 · Cited by 33 · see citing works
Accuracy of Item Performance Predictions Based on the Nedelsky Standard Setting Method
Melican, G. J., Mills, C. N., & Plake, B. S. (1989) · Educational and Psychological Measurement · doi:10.1177/0013164489492020 · Cited by 12 · see citing works
Estimating Item Difficulty with Comparative Judgments. Research Report. ETS RR-14-39.
Attali, Y., Saldivia, L. E., Jackson, C., Schuppan, F., & Wanamaker, W. (2014) · ETS Research Report Series · Cited by 1 · see citing works
Self-citationHambleton, R. K., Sireci, S. G., Swaminathan, H., Xing, D., & Rizavi, S. (2003) · Cited by 1 · see citing works
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Thorndike, R. L. (1982). Item and score conversion by pooled judgment. In P. W. Holland & D. B. Rubin (Eds.), Test equating (pp. 309-326). New York, NY: Academic.
Cited by
Sorted by citations
Hsu, F.-Y., Lee, H.-M., Chang, T.-H., & Sung, Y. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.06.007 · Cited by 41 · see citing works
A Survey on Recent Approaches to Question Difficulty Estimation from Text
Benedetto, L., Cremonesi, P., Caines, A., Buttery, P. J., Cappelli, A., Giussani, A., & Turrin, R. (2022) · ACM Computing Surveys · doi:10.1145/3556538 · Cited by 37 · see citing works
AI in medical education: uses of AI in construction type A MCQs
Rezigalla, A. A. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05250-3 · Cited by 36 · see citing works
Zhang, Y., Li, M., Jiang, H., Shi, H., Xu, B., Atkins, S. A., & Qian, X. (2019) · BMC Pediatrics · doi:10.1186/s12887-019-1670-9 · Cited by 16 · see citing works
Holmes, S. D., Meadows, M., Stockford, I., & He, Q. (2018) · International Journal of Testing · doi:10.1080/15305058.2018.1486316 · Cited by 8 · see citing works
Loginova, E., Benedetto, L., Benoit, D. F., & Cremonesi, P. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_097 · Cited by 7 · see citing works
Item Difficulty Modeling Using Fine-tuned Small and Large Language Models
Li, M., Jiao, H., Zhou, T., Zhang, N., Peters, S., & Lissitz, R. W. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644251344973 · Cited by 7 · see citing works
The difference between estimated and perceived item difficulty: An empirical study
SAYIN, A., & Bulut, O. (2024) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.1376160 · Cited by 4 · see citing works
Gedamu, S., Tadesse, T., Moges, B., & Schauber, S. K. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06012-x · Cited by 4 · see citing works
Automatic prediction of item difficulty for short-answer questions
Dueñas, G., Jiménez, S., & Baquero, J. (2015) · Computing Colombian Conference · doi:10.1109/columbiancc.2015.7333464 · Cited by 3 · see citing works
Shen, Y., Wang, S., & Xiao, H. (2024) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12340 · Cited by 2 · see citing works
Atarah, S. A., Egblewogbe, M. N. Y. H., Awoniyi, F. C., & Amankwah, A. (2025) · Africa Education Review · doi:10.1080/18146627.2025.2451425 · Cited by 2 · see citing works
Comparing Estimated and Real Item Difficulty Using Multi-Facet Rasch Analysis
SAYIN, A., & GÖREN, S. (2023) · Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi · doi:10.21031/epod.1310893 · Cited by 1 · see citing works
McGrane, J. A. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-818630-5.09023-0 · Cited by 1 · see citing works
Comparative JudgementPreprint
McGrane, J. A. (2024) · doi:10.31219/osf.io/927xk · Cited by 1 · see citing works
Multiple-Choice Questions Difficulty Prediction with Neural Networks
Reyes, D., Jiménez, A., Dartnell, P., Lions, S., & Ríos, S. A. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-41226-4_2
Question difficulty estimation via enhanced directional modality association transformer
Kim, B.-M., Park, G.-M., & Park, S.-B. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04988-5
Exploring Item Difficulty Prediction: Data Driven Approach for Item Difficulty Estimation
Lamgarraj, M., Joiron, C., Parant, A., & Dequen, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63028-6_36
Curcin, M., & Lee, M. Z. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1538486
Optimal Item Calibration in the Context of the Swedish Scholastic Aptitude Test
Bjermo, J., Fackle‐Fornius, E., & Miller, F. (2026) · Applied Psychological Measurement · doi:10.1177/01466216261420758
Kadam, S., Khan, S., Christopher, J. J., Kumar, P. P., & Satpathi, D. K. (2026) · International Journal of Mathematical Engineering and Management Sciences · doi:10.33889/ijmems.2026.11.2.028
- Authors
- Attali, Y., Saldivia, L., Jackson, C., Schuppan, F., & Wanamaker, W.
- Affiliations
- Educational Testing Service (US)
- Corresponding author
- Attali, Yigal
- Venue
- ETS Research Report Series
- Publisher
- Wiley
- Published
- 2014-12
- Volume
- 2014
- Issue
- 2
- Pages
- 1-8
- ISSN
- 2330-8516 (ISSN-L)
- OpenAlex
- W1688896956
- Cited by
- 22
- References
- 24
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 6 · 2023: 3 · 2022: 2 · 2021: 1 · 2019: 1 · 2018: 2 · 2015: 1 · 2014: 1
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ETS Research Report Series (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Educational Assessment and Pedagogy
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- item difficulty estimation, comparative judgment, rank order, test development, content experts, inter-rater reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
