Capitalization on Item Calibration Error in Adaptive Testing
Abstract
In adaptive testing, item selection is sequentially optimized during the test. Because the optimization takes place over a pool of items calibrated with estimation error, capitalization on chance is likely to occur. How serious the consequences of this phenomenon are depends not only on the distribution of the estimation errors in the pool or the conditional ratio of the test length to the pool size given ability, but may also depend on the structure of the item selection criterion used. A simulation study demonstrated a dramatic impact of capitalization on estimation errors on ability estimation. Four different strategies to minimize the likelihood of capitalization on error in computerized adaptive testing are discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
IRT places item parameters and examinee ability on a common interval scale, allowing direct comparison of abilities even when examinees respond to entirely different sets of items, forming the theoretical basis for CAT.
“IRT places item parameters and examinee ability on a common interval scale (typically Mean = 0, SD = 1). This allows for the direct comparison of abilities even when examinees respond to entirely different sets of items, forming the theoretical basis for Computerized Adaptive Testing28.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [28]
Cites
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,012 · see citing works
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,009 · see citing works
Item Response Theory: Principles and Applications
Hambleton, R. K., & Swaminathan, H. (1984) · Medical Entomology and Zoology · Cited by 1,779 · see citing works
Hambleton, R. K., & Swaminathan, H. (1985) · doi:10.1007/978-94-017-1988-9 · Cited by 1,777 · see citing works
Weighted Likelihood Estimation of Ability in Item Response Theory
Warm, T. A. (1989) · Psychometrika · doi:10.1007/bf02294627 · Cited by 1,216 · see citing works
Statistical theories of mental test scores
Lord, F. M., Novick, M. R., & Birnbaum, A. (1974) · Cited by 1,174 · see citing works
Computerized Adaptive Testing: A Primer
Wainer, H., Dorans, N. J., Flaugher, R. L., Green, B. F., & Mislevy, R. J. (1990) · Medical Entomology and Zoology · Cited by 1,169 · see citing works
A Global Information Approach to Computerized Adaptive Testing
Chang, H., & Ying, Z. (1996) · Applied Psychological Measurement · doi:10.1177/014662169602000303 · Cited by 369 · see citing works
A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing
Owen, R. J. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479871 · Cited by 289 · see citing works
A Method for Severely Constrained Item Selection in Adaptive Testing
Stocking, M. L., & Swanson, L. (1993) · Applied Psychological Measurement · doi:10.1177/014662169301700308 · Cited by 215 · see citing works
A Model for Optimal Constrained Adaptive Testing
van der Linden, W. J., & Reese, L. M. (1998) · Applied Psychological Measurement · doi:10.1177/01466216980223006 · Cited by 129 · see citing works
Some New Item Selection Criteria for Adaptive Testing
Veerkamp, W. J. J., & Berger, M. P. F. (1997) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986022002203 · Cited by 122 · see citing works
Item exposure control in CAT-ASVAB.
Hetter, R. D., & Sympson, J. B. (1997) · American Psychological Association eBooks · doi:10.1037/10244-014 · Cited by 79 · see citing works
LORD, F. M. (1983) · New Horizons in Testing · doi:10.1016/b978-0-12-742780-5.50011-1 · Cited by 69 · see citing works
Influence of Item Parameter Estimation Errors in Test Development
Hambleton, R. K., Jones, R., & Rogers, H. J. (1993) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1993.tb01071.x · Cited by 60 · see citing works
Optimal Sequential Designs for On-line Item Estimation
Jones, D. H., & Jin, Z. (1994) · Psychometrika · doi:10.1007/bf02294265 · Cited by 46 · see citing works
Item Parameter Estimation Errors and Their Influence on Test Information Functions
Hambleton, R. K., & Jones, R. W. (1994) · Applied Measurement in Education · doi:10.1207/s15324818ame0703_1 · Cited by 39 · see citing works
An Empirical Bayesian Approach to Item Banking
van der Linden, W. J., & Eggen, T. J. H. M. (1986) · Applied Psychological Measurement · doi:10.1177/014662168601000403 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Item Selection and Ability Estimation in Adaptive Testing
van der Linden, W. J., & Pashley, P. J. (2000) · Springer eBooks · doi:10.1007/0-306-47531-6_1 · Cited by 123 · see citing works
Computerized Adaptive Testing with R: Recent Updates of the Package catR
Magis, D., & Barrada, J. R. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.c01 · Cited by 83 · see citing works
Wang, S., & Wang, T. (2001) · Applied Psychological Measurement · doi:10.1177/01466210122032163 · Cited by 79 · see citing works
Modeling Rule-Based Item Generation
Geerlings, H., Glas, C. A. W., & van der Linden, W. J. (2011) · Psychometrika · doi:10.1007/s11336-011-9204-x · Cited by 58 · see citing works
Self-citationItem Selection and Ability Estimation in Adaptive Testing
van der Linden, W. J., & Pashley, P. J. (2009) · Springer eBooks · doi:10.1007/978-0-387-85461-8_1 · Cited by 55 · see citing works
Implementing Content Constraints in Alpha-Stratified Adaptive Testing Using a Shadow Test Approach
van der Linden, W. J., & Chang, H. (2003) · Applied Psychological Measurement · doi:10.1177/0146621602250531 · Cited by 37 · see citing works
Online Calibration Via Variable Length Computerized Adaptive Testing
Chang, Y. I., & Lu, H.-Y. (2009) · Psychometrika · doi:10.1007/s11336-009-9133-0 · Cited by 35 · see citing works
The Effect of Examinee Motivation on Test Construction Within an IRT Framework
van Barneveld, C. (2006) · Applied Psychological Measurement · doi:10.1177/0146621606286206 · Cited by 33 · see citing works
The Influence of Item Calibration Error on Variable-Length Computerized Adaptive Testing
Patton, J. M., Cheng, Y., Yuan, K., & Diao, Q. (2012) · Applied Psychological Measurement · doi:10.1177/0146621612461727 · Cited by 30 · see citing works
Optimal Design in Educational Testing
Buyske, S. G. (2005) · doi:10.1002/0470857005.ch1 · Cited by 26 · see citing works
A Fast and Simple Algorithm for Bayesian Adaptive Testing
van der Linden, W. J., & Ren, H. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619858970 · Cited by 24 · see citing works
An Automatic Online Calibration Design in Adaptive Testing
Makransky, G., & Glas, C. A. W. (2010) · Cited by 24 · see citing works
Self-citationSorrel, M. Á., Barrada, J. R., TORRE, J. D. L., & Abad, F. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0227196 · Cited by 22 · see citing works
Variable‐Length Computerized Adaptive Testing Using the Higher Order DINA Model
Hsu, C., & Wang, W. (2015) · Journal of Educational Measurement · doi:10.1111/jedm.12069 · Cited by 21 · see citing works
25 Statistical Aspects of Adaptive Testing
van der Linden, W. J., & Glas, C. A. W. (2006) · Handbook of statistics · doi:10.1016/s0169-7161(06)26025-5 · Cited by 20 · see citing works
Self-citationItem Selection Rules in Computerized Adaptive Testing
Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (2009) · Methodology · doi:10.1027/1614-2241.5.1.7 · Cited by 19 · see citing works
Computerized Adaptive Testing: The Capitalization on Chance Problem
Olea, J., Barrada, J. R., Abad, F. J., Ponsoda, V., & Cuevas, L. (2012) · The Spanish Journal of Psychology · doi:10.5209/rev_sjop.2012.v15.n1.37348 · Cited by 18 · see citing works
Optimal Test Design With Rule-Based Item Generation
Geerlings, H., van der Linden, W. J., & Glas, C. A. W. (2012) · Applied Psychological Measurement · doi:10.1177/0146621612468313 · Cited by 17 · see citing works
Self-citationLiu, Y., Cheng, Y., & Liu, H. (2020) · Educational and Psychological Measurement · doi:10.1177/0013164419895068 · Cited by 16 · see citing works
The Problem of Bias in Person Parameter Estimation in Adaptive Testing
Doebler, A. (2012) · Applied Psychological Measurement · doi:10.1177/0146621612443304 · Cited by 15 · see citing works
Robust Computerized Adaptive Testing
Veldkamp, B. P., & Verschoor, A. J. (2019) · Methodology of educational measurement and assessment · doi:10.1007/978-3-030-18480-3_15 · Cited by 14 · see citing works
Cross-Validating Item Parameter Estimation in Adaptive Testing
van der Linden, W. J., & Glas, C. A. W. (2001) · Lecture notes in statistics · doi:10.1007/978-1-4613-0169-1_12 · Cited by 14 · see citing works
Self-citationEffects of Item Calibration Errors on Computerized Adaptive Testing under Cognitive Diagnosis Models
Huang, H. (2018) · Journal of Classification · doi:10.1007/s00357-018-9265-y · Cited by 14 · see citing works
So Many Questions, So Little Time: Integrating Adaptive Inventories into Public Opinion Research
Montgomery, J., & Rossiter, E. (2019) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smz027 · Cited by 14 · see citing works
Yu, X., Cheng, Y., & Chang, H. (2019) · Methodology of educational measurement and assessment · doi:10.1007/978-3-030-05584-4_15 · Cited by 14 · see citing works
a -Stratified Computerized Adaptive Testing in the Presence of Calibration Error
Cheng, Y., Patton, J. M., & Shao, C. (2014) · Educational and Psychological Measurement · doi:10.1177/0013164414530719 · Cited by 13 · see citing works
Psychometric aspects of pupil monitoring systems
Glas, C. A. W., & Geerlings, H. (2009) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2009.10.006 · Cited by 13 · see citing works
Self-citationKolen, M. J., & Brennan, R. L. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0317-7_8 · Cited by 13 · see citing works
Application of Optimal Designs to Item Calibration
Lu, H.-Y. (2014) · PLoS ONE · doi:10.1371/journal.pone.0106747 · Cited by 12 · see citing works
Choi, S. W., & van der Linden, W. J. (2017) · Quality of Life Research · doi:10.1007/s11136-017-1650-1 · Cited by 12 · see citing works
A New Online Calibration Method Based on Lord’s Bias-Correction
He, Y., Chen, P., Li, Y., & Zhang, S. (2017) · Applied Psychological Measurement · doi:10.1177/0146621617697958 · Cited by 11 · see citing works
Bayesian adaptive testing with polytomous items
Ren, H., Choi, S. W., & van der Linden, W. J. (2020) · Behaviormetrika · doi:10.1007/s41237-020-00114-8 · Cited by 10 · see citing works
More efficient fully Bayesian adaptive testing with a revised proposal distribution
Niu, L., & Choi, S. W. (2022) · Behaviormetrika · doi:10.1007/s41237-021-00156-6 · Cited by 8 · see citing works
Tsaousis, I., Sideridis, G. D., & AlGhamdi, H. M. (2021) · Journal of Psychoeducational Assessment · doi:10.1177/07342829211027753 · Cited by 7 · see citing works
Accounting for item calibration error in computerized adaptive testing
Fink, A., König, C., & Frey, A. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02649-8 · Cited by 7 · see citing works
Online Calibration of Polytomous Items Under the Graded Response Model
Xiong, J., Ding, S., Luo, F., & Luo, Z. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03085 · Cited by 6 · see citing works
Sun, X., Liu, Y., Xin, T., & Song, N. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.575141 · Cited by 6 · see citing works
Berger, S., Verschoor, A. J., Eggen, T. J. H. M., & Moser, U. (2019) · Journal of Educational Measurement · doi:10.1111/jedm.12203 · Cited by 5 · see citing works
Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error
Cooperman, A. W., Weiss, D. J., & Wang, C. (2021) · Educational and Psychological Measurement · doi:10.1177/00131644211033902 · Cited by 3 · see citing works
Desarrollo de un banco de ítems para medir conocimiento en estudiantes universitarios
Cupani, M., Ghío, F. B., Leal, M. F., Giraudo, G. M., Zamparella, T. C., Piumatti, G., Casalotti, A. B., Ramírez, J. C., Arranz, M. A., Farías, A. N., et al. (2017) · Revista de psicología · doi:10.5354/0719-0581.2016.44808 · Cited by 4 · see citing works
Chapter 1 Item Selection and Ability Estimation in Adaptive Testing
van der Linden, W. J., & Pashley, P. J. (2010) · Cited by 4 · see citing works
Cao, X., Lin, Y., Liu, D., Zheng, F., & Duh, H. B.-L. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02228-9 · Cited by 6 · see citing works
Optimal item pool design for a highly constrained computerized adaptive test
Reckase, M. D., & He, W. (2010) · ProQuest LLC eBooks · Cited by 3 · see citing works
Condy, E. E., Kaat, A. J., Becker, L., Sullivan, N. R., Soorya, L. V., Berger, N. I., Berry‐Kravis, E., Michalak, C., & Thurm, A. E. (2021) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2021.103993 · Cited by 3 · see citing works
Compound Optimal Design for Online Item Calibration Under the Two-Parameter Logistic Model
song, L., & Wang, W. (2025) · Applied Psychological Measurement · doi:10.1177/01466216251316276 · Cited by 2 · see citing works
A Two-Level Adaptive Test Battery
van der Linden, W. J., Niu, L., & Choi, S. W. (2023) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986231209447 · Cited by 2 · see citing works
Kárász, J. T., & Takács, S. (2021) · Alkalmazott Matematikai Lapok · doi:10.37070/aml.2021.38.1.04 · Cited by 2 · see citing works
The Effects of Examinee Motivation on Multiple-Choice Item Parameter Estimates
van Barneveld, C. (2003) · Alberta Journal of Educational Research · doi:10.55016/ojs/ajer.v49i3.54985 · Cited by 2 · see citing works
computerized adaptive testing in industrial and organizational psychology
Makransky, G. (2011) · doi:10.3990/1.9789036533164 · Cited by 2 · see citing works
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
“It enables a level of measurement fi delity previously achievable only through exhaustive, cost-prohibitive methods, but at a mere 1.3% of the item cost and 2% of the computational and temporal cost.”
InfluentialPatton, J. M., Cheng, Y., Yuan, K., & Diao, Q. (2012) · Psychological test and assessment modeling · Cited by 1 · see citing works
A Community-Approach to Item Calibration for Testing Math-Skills in Engineering
Aral, N., & Oppl, S. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-93904-5_46 · Cited by 1 · see citing works
Helbling, L. A., Berger, S., & Verschoor, A. J. (2021) · Frontiers in Education · doi:10.3389/feduc.2021.679864
Closed formula of test length required for adaptive testing with medium probability of solution
Kárász, J. T., Széll, K., & Takács, S. (2023) · Quality Assurance in Education · doi:10.1108/qae-03-2023-0042
Investigation of a multistage adaptive test based on test assembly methods
Doğruöz, E., & Kelecioğlu, H. (2024) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.1268614
Wu, Y., Huang, Y., Chen, C., & Chen, P. (2025) · Journal of Educational Measurement · doi:10.1111/jedm.12430
Effects of Item Calibration Error on Applications of Item Response Theory
Jeffrey, M. P., & Cheng, Y. (2014) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-62396-597-620251008
Online Calibration with Cognitive Diagnostic Assessment
Chen, P., & Xin, T. (2014) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-62396-597-620251018
Multidimensional Bayesian adaptive testing
Fink, A., König, C., & Frey, A. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03123-9
Variable-length fully Bayesian adaptive testing and associated stopping criteria
Niu, L., & Choi, S. W. (2026) · Behavior Research Methods · doi:10.3758/s13428-026-03148-0
Recovering Accuracy From Miscalibrated Item Banks: Scoring Error Is Fixable, Selection Error Is Not
Xiao, X., Domingue, B. W., Ram, N., & Frank, M. C. (2026) · Educational and Psychological Measurement · doi:10.1177/00131644261486854
OPTIVAL: An R package for model-based optimal test validity analysis
Navarro-González, D., Ferrando, P. J., & Morales-Vives, F. (2026) · SoftwareX · doi:10.1016/j.softx.2026.103157
- Authors
- van der Linden, W. J., & Glas, C. A. W.
- Affiliations
- University of Twente (NL)
- ORCID
- Glas, Cees A.W.
- Venue
- Applied Measurement in Education
- Publisher
- Informa UK Limited
- Published
- 2000-01
- Volume
- 13
- Issue
- 1
- Pages
- 35-53
- ISSN
- 0895-7347 (ISSN-L)
- 1532-4818
- OpenAlex
- W2006461110
- Cited by
- 63
- References
- 18
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 1 · 2023: 3 · 2022: 3 · 2021: 5 · 2020: 5 · 2019: 6 · 2018: 1 · 2017: 4 · 2015: 1 · 2014: 5 · 2012: 5
- FWCI
- 3.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Applied Measurement in Education (publishedVersion)
- University of Twente Research Information (publishedVersion)
- University of Twente Research Information (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- adaptive testing, computerized adaptive testing, item selection, estimation error, ability estimation, test length, item pool
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
