Construct‐Irrelevant Variance in High‐Stakes Testing
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
There are many threats to validity in high‐stakes achievement testing. One major threat is construct‐irrelevant variance (CIV). This article defines CIV in the context of the contemporary, unitary view of validity and presents logical arguments, hypotheses, and documentation for a variety of CIV sources that commonly threaten interpretations of test scores. A more thorough study of CIV is recommended.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Statistical Theories of Mental Test Scores
Rozeboom, W. W., Lord, F. M., Novick, M. R., & Birnbaum, A. (1969) · American Educational Research Journal · doi:10.2307/1162101 · Cited by 8,022
Cohen, D. (1998) · The Women's Review of Books · doi:10.2307/4022859 · Cited by 4,261
Messick, S. (1995) · American Psychologist · doi:10.1037/0003-066x.50.9.741 · Cited by 4,198
Educational measurement, 3rd ed.
Linn, R. L. (1989) · Cited by 2,883
Standards for Educational and Psychological Testing.
García, E. J. (2017) · Revista Española de Pedagogía · doi:10.22550/2174-0909.3051 · Cited by 2,217
Differential Item Functioning.
Gudgeon, A. C., Holland, P. W., & Wainer, H. (1994) · The Statistician · doi:10.2307/2348590 · Cited by 2,091
Correlates, Causes, Effects, and Treatment of Test Anxiety
Hembree, R. (1988) · Review of Educational Research · doi:10.3102/00346543058001047 · Cited by 1,723
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121
Educational Assessment of Students
Nitko, A. J. (1996) · Cited by 1,033
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973
Self-citationPartners in Cognition: Extending Human Intelligence with Intelligent Technologies
SALOMON, G., PERKINS, D. N., & GLOBERSON, T. (1991) · Educational Researcher · doi:10.3102/0013189x020003002 · Cited by 948
Complex, Performance-Based Assessment: Expectations and Validation Criteria
Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991) · Educational Researcher · doi:10.2307/1176232 · Cited by 926
Fleming, C. M., & Travers, R. M. W. (1955) · British Journal of Educational Studies · doi:10.2307/3119507 · Cited by 910
An argument-based approach to validity.
Kane, M. T. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.112.3.527 · Cited by 901
Standards of validity and the validity of standards in performance asessment
Messick, S. (1995) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1995.tb00881.x · Cited by 697
Developing and Validating Multiple-choice Test Items
Haladyna, T. M. (2004) · Routledge eBooks · doi:10.4324/9780203825945 · Cited by 584
Self-citationArgument-Based Approach to Validity
(2022) · The SAGE Encyclopedia of Research Design · doi:10.4135/9781071812082.n28 · Cited by 556
Abilities Are Forms of Developing Expertise
Sternberg, R. J. (1998) · Educational Researcher · doi:10.2307/1176608 · Cited by 322
Cheating on Tests: How To Do It, Detect It, and Prevent It
Cizek, G. J. (1999) · Routledge eBooks · doi:10.4324/9781410601520 · Cited by 321
Test Anxiety: A Major Educational Problem and What Can Be Done about It
Hill, K. T., & Wigfield, A. (1984) · The Elementary School Journal · doi:10.1086/461395 · Cited by 286
Effects of Test Anxiety and Evaluative Threat on Students' Achievement and Motivation
Hancock, D. R. (2001) · The Journal of Educational Research · doi:10.1080/00220670109598764 · Cited by 270
Factors Influencing the English Reading Test Performance of Spanish-Speaking Hispanic Children
Garcia, G. E. (1991) · Reading Research Quarterly · doi:10.2307/747894 · Cited by 264
Improving student achievement: what state NAEP test scores tell us
(2001) · Choice Reviews Online · doi:10.5860/choice.38-4019 · Cited by 261
Cognition and the question of test item format
Martinez, M. E. (1999) · Educational Psychologist · doi:10.1207/s15326985ep3404_2 · Cited by 242
THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT
MESSICK, S. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb01030.x · Cited by 242
Fitzgerald, J. (1995) · Review of Educational Research · doi:10.2307/1170711 · Cited by 220
Threats to the Valid Use of Assessments
Crooks, T. J., Kane, M. T., & Cohen, A. S. (1996) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969594960030302 · Cited by 208
Mislevy, R. J. (1996) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1996.tb00498.x · Cited by 208
A Developmental Perspective on Standardized Achievement Testing
Paris, S. G., Lawton, T. A., Turner, J. C., & Roth, J. L. (1991) · Educational Researcher · doi:10.3102/0013189x020005012 · Cited by 182
Impact of Accommodation Strategies on English Language Learners' Test Performance
Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2000.tb00034.x · Cited by 170
Methods for Improving Standardized Test Scores: Fruitful, Fruitless, or Fraudulent?
Mehrens, W. A., & Kaminski, J. (1989) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1989.tb00304.x · Cited by 165
Assessing Differential Item Functioning in Performance Assessment: Review and Recommendations
Penfield, R. D., & Lam, T. C. M. (2000) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2000.tb00033.x · Cited by 161
Validating High‐Stakes Testing Programs
Kane, M. (2002) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2002.tb00083.x · Cited by 155
Validity Issues in Computer‐Based Testing
Huff, K., & Sireci, S. G. (2001) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2001.tb00066.x · Cited by 149
The Effects of Standardized Testing on Teaching and Schools
Herman, J. L., & Golan, S. (1993) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.1993.tb00550.x · Cited by 126
Linn, R. L., Graue, E., & Sanders, N. M. (1990) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1990.tb00372.x · Cited by 123
Cole, N. S., & Moss, P. (1989) · Cited by 123
Downing, S. M. (2002) · Advances in Health Sciences Education · doi:10.1023/a:1021112514626 · Cited by 121
Self-citationAn additive model of test anxiety: Role of exam-specific expectations.
Zohar, D. M. (1998) · Journal of Educational Psychology · doi:10.1037/0022-0663.90.2.330 · Cited by 109
Detection of Aberrant Item Score Patterns: A Review of Recent Developments
Meijer, R. R., & Sijtsma, K. (1995) · Applied Measurement in Education · doi:10.1207/s15324818ame0803_5 · Cited by 94
Teaching and Testing to Develop Fluid Abilities
Lohman, D. F. (1993) · Educational Researcher · doi:10.2307/1176750 · Cited by 81
The Impact of Mandated Standardized Testing on Minority Students
Lomax, R. G., West, M. M., Harmon, M., et al. (1995) · The Journal of Negro Education · doi:10.2307/2967240 · Cited by 80
Uses and Abuses of Achievement Test Scores
Nolen, S. B., Haladyna, T. M., & Haas, N. S. (1992) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1992.tb00234.x · Cited by 78
Self-citationConstruct-irrelevant Variance and Flawed Test Questions
Downing, S. M. (2002) · Academic Medicine · doi:10.1097/00001888-200210001-00032 · Cited by 76
Self-citationDeMars, C. E. (1998) · Applied Measurement in Education · doi:10.1207/s15324818ame1103_4 · Cited by 61
How to Prepare Students for PerEorrnance Assessments
Mehrens, W. A., Popham, W. J., & Ryan, J. M. (1998) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1998.tb00617.x · Cited by 43
Book Reviews: Response to Rosa Hernández Sheets's Review of Race and Culture
Dilg, M. A. (2000) · Educational Researcher · doi:10.3102/0013189x029009024 · Cited by 25
Seeking Redemption for Our Psychometric Sins
Popham, W. J. (2003) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2003.tb00117.x · Cited by 22
Cited by
Maeda, Y., & Yoon, S. Y. (2013) · Educational Psychology Review · doi:10.1007/s10648-012-9215-x · Cited by 398
Confirmatory Factor Analytic Procedures for the Determination of Measurement Invariance
French, B. F., & Finch, W. H. (2006) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1303_3 · Cited by 313
Measuring Learning Outcomes in Higher Education: Motivation Matters
Liu, O. L., Bridgeman, B., & Adler, R. M. (2012) · Educational Researcher · doi:10.3102/0013189x12459679 · Cited by 285
“…students’ lack of motivation to perform well on the tests could pose a serious threat to the validity of the test scores and the interpretation accuracy of the test results (Banta, 2008; Haladyna & Downing, 2004; Liu, 2011b; S. L. Wise & DeMars, 2005, 2010; V. L. Wise, Wise, & Bhola, 2006).”
BackgroundValidity threats: overcoming interference with proposed interpretations of assessment data
Downing, S. M., & Haladyna, T. M. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01777.x · Cited by 282
Self-citationLuo, J. (., & Dawson, P. (2026) · Studies in Higher Education · doi:10.1080/03075079.2025.2552825 · Cited by 17
Metadata
- Authors
- Haladyna, T. M., & Downing, S. M.
- Venue
- Educational Measurement: Issues and Practice
- Publisher
- Wiley
- Published
- 2004-03
- Volume
- 23
- Issue
- 1
- Pages
- 17-27
- ISSN
- 0731-1745 (ISSN-L)
- 1745-3992
- OpenAlex
- W2110306959
- Cited by
- 386
- References
- 49
- FWCI
- 14.67
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.