The importance of measurement invariance in neurocognitive ability testing
Open accessThe DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
OBJECTIVE: Neurocognitive test batteries such as recent editions of the Wechsler's Adult Intelligence Scale (WAIS-III/WAIS-IV) typically use nation-level population-based norms. The question is whether these batteries function in the same manner across different subgroups based on gender, age, educational background, socioeconomic status, ethnicity, mother tongue, or race. Here, the author argues that measurement invariance is a core issue in determining whether population-based norms are valid for different subgroups. METHOD: The author introduces measurement invariance, argues why it is an important topic of study, discusses why invariance might fail in cognitive ability testing, and reviews a dozen studies of invariance of commonly used neurocognitive test batteries. RESULTS: In over half of the reviewed studies, IQ batteries were not found to be measurement invariant across groups based on ethnicity, gender, educational background, cohort, or age. Apart from age and cohort, test manuals do not take such lack of invariance into account in computing full-scale IQ scores or normed domain scores. CONCLUSIONS: Measurement invariance is crucial for valid use of neurocognitive tests in clinical, educational, and professional practice. The appropriateness of population-based norms to particular subgroups should depend also on whether measurement invariance holds with respect to important subgroups.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In order to be able to compare test scores of individuals from different populations or groups in a meaningful and reliable way, it is essential for any diagnostic instrument used to provide measures that have the same meaning across those groups in question (e.g., Chen and Zhu 2012; Millsap and Kwok 2004; Wicherts 2016).”
From Measurement Invariance of the WISC-V across a Clinical Sample of Children and Adolescents with ADHD and a Matched Control Group (2024) · cites this paper as Wicherts 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Measurement Invariance, Factor Analysis and Factorial Invariance
Meredith, W. (1993) · Psychometrika · doi:10.1007/bf02294825 · Cited by 4,536 · cited 1×
“Similarly, testing of measurement invariance in the confirmatory factor analyses, where the term strict factorial invariance (Meredith, 1993) is common, has become increasingly popular in recent decades as a means of studying the fairness of cognitive ability batteries like the Wechsler scales.”
MethodByrne, B. M., Shavelson, R. J., & Muthén, B. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.105.3.456 · Cited by 4,115
Wechsler Adult Intelligence Scale--Third Edition
Wechsler, D. (2019) · PsycTESTS Dataset · doi:10.1037/t49755-000 · Cited by 2,327
Differential Item Functioning.
Gudgeon, A. C., Holland, P. W., & Wainer, H. (1994) · The Statistician · doi:10.2307/2348590 · Cited by 2,091
Massive IQ gains in 14 nations: What IQ tests really measure.
Flynn, J. R. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.101.2.171 · Cited by 1,918
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1061 · Cited by 1,674
Statistical Approaches to Measurement Invariance
Millsap, R. E. (2012) · doi:10.4324/9780203821961 · Cited by 1,554
McGrew, K. S. (2009) · Intelligence · doi:10.1016/j.intell.2008.08.004 · Cited by 1,354 · cited 2×
“The trouble perhaps starts already with the fact that any given cognitive item is strongly dependent on a number of cognitive processes in a complex multivariate fashion, covering a great deal of cognitive ability factors in healthy populations (McGrew, 2009) and perhaps even more factors in unhealthy populations.”
BackgroundThe mean IQ of Americans: Massive gains 1932 to 1978.
Flynn, J. R. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.29 · Cited by 1,206
The Clinical Neuropsychologist
(2013) · The Clinical Neuropsychologist · doi:10.1080/13854046.2013.852397 · Cited by 588
Item bias and item response theory
Mellenbergh, G. J. (1989) · International Journal of Educational Research · doi:10.1016/0883-0355(89)90002-5 · Cited by 553 · cited 1×
“Mellenbergh (1989) used this notion to define measurement invariance more generally in statistical terms.”
MethodMethodology Review: Statistical Approaches for Assessing Measurement Bias
Millsap, R. E., & Everson, H. T. (1993) · Applied Psychological Measurement · doi:10.1177/014662169301700401 · Cited by 491
Evaluating the Impact of Partial Factorial Invariance on Selection in Two Populations
Millsap, R. E., & Kwok, O.-M. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.1.93 · Cited by 414
When Does Measurement Invariance Matter?
Borsboom, D. (2006) · Medical Care · doi:10.1097/01.mlr.0000245143.08679.cc · Cited by 302 · cited 1×
“When tests are used for diagnoses and selection purposes, the issue of measurement invariance is crucial (Borsboom, 2006).”
MethodAre intelligence tests measurement invariant over time? Investigating the nature of the Flynn effect
Wicherts, J. M., Dolan, C. V., Hessen, D. J., Oosterveld, P., van Baal, G. C. M., Boomsma, D. I., & Span, M. M. (2004) · Intelligence · doi:10.1016/j.intell.2004.07.002 · Cited by 211 · cited 3×
“young adults taking the WAIS either in the 1960s or in the 2000s) were associated with failures of measurement invariance (Wicherts et al., 2004).”
BackgroundResultInfluentialSelf-citationWicherts, J. M., & Dolan, C. V. (2010) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2010.00182.x · Cited by 127 · cited 7×
“Wicherts and Dolan (2010) discuss numerous examples of potential reasons for intercept differences in confirmatory factor analyses of IQ batteries, and these include issues such as test-wiseness (familiarity with testing), test-taking strategies (e.g. tendencies to guess), familiarity with the test…”
BackgroundMethodResultInfluentialSelf-citationStereotype Threat and Group Differences in Test Performance: A Question of Measurement Invariance.
Wicherts, J. M., Dolan, C. V., & Hessen, D. J. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.89.5.696 · Cited by 126
Self-citationWicherts, J. M., Dolan, C. V., Carlson, J. S., & van der Maas, H. L. J. (2010) · Learning and Individual Differences · doi:10.1016/j.lindif.2009.12.001 · Cited by 102
Self-citationInvestigating Spearman's Hypothesis by Means of Multi-Group Confirmatory Factor Analysis
Dolan, C. V. (2000) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3501_2 · Cited by 99 · cited 5×
“and Dolan (2010) found a verbal meaning subtest to show lower averages among ethnic minorities, supposedly because of an additional ethnic difference on knowledge of Dutch words among ethnic minorities that went beyond the group difference on the latent ability factor on which this subtest loaded.”
BackgroundInfluentialSex differences on the Dutch WAIS-III
van der Sluis, S., Posthuma, D., Dolan, C. V., de Geus, E. J. C., Colom, R., & Boomsma, D. I. (2005) · Intelligence · doi:10.1016/j.intell.2005.08.002 · Cited by 97 · cited 1×
“In two studies (one in Spain and the other in the Netherlands), the WAIS-III was found to be partially measurement invariant across sex (Dolan et al., 2006; van der Sluis et al., 2006).”
BackgroundShuttleworth-Edwards, A. B. (2016) · The Clinical Neuropsychologist · doi:10.1080/13854046.2016.1204011 · Cited by 72
Are cognitive differences between immigrant and majority groups diminishing?
te Nijenhuis, J., de Jong, M., Evers, A., & van der Flier, H. (2004) · European Journal of Personality · doi:10.1002/per.511 · Cited by 62
Dolan, C. V., Colom, R., Abad, F. J., Wicherts, J. M., Hessen, D. J., & van de Sluis, S. (2006) · Intelligence · doi:10.1016/j.intell.2005.09.003 · Cited by 56 · cited 2×
“In two studies (one in Spain and the other in the Netherlands), the WAIS-III was found to be partially measurement invariant across sex (Dolan et al., 2006; van der Sluis et al., 2006).”
BackgroundSelf-citationNiileksela, C. R., Reynolds, M. R., & Kaufman, A. S. (2013) · Psychological Assessment · doi:10.1037/a0031175 · Cited by 55 · cited 2×
“other studies have concerned age differences on the Wechsler scales and found invariance to be generally tenable (Niileksela et al., 2013).”
BackgroundA knowledge-based theory of rising scores on “culture-free” tests.
Fox, M. C., & Mitchum, A. L. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0030155 · Cited by 50
Two failures of Spearman's hypothesis: The GATB in Holland and the JAT in South Africa
Dolan, C. (2004) · Intelligence · doi:10.1016/j.intell.2003.09.001 · Cited by 50 · cited 3×
“Here, the picture is quite mixed, with some U.S. studies supporting measurement invariance at the scale level (Dolan, 2000), and others, including some in South Africa and the Netherlands, showing failures of measurement invariance (Dolan et al., 2004; Wicherts & Dolan, 2010).”
BackgroundResultFactor invariance between genders on the Wechsler Intelligence Scale for Children–Fifth Edition
Chen, H., Zhang, O., Raiford, S. E., Zhu, J., & Weiss, L. G. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.05.020 · Cited by 46 · cited 2×
“In a recent study, the authors did conclude that the WISc-IV showed invariance with respect to gender (chen et al., 2015), highlighting that one should not overgeneralize (non) invariance but rather study it for all relevant versions and contexts.”
BackgroundMeasurement Invariance of WISC-IV across Normative and Clinical Samples
Chen, H., & Zhu, J. (2012) · Personality and Individual Differences · doi:10.1016/j.paid.2011.10.006 · Cited by 46 · cited 10×
“yes Chen and Zhu (2012) WiC-iV normative vs. clinical samples u.s. partial Dolan (2000) WisC-r Blacks and Whites in the u.s. yes Dolan, roorda, and Wicherts (2004) JaT south african ethnic groups no Dolan et al.”
BackgroundInfluentialTommasi, M., Pezzuti, L., Colom, R., Abad, F. J., Saggino, A., & Orsini, A. (2015) · Intelligence · doi:10.1016/j.intell.2015.02.005 · Cited by 35 · cited 3×
“…might show bias with respect to certain educational levels. one recent study found the Italian WAIS-r to be invariant across educational levels (Tommasi et al., 2015), whereas a study involving the Spanish WAIS-IV (Abad et al., in press) showed several subtests to display non-uniform bias with…”
BackgroundResultInfluentialReynolds, M. R., Ingram, P. B., Seeley, J. S., & Newby, K. D. (2013) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2013.06.029 · Cited by 34 · cited 3×
“(Chen & Zhu, 2012; reynolds et al., 2013). For instance, reynolds et al. (2013) found that several of the subtests of the WAIS-IV did not show measurement invariance when comparing normal samples to a sample with intellectual disabilities.”
BackgroundAbad, F. J., Sorrel, M. A., Román, F. J., & Colom, R. (2016) · Psychological Assessment · doi:10.1037/pas0000228 · Cited by 24
Ethnic Bias and Fairness in Personnel Selection: Evidence and Consequences
Evers, A., Nijenhuis, J. T., & der Flier, H. V. (2017) · Wiley eBooks · doi:10.1002/9781405164221.ch14 · Cited by 12
The Blackwell Handbook of Personnel Selection
(2005) · Blackwell Publishing Ltd eBooks · doi:10.1111/b.9781405117029.2005.00018.x · Cited by 5
Barbero (2014) · doi:10.1016/j · Cited by 1
Research works printed but not linked: 3
- 10.1037/0033-2909.105.3.456 · doi:10.1037/0033-2909.105.3.456 · Paper page
- Taylor N., 2008, Uses and abuses of intelligence. Studies advancing Spearman and Raven’s quest for non-arbitrary metrics, 371
- Wechsler D., 2014, Wechsler adult intelligence scale-fourth SA edition
Cited by
Persistence and Fade-Out of Educational-Intervention Effects: Mechanisms and Potential Solutions
Bailey, D. H., Duncan, G. J., Cunha, F., Foorman, B. R., & Yeager, D. S. (2020) · Psychological Science in the Public Interest · doi:10.1177/1529100620915848 · Cited by 284
Robitzsch, A., & Lüdtke, O. (2023) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2023.2191292 · Cited by 185
Risky business: Correlation and causation in longitudinal studies of skill development.
Bailey, D. H., Duncan, G. J., Watts, T. W., Clements, D. H., & Sarama, J. H. (2018) · American Psychologist · doi:10.1037/amp0000146 · Cited by 115
“Future assessments and research designs, including those that investigate measurement invariance across subgroups (Wicherts, 2016), are needed to investigate such possibilities.”
Wu, J., & Cheng, T. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.08.016 · Cited by 120
Avila, J. F., Rentería, M. A., Witkiewitz, K. A., Verney, S. P., Vonk, J. M. J., & Manly, J. J. (2019) · Neuropsychology · doi:10.1037/neu0000584 · Cited by 80
“This relationship must be independent to make meaningful comparisons of latent mean differences (Wicherts, 2016).”
Background
Metadata
- Authors
- Wicherts, J. M.
- Affiliations
- Tilburg University (NL)
- Corresponding author
- Wicherts, Jelte M.
- ORCID
- Wicherts, Jelte M.
- Venue
- The Clinical Neuropsychologist
- Publisher
- Informa UK Limited
- Published
- 2016-06-30
- Volume
- 30
- Issue
- 7
- Pages
- 1006-1016
- ISSN
- 0920-1637 (ISSN-L)
- 1385-4046
- 1744-4144
- OpenAlex
- W2471250754
- Cited by
- 89
- References
- 33
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 14 · 2023: 19 · 2022: 8 · 2021: 5 · 2020: 10 · 2019: 5 · 2018: 7 · 2017: 5
- FWCI
- 1.96
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- The Clinical Neuropsychologist (publishedVersion, cc-by-nc-nd)
- Research portal (Tilburg University) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (452-11-004)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- measurement invariance, neurocognitive testing, WAIS-III, WAIS-IV, subgroup differences, gender differences, ethnicity, socioeconomic status, educational background, test validity, reference values
- MeSH terms
- Adult, Cognition, Ethnicity, Female, Humans, Intelligence Tests, Male, Neuropsychological Tests, Wechsler Scales, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.