Stereotype Threat and Group Differences in Test Performance: A Question of Measurement Invariance.
Open accessAbstract
Studies into the effects of stereotype threat (ST) on test performance have shed new light on race and sex differences in achievement and intelligence test scores. In this article, the authors relate ST theory to the psychometric concept of measurement invariance and show that ST effects may be viewed as a source of measurement bias. As such, ST effects are detectable by means of multi-group confirmatory factor analysis. This enables research into the generalizability of ST effects to real-life or high-stakes testing. The modeling approach is described in detail and applied to 3 experiments in which the amount of ST for minorities and women was manipulated. Results indicate that ST results in measurement bias of intelligence and mathematics tests.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788 · see citing works
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.6.1173 · Cited by 91,542 · see citing works
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532 · see citing works
Alternative Ways of Assessing Model Fit
Browne, M. W., & Cudeck, R. (1992) · Sociological Methods & Research · doi:10.1177/0049124192021002005 · Cited by 27,200 · see citing works
Comparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266 · see citing works
Structural Equations with Latent Variables
Bollen, K. A. (1989) · Wiley series in probability and statistics · doi:10.1002/9781118619179 · Cited by 17,512 · see citing works
Structural Equations with Latent Variables.
Clogg, C. C., & Bollen, K. A. (1991) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2072165 · Cited by 17,471 · see citing works
Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance
Cheung, G. W., & Rensvold, R. B. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0902_5 · Cited by 16,280 · see citing works
Testing Structural Equation Models.
Mueller, C. W., Bollen, K. A., & Long, J. S. (1994) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074955 · Cited by 14,211 · see citing works
Applied Multivariate Statistics for the Social Sciences
Stevens, J. P. (2012) · Routledge eBooks · doi:10.4324/9780203843130 · Cited by 10,652 · see citing works
A comparison of methods to test mediation and other intervening variable effects
MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.83 · Cited by 9,406 · see citing works
Testing Structural Equation Models.
Clogg, C. C., Bollen, K. A., & Long, J. S. (1995) · Social Forces · doi:10.2307/2580595 · Cited by 8,244 · see citing works
Stereotype threat and the intellectual test performance of African Americans.
Steele, C. M., & Aronson, J. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.69.5.797 · Cited by 8,181 · see citing works
LISREL 7 : user's reference guide
Jöreskog, K. G., & Sörbom, D. (1989) · Medical Entomology and Zoology · Cited by 6,783 · see citing works
A threat in the air: How stereotypes shape intellectual identity and performance
Steele, C. M. (1997) · American Psychologist · doi:10.1037/0003-066x.52.6.613 · Cited by 6,770 · see citing works
Measurement Invariance, Factor Analysis and Factorial Invariance
Meredith, W. (1993) · Psychometrika · doi:10.1007/bf02294825 · Cited by 4,536 · see citing works
Stereotype Threat and Women's Math Performance
Spencer, S. J., Steele, C. M., & Quinn, D. M. (1999) · Journal of Experimental Social Psychology · doi:10.1006/jesp.1998.1373 · Cited by 3,435 · see citing works
Alternative ways of assessing model fit
Browne, M. (1993) · Medical Entomology and Zoology · Cited by 2,595 · see citing works
The two disciplines of scientific psychology.
Cronbach, L. J. (1957) · American Psychologist · doi:10.1037/h0043943 · Cited by 2,556 · see citing works
Contending with group image: The psychology of stereotype and social identity threat
Steele, C. M., Spencer, S. J., & Aronson, J. (2002) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(02)80009-0 · Cited by 2,201 · see citing works
Simultaneous Factor Analysis in Several Populations
Jöreskog, K. G. (1971) · Psychometrika · doi:10.1007/bf02291366 · Cited by 2,051 · cited 1× · see citing works
“In case of several correlated common factors, a sufficient number of elements in i should be fixed to zero to avoid rotational indeterminacy (Bollen, 1989; Jo¨reskog, 1971).”
Applied Multivariate Statistics for the Social Sciences
Schumm, W. R., & Stevens, J. P. (1993) · The American Statistician · doi:10.2307/2685203 · Cited by 1,532 · see citing works
Converging evidence that stereotype threat reduces working memory capacity.
Schmader, T., & Johns, M. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.3.440 · Cited by 1,112 · see citing works
Sex Differences in Mental Test Scores, Variability, and Numbers of High-Scoring Individuals
Hedges, L. V., & Nowell, A. (1995) · Science · doi:10.1126/science.7604277 · Cited by 1,112 · see citing works
Applied Multivariate Statistics for the Social Sciences.
Brooks, S. P., & Stevens, J. P. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348967 · Cited by 1,035 · see citing works
When White Men Can't Do Math: Necessary and Sufficient Factors in Stereotype Threat
Aronson, J. M., Lustina, M. J., Good, C., Keough, K. A., Steele, C. M., & Brown, J. L. (1999) · Journal of Experimental Social Psychology · doi:10.1006/jesp.1998.1371 · Cited by 952 · see citing works
Advances in Psychology Research
Columbus, F. H., Shohov, S. P., & Columbus, A. M. (2001) · Medical Entomology and Zoology · Cited by 911 · see citing works
The effects of stereotype activation on behavior: A review of possible mechanisms.
Wheeler, S. C., & Petty, R. E. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.6.797 · Cited by 769 · see citing works
A GENERAL METHOD FOR STUDYING DIFFERENCES IN FACTOR MEANS AND FACTOR STRUCTURE BETWEEN GROUPS
Sörbom, D. (1974) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1974.tb00543.x · Cited by 679 · see citing works
African Americans and High Blood Pressure: The Role of Stereotype Threat
Blascovich, J., Spencer, S. J., Quinn, D. M., & Steele, C. M. (2001) · Psychological Science · doi:10.1111/1467-9280.00340 · Cited by 628 · see citing works
Beilock, S. L., & Carr, T. H. (2005) · Psychological Science · doi:10.1111/j.0956-7976.2005.00789.x · Cited by 596 · see citing works
Item bias and item response theory
Mellenbergh, G. J. (1989) · International Journal of Educational Research · doi:10.1016/0883-0355(89)90002-5 · Cited by 553 · see citing works
Methodology Review: Statistical Approaches for Assessing Measurement Bias
Millsap, R. E., & Everson, H. T. (1993) · Applied Psychological Measurement · doi:10.1177/014662169301700401 · Cited by 492 · see citing works
Raju, N. S., Laffitte, L. J., & Byrne, B. M. (2002) · Journal of Applied Psychology · doi:10.1037/0021-9010.87.3.517 · Cited by 428 · see citing works
Stereotype Threat and Arousal: Effects on Women's Math Performance
O’Brien, L. T., & Crandall, C. S. (2003) · Personality and Social Psychology Bulletin · doi:10.1177/0146167203029006010 · Cited by 413 · see citing works
Walton, G. M., & Cohen, G. L. (2003) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(03)00019-2 · Cited by 412 · see citing works
On specifying the null model for incremental fit indices in structural equation modeling.
Widaman, K. F., & Thompson, J. S. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.1.16 · Cited by 401 · see citing works
Solo status, stereotype threat, and performance expectancies: their effects on women's performance
Sekaquaptewa, D., & Thompson, M. (2003) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(02)00508-5 · Cited by 386 · see citing works
Seibt, B., & Förster, J. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.87.1.38 · Cited by 376 · see citing works
Testing Stereotype Threat: Does Anxiety Explain Race and Sex Differences in Achievement?
Osborne, J. W. (2001) · Contemporary Educational Psychology · doi:10.1006/ceps.2000.1052 · Cited by 359 · see citing works
Stigma on my mind: individual differences in the experience of stereotype threat
Brown, R. P., & Pinel, E. C. (2003) · Journal of Experimental Social Psychology · doi:10.1016/s0022-1031(03)00039-8 · Cited by 354 · see citing works
Quinn, D. M., & Spencer, S. J. (2001) · Journal of Social Issues · doi:10.1111/0022-4537.00201 · Cited by 333 · see citing works
Adjusting researchers’ approach to adjustment: On the use of covariates when testing interactions
Yzerbyt, V. Y., Müller, D., & Judd, C. M. (2003) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2003.10.001 · Cited by 332 · see citing works
Stereotype Threat Undermines Intellectual Performance by Triggering a Disruptive Mental Load
Croizet, J., Desprès, G., Gauzins, M.-E., Huguet, P., Leyens, J., & Méot, A. (2004) · Personality and Social Psychology Bulletin · doi:10.1177/0146167204263961 · Cited by 322 · see citing works
When Does Measurement Invariance Matter?
Borsboom, D. (2006) · Medical Care · doi:10.1097/01.mlr.0000245143.08679.cc · Cited by 303 · see citing works
Ben‐Zeev, T., Fein, S., & Inzlicht, M. (2004) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2003.11.007 · Cited by 296 · see citing works
The role of factorial invariance in modeling growth and change.
Meredith, W. H., & Horn, J. L. (2001) · American Psychological Association eBooks · doi:10.1037/10409-007 · Cited by 286 · see citing works
Are intelligence tests measurement invariant over time? Investigating the nature of the Flynn effect
Wicherts, J. M., Dolan, C. V., Hessen, D. J., Oosterveld, P., van Baal, G. C. M., Boomsma, D. I., & Span, M. M. (2004) · Intelligence · doi:10.1016/j.intell.2004.07.002 · Cited by 211 · see citing works
Self-citationSmith, J. L. (2004) · Educational Psychology Review · doi:10.1023/b:edpr.0000034020.20317.89 · Cited by 208 · see citing works
Inzlicht, M., & Ben‐Zeev, T. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.4.796 · Cited by 192 · see citing works
A Monte-Carlo Study of Confirmatory Factor Analytic Tests of Measurement Equivalence/Invariance
Meade, A. W., & Lautenschlager, G. J. (2004) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1101_5 · Cited by 188 · see citing works
Keller, J. (2002) · Sex Roles · doi:10.1023/a:1021003307511 · Cited by 185 · see citing works
Stricker, L. J., & Ward, W. C. (2004) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2004.tb02564.x · Cited by 184 · see citing works
Cullen, M. J., Hardison, C. M., & Sackett, P. R. (2004) · Journal of Applied Psychology · doi:10.1037/0021-9010.89.2.220 · Cited by 165 · see citing works
Smith, J. L., & White, P. H. (2002) · Sex Roles · doi:10.1023/a:1021051223441 · Cited by 145 · see citing works
Invariance in measurement and prediction: Their relationship in the single-factor case.
Millsap, R. E. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.3.248 · Cited by 142 · see citing works
Lubke, G. (2003) · Intelligence · doi:10.1016/s0160-2896(03)00051-5 · Cited by 122 · see citing works
Development of the Domain Identification Measure: A Tool for Investigating Stereotype Threat Effects
Smith, J. L., & White, P. H. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971635 · Cited by 108 · see citing works
Investigating Spearman's Hypothesis by Means of Multi-Group Confirmatory Factor Analysis
Dolan, C. V. (2000) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3501_2 · Cited by 99 · see citing works
Self-citationPloyhart, R. E., Ziegert, J. C., & McFarland, L. A. (2003) · Human Performance · doi:10.1207/s15327043hup1603_4 · Cited by 83 · see citing works
Dagevos, J., Gijsberts, M. I. L., & van Praag, C. (2003) · Data Archiving and Networked Services (DANS) · doi:10.48592/1097 · Cited by 80 · see citing works
Stereotype Threat and Employment Testing: A Commentary
Steele, C. M., & Davies, P. G. (2003) · Human Performance · doi:10.1207/s15327043hup1603_7 · Cited by 66 · see citing works
An Examination of Stereotype Threat in a Motivational Context
McFarland, L. A., Lev-Arey, D., & Ziegert, J. C. (2003) · Human Performance · doi:10.1207/s15327043hup1603_2 · Cited by 63 · see citing works
Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711003770480020 · Cited by 62 · see citing works
Self-citationNguyen, H.-H. D., O'Neal, A., & Ryan, A. M. (2003) · Human Performance · doi:10.1207/s15327043hup1603_5 · Cited by 56 · cited 2× · see citing works
“Similar to Study 1 above, stereotype threat was manipulated by both an ethnicity prime and by test diagnosticity (Nguyen et al., 2003).”
BackgroundResultGender Differences by Item Difficulty Interactions in Multiple-Choice Mathematics Items
Bielinski, J., & Davison, M. L. (1998) · American Educational Research Journal · doi:10.2307/1163444 · Cited by 54 · see citing works
Using restricted factor analysis to detect item bias
Oort, F. J. (1992) · Data Archiving and Networked Services (DANS) · Cited by 54 · see citing works
Two failures of Spearman's hypothesis: The GATB in Holland and the JAT in South Africa
Dolan, C. (2004) · Intelligence · doi:10.1016/j.intell.2003.09.001 · Cited by 50 · see citing works
Self-citationValidity of the Differential Aptitude Test for the Assessment of Immigrant Children
Nijenhuis, J. T., Evers, A., & Mur, J. P. (2000) · Educational Psychology · doi:10.1080/014434100110416 · Cited by 43 · see citing works
Stereotype Threat in Applied Selection Settings: A Commentary
Sackett, P. R. (2003) · Human Performance · doi:10.1207/s15327043hup1603_6 · Cited by 41 · see citing works
Verkuyten, M., & Thijs, J. T. (2004) · British Journal of Educational Psychology · doi:10.1348/000709904322848842 · Cited by 41 · see citing works
Dolan, C. V., & Hamaker, E. L. (2001) · Cited by 34 · see citing works
Self-citationStereotype Threat Research and the Assumptions Underlying Analysis of Covariance.
Wicherts, J. M. (2005) · American Psychologist · doi:10.1037/0003-066x.60.3.267 · Cited by 32 · see citing works
Self-citationBielinski, J., & Davison, M. L. (2001) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2001.tb01116.x · Cited by 32 · see citing works
Contextual aspects of ethnic stereotypes and interethnic evaluations
Kleinpenning, G., & Hagendoorn, L. (1991) · European Journal of Social Psychology · doi:10.1002/ejsp.2420210406 · Cited by 24 · see citing works
Donaldson, G. W. (2003) · Statistics in Medicine · doi:10.1002/sim.1558 · Cited by 20 · see citing works
The Relative Importance of Ethnicity: Ethnic Categorization among Older Children
Verkuyten, M., & Kinket, B. (1999) · International Journal of Psychology · doi:10.1080/002075999400005 · Cited by 16 · see citing works
The investigation of Spearman's hypothesis and the failure to understand factor analysis.
Millsap, R. E. (1997) · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Dagevos, 2003, Rapportage minder-heden 2003
- Evers, 1992, Handleiding DAT '83
- Joreskog, 2003, LISREL 8.5.
- Saris, 1993, Power evaluations in structural equation models., 181
- Stinissen, 1970, Wechsler Adult Intelligence Scale (WAIS)-Dutch Edition Test Manual.
- Thurstone, 1958, Manual for the SRA Primary Mental Abilities Test 11-17.
- Thurstone, 1962, Primary mental abilities for Grades 9-12.
- Wicherts, 2003, Stereotype threat and intelligence test scores of minority and majority high school students (Internal Report).
Cited by
Sorted by citations
Nguyen, H.-H. D., & Ryan, A. M. (2008) · Journal of Applied Psychology · doi:10.1037/a0012702 · Cited by 845 · see citing works
Developmental cognitive neuroscience using latent change score models: A tutorial and applications
Kievit, R. A., Brandmaier, A. M., Ziegler, G., van Harmelen, A.-L., de Mooij, S. M. M., Moutoussis, M., Goodyer, I. M., Bullmore, E., Jones, P. B., Fonagy, P., et al. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.11.007 · Cited by 516 · see citing works
Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis
Flore, P. C., & Wicherts, J. M. (2015) · Journal of School Psychology · doi:10.1016/j.jsp.2014.10.002 · Cited by 410 · see citing works
Self-citation<b>blavaan</b> : Bayesian Structural Equation Models via Parameter Expansion
Merkle, E. C., & Rosseel, Y. (2018) · Journal of Statistical Software · doi:10.18637/jss.v085.i04 · Cited by 360 · see citing works
Berry, C. M., Clark, M. A., & McClure, T. K. (2011) · Journal of Applied Psychology · doi:10.1037/a0023222 · Cited by 99 · see citing works
Shewach, O. R., Sackett, P. R., & Quint, S. (2019) · Journal of Applied Psychology · doi:10.1037/apl0000420 · Cited by 97 · see citing works
The Importance of Measurement Invariance in Neurocognitive Ability Testing
Wicherts, J. M. (2016) · The Clinical Neuropsychologist · doi:10.1080/13854046.2016.1205136 · Cited by 89 · see citing works
Self-citationComment on “Poverty Impedes Cognitive Function”
Wicherts, J. M., & Scholten, A. Z. (2013) · Science · doi:10.1126/science.1246680 · Cited by 38 · see citing works
Self-citationEdelsbrunner, P. A., Malone, S., Hofer, S. I., Küchemann, S., Kuhn, J., Schmid, R., Altmeyer, K., Brünken, R., & Lichtenberger, A. (2023) · International Journal of STEM Education · doi:10.1186/s40594-023-00435-6 · Cited by 19 · see citing works
Motivated cultural worldview adherence and culturally loaded test performance
Landau, M. J., Greenberg, J., & Rothschild, Z. K. (2009) · Personality and Social Psychology Bulletin · doi:10.1177/0146167208329630 · Cited by 17 · see citing works
Factors Related to Mean Score Group Differences on Cognitive Ability Tests: A Systematic Review
Cuppello, S., Zibarras, L. D., & Corr, P. J. (2025) · Trends in Psychology · doi:10.1007/s43076-025-00504-5
- Authors
- Wicherts, J. M., Dolan, C. V., & Hessen, D. J.
- Affiliations
- University of Amsterdam (NL)
- Venue
- Journal of Personality and Social Psychology
- Published
- 2005-11
- Volume
- 89
- Issue
- 5
- Pages
- 696-716
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2143055923
- Cited by
- 126
- References
- 78
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 6 · 2022: 7 · 2021: 5 · 2020: 2 · 2019: 10 · 2018: 5 · 2017: 8 · 2016: 8 · 2015: 8 · 2014: 4 · 2013: 10 · 2012: 10
- FWCI
- 7.95
- Type
- journal article
- Language
- en
- Open access
- green
- dspace.library.uu.nl
- Licence
- other-oa
- Locations
- Journal of Personality and Social Psychology (publishedVersion)
- PubMed (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Sociology and Political Science › Social and Intergroup Psychology
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Education, Achievement, and Giftedness
- Keywords
- stereotype threat, measurement invariance, measurement bias, intelligence test scores, mathematics scores, racial differences, sex differences, minority groups, women, high-stakes testing, multi-group confirmatory factor analysis
- MeSH terms
- Adolescent, Adult, Anxiety, Aptitude Tests, Factor Analysis, Statistical, Female, Humans, Intelligence Tests, Male, Mathematics, Minority Groups, Netherlands, Psychometrics, Stereotyping, Women, Models, Statistical, Bias, Multivariate Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review