CoChat
Sign Up
Cited by 19Cognitive Abilities and TestingMedical Education and AdmissionsPsychometric Methodologies and Testing

A critical review of the use of cognitive ability testing for selection into graduate and higher professional occupations

Open access
Woods, Stephen A., Patterson, Fiona·2024·Journal of Occupational and Organizational Psychology
DOI 10.1111/joop.12470
Open this paper in CoChat — free accountOpen PDF

Abstract

This article presents a critical review of the use of cognitive ability testing for access to graduate and higher professional occupations to promote further debate and reflection in both the academic and practitioner community. The main contentions are that the practice of applying cognitive ability testing in these contexts has strong potential to both maintain and exacerbate social inequality in access to higher occupations and professions, and that validity evidence does not justify this to the extent that has previously been presumed. Five critical observations are examined, namely (1) evidence of adverse impact in test outcomes; (2) the tendency to position cognitive ability testing early in selection processes in high‐volume recruitment; (3) recent evidence challenging the meta‐analytic validity of cognitive ability tests; (4) weaknesses in historical primary validity studies; (5) conceptually flawed examination of differential validity evidence in the literature. Implications for practice are discussed, contrasting strategies that involve modifying selection systems that include cognitive testing, versus removing and replacing cognitive tests.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/joop.12470

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 87 · Cited by 19

Research works printed but not linked: 77

  1. S Ackerman, P. L. (1987). Individual differences in skill learning: An integration of psychometric and information processing perspectives. Psychological Bulletin, 102(1), 3-27. https:// doi. org/ 10. 1037/ 0033-2909. 102.1. 3 · DOI
  2. Aguinis, H., Culpepper, S. A., & Pierce, C. A. (2010). Revival of test bias research in preemployment testing. Journal of Applied Psycholog y, 95(4), 648-680.
  3. Aguinis, H., Culpepper, S. A., & Pierce, C. A. (2016). Differential prediction generalization in college admissions testing. Journal of Educational Psycholog y, 108(7), 1045-1059.
  4. Austin, P. C., van Klaveren, D., Vergouwe, Y., Nieboer, D., Lee, D. S., & Steyerberg, E. W. (2016). Geographic and tempo- ral validity of prediction models: Different approaches were useful to examine model performance. Journal of Clinical Epidemiolog y, 79, 76-85.
  5. Berry, C. M., Clark, M. A., & McClure, T. K. (2011). Racial/ethnic differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review. Journal of Applied Psycholog y, 96(5), 881-906.
  6. Berry, C. M., & Zhao, P. (2015). Addressing criticisms of existing predictive bias research: Cognitive ability test scores still overpredict African Americans' job performance. Journal of Applied Psycholog y, 100(1), 162-179.
  7. Bertua, C., Anderson, N., & Salgado, J. F. (2005). The predictive validity of cognitive ability tests: A UK meta-analysis. Journal of Occupational and Organizational Psycholog y, 78(3), 387-409. doi:10.1348/096317905X26994 · DOI · Paper page
  8. Blanden, J. (2020). Education and inequality. In J. Jerrim, S. McNally, A. Björklund, J. Radl, M. Hällsten, & C. Rauh (Eds.), Handbook of the economics of education (pp. 119-131). Academic Press.
  9. Blanden, J., & Gregg, P. (2004). Family income and educational attainment: A review of approaches and evidence for Britain. Oxford Review of Economic Policy, 20(2), 245-263.
  10. Blanden, J., Katz, I., & Redmond, G. (2012). Persistent inequality? A comparison of the impact of family background on chil- dren's outcomes in the UK and Australia. In J. Ermisch, M. Jantti, T. Smeeding, & J. Wilson (Eds.), Inequality from childhood to adulthood: A cross-national perspective on the transmission of advantage. Russell Sage Foundation.
  11. Blanden, J., & Machin, S. (2010). Intergenerational inequality in early years assessments. In K. Hansen, H. Joshi, & S. Dex (Eds.), Children of the 21st century (pp. 153-168). Policy Press.
  12. Borghans, L., Golsteyn, B. H., Heckman, J. J., & Humphries, J. E. (2016). What grades and achievement tests measure. Proceedings of the National Academy of Sciences of the United States of America, 113(47), 13354-13359.
  13. Bunting, B. P., & Mooney, E. (2001). The effects of practice and coaching on test results for educational selection at eleven years of age. Educational Psycholog y, 21(3), 243-253.
  14. Busche, K., Elks, M. L., Hanson, J. T., Jackson-Williams, L., Manuel, R. S., Parsons, W. L., Wofsy, D., & Yuan, K. (2020). The validity of scores from the new MCAT exam in predicting student performance: Results from a multisite study. Academic Medicine, 95(3), 387-395.
  15. Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990). Modeling job performance in a population of jobs. Personnel Psycholog y, 43(2), 313-575.
  16. Clapp, J. T., & Gordon, E. K. (2022). Selecting trainees: Too much focus on predictive metrics, not enough on holistic review. Medical Education, 56(2), 139-141.
  17. Cleland, J., Alexander, K., & Nicholson, S. (2018). Widening access: Achieving equity in medical education, pp. In F. Patterson & L. Zibarras (Eds.), Selection and recruitment in the healthcare professions: Research, theory and practice (pp. 249-274). Palgrave Macmillan.
  18. Dahlke, J. A., Kostal, J. W., Sackett, P. R., & Kuncel, N. R. (2018). Changing abilities vs. changing tasks: Examining validity degradation with test scores and college performance criteria both assessed longitudinally. The Journal of Applied Psycholog y, 103(9), 980-1000.
  19. Dahlke, J. A., & Sackett, P. R. (2022). On the assessment of predictive bias in selection systems with multiple predictors. The Journal of Applied Psycholog y, 107(11), 1995-2012.
  20. Dahlke, J. A., Sackett, P. R., & Kuncel, N. R. (2019). Effects of range restriction and criterion contamination on differential validity of the SAT by race/ethnicity and sex. Journal of Applied Psycholog y, 104(6), 814-831.
  21. Dalal, R. S., Alaybek, B., & Lievens, F. (2020). Within-person job performance variability over short timeframes: Theory, em- pirical research, and practice. Annual Review of Organizational Psycholog y and Organizational Behavior, 7(1), 421-449. doi:10.1146/ annurev-orgpsych-012119-045350 · DOI
  22. De Corte, W., & Lievens, F. (2003). A practical procedure to estimate the quality and the adverse impact of single-stage selection decisions. International Journal of Selection and Assessment, 11(1), 89-97.
  23. De Corte, W., Lievens, F., & Sackett, P. R. (2006). Predicting adverse impact and mean criterion performance in multistage selection. Journal of Applied Psycholog y, 91(3), 523-537.
  24. De Corte, W., Lievens, F., & Sackett, P. R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psycholog y, 92(5), 1380-1393.
  25. DeShon, R. P., Smith, M. R., Chan, D., & Schmitt, N. (1998). Can racial differences in cognitive test performance be reduced by presenting problems in a social context? Journal of Applied Psycholog y, 83(3), 438-451. doi:10.1037/0021-9010.83.3.438 · DOI
  26. Fernald, A., Marchman, V. A., & Weisleder, A. (2013). SES differences in language processing skill and vocabulary are evident at 18 months. Developmental Science, 16(2), 234-248.
  27. Fortin, Y., Kulasegaram, K. M., Kancir, J. N., Moineau, G., Carpentier, A., & Hanson, M. D. (2018). The direct economic and opportunity costs of the medical college admissions test (MCAT) for Canadian medical students. MedEdPublish, 7(243), 243.
  28. Frey, M. C., & Detterman, D. K. (2004). Scholastic assessment or g? The relationship between the scholastic assessment test and general cognitive ability. Psychological Science, 15(6), 373-378.
  29. Gómez, J. M. (2023). Epistemic oppression, construct validity, and scientific rigor: Commentary on Woo et al.(2022). Perspectives on Psychological Science, 18(1), 32-38.
  30. Greatrix, R., Nicholson, S., & Anderson, S. (2021). Does the UKCAT predict performance in medical and dental school? A systematic review. BMJ Open, 11(1), e040128.
  31. Griebie, A., Bazian, I., Demeke, S., Priest, R., Sackett, P. R., & Kuncel, N. R. (2022). A contemporary look at the relationship be- tween cognitive ability and job performance. Poster presented at the 37th Annual Conference of the Society for Industrial and Organizational Psychology, Seattle, WA.
  32. Griffin, B., Carless, S., & Wilson, I. (2013). The effect of commercial coaching on selection test performance. Medical Teacher, 35(4), 295-300.
  33. Guenole, N., Englert, P., & Taylor, P. J. (2003). Ethnic group differences in cognitive ability test scores within a New Zealand applicant sample. New Zealand Journal of Psycholog y, 32, 49-54.
  34. Hajar, A. (2020). The association between private tutoring and access to grammar schools: Voices of year 6 pupils and teachers in south-East England. British Educational Research Journal, 46(3), 459-479.
  35. Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Moriarty Gerrard, M. O. (2007). Retesting in selection: A meta-anal- ysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psycholog y, 92(2), 373-385. doi:10.1037/0021-9010.92.2.373 · DOI
  36. Heckman, J. J. (2006). Skill formation and the economics of investing in disadvantaged children. Science, 312(5782), 1900-1902.
  37. Hickman, L., Herde, C. N., Lievens, F., & Tay, L. (2023). Automatic scoring of speeded interpersonal assessment center ex- ercises via machine learning: Initial psychometric evidence and practical guidelines. International Journal of Selection and Assessment, 31, 225-239.
  38. Hough, L. M., Oswald, F. L., & Ployhart, R. E. (2001). Determinants, detection and amelioration of adverse impact in personnel selection procedures: Issues, evidence and lessons learned. International Journal of Selection and Assessment, 9(1-2), 152-194. https:// doi. org/ 10. 1111/ 1468-2389. 00171 · DOI · Paper page
  39. Hulin, C. L., Henry, R. A., & Noon, S. L. (1990). Adding a dimension: Time as a factor in the generalizability of predictive relationships. Psychological Bulletin, 107, 328-340.
  40. Hunter, J. E., Schmidt, F. L., & Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psycholog y, 91(3), 594-612.
  41. Judge, T. A., Higgins, C. A., Thoresen, C. J., & Barrick, M. R. (1999). The big five personality traits, general mental ability, and career success across the life span. Personnel Psycholog y, 52(3), 621-652.
  42. Keil, C. T., & Cortina, J. M. (2001). Degradation of validity over time: A test and extension of Ackerman's model. Psychological Bulletin, 127(5), 673-697.
  43. Lievens, F., Ones, D. S., & Dilchert, S. (2009). Personality scale validities increase throughout medical school. Journal of Applied Psycholog y, 94(6), 1514-1535. doi:10.1037/a0016137 · DOI · Paper page
  44. Lievens, F., Patterson, F., Corstjens, J., Martin, S., & Nicholson, S. (2016). Widening access in selection using situational judge- ment tests: Evidence from the UKCAT. Medical Education, 50(6), 624-636.
  45. Lievens, F., Reeve, C. L., & Heggestad, E. D. (2007). An examination of psychometric bias due to retesting on cognitive ability tests in selection settings. Journal of Applied Psycholog y, 92(6), 1672-1682.
  46. Lievens, F., Sackett, P. R., & De Corte, W. (2022). Weighting admission scores to balance predictiveness-diversity: The Pareto- optimization approach. Medical Education, 56(2), 151-158.
  47. Locurto, C. (1990). The malleability of IQ as judged from adoption studies. Intelligence, 14(3), 275-292.
  48. Lu, B. (2018). Selection on attainment? Local authorities, pupil backgrounds, attainment and gram-mar school opportunities. Educational Review, 72, 68-87. https:// doi. org/ 10. 1080/ 00131 911. 2018. 1483893 · DOI
  49. Lucey, C. R., & Saguil, A. (2020). The consequences of structural racism on MCAT scores and medical school admissions. Academic Medicine, 95(3), 351-356. https:// doi. org/ 10. 1097/ acm. 00000000 · DOI
  50. McManus, I. C., Dewberry, C., Nicholson, S., & Dowell, J. S. (2013). The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sec- tional collaborative study of 12 UK medical schools. BMC Medicine, 11, 244. doi:10.1186/1741-7015-11-244 · DOI
  51. Nye, C. D., Ma, J., & Wee, S. (2022). Cognitive ability and job performance: Meta-analytic evidence for the validity of narrow cognitive abilities. Journal of Business and Psycholog y, 37(6), 1119-1139.
  52. Paton, L. W., McManus, I. C., Cheung, K. Y. F., Smith, D. T., & Tiffin, P. A. (2022). Can achievement at medical admission tests predict future performance in postgraduate clinical assessments? A UK-based national cohort study. BMJ Open, 12, e056129. https:// doi. org/ 10. 1136/ bmjop en-2021-056129 · DOI
  53. Patterson, F., Roberts, C., Hanson, M. D., Hampe, W., Eva, K., Ponnamperuma, G., Magzoub, M., Tekian, A., & Cleland, J. (2018). 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. Medical Teacher, 40(11), 1091-1101. doi:10.1080/0142159x.2018.1498589 · DOI
  54. Ployhart, R. E., & Holtz, B. C. (2008). The diversity-validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psycholog y, 61(1), 153-172. doi:10.1111/j.1744-6570.2008.00109.x · DOI
  55. Pyburn, K. M., Jr., Ployhart, R. E., & Kravitz, D. A. (2008). The diversity-validity dilemma: Overview and legal context. Personnel Psycholog y, 61(1), 143-151. doi:10.1111/j.1744-6570.2008.00108.x · DOI
  56. Reeve, C. L., & Bonaccio, S. (2011). On the myth and the reality of the temporal validity degradation of general mental ability test scores. Intelligence, 39(5), 255-272.
  57. Roe, R. A. (2014). Time, performance and motivation. In A. J. Shipp & Y. Fried (Eds.), Time and work, vol. 1. How time impacts individuals (pp. 63-110). Psychology Press.
  58. Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psycholog y, 52(1), 59-82.
  59. Roth, P. L., Le, H., Oh, I. S., Van Iddekinge, C. H., Buster, M. A., Robbins, S. B., & Campion, M. A. (2014). Differential validity for cognitive ability tests in employment and educational settings: Not much more than range restriction? Journal of Applied Psycholog y, 99(1), 1-20.
  60. Roth, P. L., Le, H., Oh, I.-S., Van Iddekinge, C. H., & Robbins, S. B. (2017). Who r u? On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research. Journal of Applied Psycholog y, 102(5), 802-828. https:// doi. org/ 10. 1037/ apl00 00193 · DOI
  61. Sackett, P. R., & Ellingson, J. E. (1997). The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psycholog y, 50(3), 707-721. https:// doi. org/ 10. 1111/j. 1744-6570. 1997. tb007 11. x Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. B. (2001). High-stakes testing in employment, credential- ing, and higher education: Prospects in a post-affirmative-action world. American Psychologist, 56(4), 302-318. doi:10.1037/0003-066X.56.4.302 · DOI · Paper page
  62. Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psycholog y, 107(11), 2040-2068. doi:10.1037/ · DOI · Paper page
  63. apl0000994 · DOI · Paper page
  64. Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & De Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European Community meta-analysis. Personnel Psycholog y, 56(3), 573-605.
  65. Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions. Advances in Health Sciences Education: Theory and Practice, 6, 159-175. doi:10.1023/A:1011489618208 · DOI
  66. Schleicher, D. J., Van Iddekinge, C. H., Morgeson, F. P., & Campion, M. A. (2010). If at first you don't succeed, try, try again: Understanding race, age, and gender differences in retesting score improvement. Journal of Applied Psycholog y, 95(4), 603-617.
  67. Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psycholog y, 86(1), 162-173.
  68. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274. https:// doi. org/ 10. 1037/ 0033- 2909. 124.2. 262 · DOI · Paper page
  69. Schmitt, N., Clause, C. S., & Pulakos, E. D. (1996). Subgroup differences associated with different measures of some common job-relevant constructs. International Review of Industrial and Organizational Psycholog y, 11, 115-140.
  70. Stegers-Jager, K. M., Steyerberg, E. W., Lucieer, S. M., & Themmen, A. P. N. (2014). Ethnic and social disparities in perfor- mance on medical school selection criteria. Medical Education, 49(1), 124-133. doi:10.1111/medu.12536 · DOI
  71. Stewart, S., & Haynes, C. (2015). An alternative approach to standardized testing: A model that promotes racial equity and college access. Journal of Critical Scholarship on Higher Education and Student Affairs, 2(1), 9.
  72. Strenze, T. (2007). Intelligence and socioeconomic success: A meta-analytic review of longitudinal research. Intelligence, 35(5), 401-426.
  73. te Nijenhuis, J., & van der Flier, H. (1997). Comparability of GATB scores for immigrants and majority group members: Some Dutch findings. Journal of Applied Psycholog y, 82, 675-687.
  74. Van Iddekinge, C. H., Morgeson, F. P., Schleicher, D. J., & Campion, M. A. (2011). Can I retake it? Exploring subgroup differ- ences and criterion-related validity in promotion retesting. Journal of Applied Psycholog y, 96(5), 941-955.
  75. Vincent, C., Braun, A., & Ball, S. J. (2008). Childcare, choice and social class: Caring for young children in the UK. Critical Social Policy, 28(1), 5-26.
  76. Woo, S. E., LeBreton, J. M., Keith, M. G., & Tay, L. (2023). Bias, fairness, and validity in graduate-school admissions: A psycho- metric perspective. Perspectives on Psychological Science, 18(1), 3-31.
  77. Woods, S. A., Ahmed, S., Nikolaou, I., Costa, A. C., & Anderson, N. R. (2020). Personnel selection in the digital age: A review of validity and applicant reactions, and future research challenges. European Journal of Work and Organizational Psycholog y, 29(1), 64-77. https:// doi. org/ 10. 1080/ 13594 32x. 2019. 1681401 · DOI · Paper page

Authors
Woods, S. A., & Patterson, F.
Affiliations
  • University of Surrey (GB)
  • Work Psychology Group Derby UK
Corresponding author
Woods, Stephen A.
ORCID
Venue
Journal of Occupational and Organizational Psychology
Publisher
Wiley
Published
2023-10-25
Volume
97
Issue
1
Pages
253-272
ISSN
  • 0963-1798 (ISSN-L)
  • 2044-8325
DOI
10.1111/joop.12470
OpenAlex
W4387937446
Cited by
19
References
87
Citations by year
2026: 6 · 2025: 8 · 2024: 5
FWCI
4.91
Type
journal article
Language
en
Open access
Licence
Locations
Journal of Occupational and Organizational Psychology (publishedVersion, cc-by)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
cognitive ability testing, graduate recruitment, selection validity, adverse impact, differential validity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review