CoChat
Sign Up
Cited by 344Cognitive Abilities and TestingEducation, Achievement, and GiftednessPsychological Testing and Assessment

RETRACTED: Role of test motivation in intelligence testing

Open access
Duckworth, Angela Lee, Quinn, Patrick D., Lynam, Donald R., Loeber, Rolf, Stouthamer-Loeber, Magda·2011·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1018601108PMC PMC3093513
Open this paper in CoChat — free account

Abstract

Intelligence tests are widely assumed to measure maximal intellectual performance, and predictive associations between intelligence quotient (IQ) scores and later-life outcomes are typically interpreted as unbiased estimates of the effect of intellectual ability on academic, professional, and social life outcomes. The current investigation critically examines these assumptions and finds evidence against both. First, we examined whether motivation is less than maximal on intelligence tests administered in the context of low-stakes research situations. Specifically, we completed a meta-analysis of random-assignment experiments testing the effects of material incentives on intelligence-test performance on a collective 2,008 participants. Incentives increased IQ scores by an average of 0.64 SD, with larger effects for individuals with lower baseline IQ scores. Second, we tested whether individual differences in motivation during IQ testing can spuriously inflate the predictive validity of intelligence for life outcomes. Trained observers rated test motivation among 251 adolescent boys completing intelligence tests using a 15-min “thin-slice” video sample. IQ score predicted life outcomes, including academic performance in adolescence and criminal convictions, employment, and years of education in early adulthood. After adjusting for the influence of test motivation, however, the predictive validity of intelligence for life outcomes was significantly diminished, particularly for nonacademic outcomes. Collectively, our findings suggest that, under low-stakes research conditions, some individuals try harder than others, and, in this context, test motivation can act as a third-variable confound that inflates estimates of the predictive validity of intelligence for life outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1018601108This paper is marked retracted.

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Psychometric tests are culturally specific, affected by education, motivation, and a wide range of other environmental factors, and improve with coaching and on re-testing.

    “Psychometric tests are culturallyspecific, affected by education, by motivation, as well as a wide range of other environmental factors, and improve with coaching and on re-testing”

    From ‘National IQ’ datasets do not provide accurate, unbiased or comparable measures of cognitive ability worldwide (2022) · cites this paper as Duckworth et al., 2011

    The Abstract reports that incentives increased IQ scores, supporting the claim that test performance is affected by motivation. The Discussion also identifies attentiveness, anxiety, emotional ability, and physical health as relevant to performance. Our copy does not establish that the tests are culturally specific or affected by education, nor does it report improvement with coaching or re-testing; its discussion of cross-cultural differences concerns test motivation, not the cultural specificity of the tests. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Motivation might partly account for differences in studies using untreated controls, and in treated-control studies where groups have different intervention expectancies, based on prior work linking motivation to intelligence test performance.

    “For studies in which untreated controls are used (and even in studies with treated controls where the subjects have different expectancies than the training group about the intervention) motivation could partially explain differences, given previous research showing effects of motivation on intelligence tests (Duckworth, Quinn, Lynam, Loeber, & Stouthmaer-Loeber, 2011).”

    From Working Memory Training Does Not Improve Performance on Measures of Intelligence or Other Measures of “Far Transfer” (2016) · cites this paper as Duckworth, Quinn, Lynam, Loeber, & Stouthmaer-Loeber, 2011

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. [2]Hogan R. In defense of personality measurement: New wine for old whiners. Hum Perform. 2005;18:331–341. [Google Scholar]
  2. [3]Jensen AR. The G Factor: The Science of Mental Ability. Westport, CT: Praeger/ Greenwood; 1998. [Google Scholar]
  3. [4]Judge TA, Colbert AE, Ilies R. Intelligence and leadership: A quantitative review and test of theoretical propositions. J Appl Psychol. 2004;89:542–552. doi: 10.1037/0021-9010.89.3.542. [DOI] [PubMed] [Google Scholar] · DOI · Paper page
  4. [5]Neisser U, et al. Intelligence: Knowns and unknowns. Am Psychol. 1996;51:77–101. [Google Scholar]
  5. [7]Sternberg RJ, Grigorenko EL, Bundy DA. The predictive value of IQ. Merrill-Palmer Q. 2001;47:1–41. [Google Scholar]
  6. [8]Boring EG. Intelligence as the tests test it. New Repub. 1923;35:35–37. [Google Scholar]
  7. [9]Wechsler D. Nonintellective factors in general intelligence. Psychol Bull. 1940;37:444–445. [Google Scholar]
  8. [11]Wechsler D. Wechsler Intelligence Scale for Children–Revised. San Antonio, TX: The Psychological Corporation; 1974. [Google Scholar]
  9. [12]MacNicol K. Effects of Varying Order of Item Difficulty in an Unspeeded Verbal Test. Princeton, NJ: Educational Testing Services; 1960. [Google Scholar]
  10. [13]Revelle W. Individual differences in personality and motivation: “non-cognitive” determinants of cognitive performance. In: Baddeley AD, Weiskrantz AD, editors. Attention: Selection, Awareness, and Control.A Tribute to Donald Broadbent. Oxford: Oxford Univ Press; 1993. pp. 346–373. [Google Scholar]
  11. [14]Arvey RD, Strickland W, Drauden G, Martin C. Motivational components of test taking. Person Psychol. 1990;43:695–716. [Google Scholar]
  12. [15]O'Neil HF, Jr., Sugrue B, Baker EL. Effects of motivational interventions on the National Assessment of Educational Progress mathematics performance. Educ Assess. 1995/1996;3:135–157. [Google Scholar]
  13. [16]Benton AL. Influence of incentives upon intelligence test scores of school children. J Genet Psychol. 1936;49:494–497. [Google Scholar]
  14. [17]Bergan A, McManis DL, Melchert PA. Effects of social and token reinforcement on WISC block design performance. Percept Mot Skills. 1971;32:871–880. doi: 10.2466/pms.1971.32.3.871. [DOI] [PubMed] [Google Scholar] · DOI
  15. [18]Blanding KM, Richards J, Bradley-Johnson S, Johnson CM. The effect of token reinforcement on McCarthy Scale performance for White preschoolers of low and high social position. J Behav Educ. 1994;4:33–39. [Google Scholar]
  16. [19]Bradley-Johnson S, Graham DP, Johnson CM. Token reinforcement on WISC–R performance for White, low-socioeconomic, upper and lower elementary-school-age students. J Sch Psychol. 1986;24:73–79. [Google Scholar]
  17. [20]Bradley-Johnson S, Johnson CM, Shanahan RH, Rickert VI, Tardona DR. Effects of token reinforcement on WISC-R performance of Black and White, low socioeconomic second graders. Behav Assess. 1984;6:365–373. [Google Scholar]
  18. [21]Breuning SE, Zella WF. Effects of individualized incentives on norm-referenced IQ test performance of high school students in special education classes. J Sch Psychol. 1978;16:220–226. [Google Scholar]
  19. [22]Clingman J, Fowler RL. The effects of primary reward on the I.Q. performance of grade-school children as a function of initial I.Q. level. J Appl Behav Anal. 1976;9:19–23. doi: 10.1901/jaba.1976.9-19. [DOI] [PMC free article] [PubMed] [Google Scholar] · DOI
  20. [23]Devers R, Bradley-Johnson S. The effect of token reinforcement on WISC-R performance for fifth- through ninth-grade American Indians. Psychol Rec. 1994;44:441–449. [Google Scholar]
  21. [24]Dickstein LS, Ayers J. Effect of an incentive upon intelligence test performance. Psychol Rep. 1973;33:127–130. [Google Scholar]
  22. [25]Edlund CV. The effect on the test behavior of children, as reflected in the I.Q. scores, when reinforced after each correct response. J Appl Behav Anal. 1972;5:317–319. doi: 10.1901/jaba.1972.5-317. [DOI] [PMC free article] [PubMed] [Google Scholar] · DOI
  23. [26]Ferguson HH. Incentives and an intelligence test. Australasian J Psychol Philos. 1937;15:39–53. [Google Scholar]
  24. [27]Galbraith G, Ott J, Johnson CM. The effects of token reinforcement on WISC-R performance of low-socioeconomic Hispanic second-graders. Behav Assess. 1986;8:191–194. [Google Scholar]
  25. [28]Gerwell EL. Hempstead, NY: Hofstra University; 1981. Tangible and verbal reinforcement effects on fluid and crystallized intelligence in the aged. PhD dissertation. [Google Scholar]
  26. [29]Graham GA. Washington, DC: George Washington University; 1971. The effects of material and social incentives on the performance on intelligence test tasks by lower class and middle class Negro preschool children. PhD dissertation. [Google Scholar]
  27. [30]Holt MM, Hobbs TR. The effects of token reinforcement, feedback and response cost on standardized test performance. Behav Res Ther. 1979;17:81–83. [Google Scholar]
  28. [31]Kapenis JT. Vermillion, SD: University of South Dakota; 1979. The differential effects of various reinforcements and socioeconomic status upon Peabody Picture Vocabulary Test performance. PhD dissertation. [Google Scholar]
  29. [32]Kieffer DA, Goh DS. The effect of individually contracted incentives on intelligence test performance of middle- and low-SES children. J Clin Psychol. 1981;37:175–179. [Google Scholar]
  30. [33]Lloyd ME, Zylla TM. Effect of incentives delivered for correctly answered items on the measured IQs of children of low and high IQ. Psychol Rep. 1988;63:555–561. [Google Scholar]
  31. [34]Saigh PA, Antoun FT. WISC-R incentives and the academic achievement of conduct disordered adolescent females: A validity study. J Clin Psychol. 1983;39:771–774. doi: 10.1002/1097-4679(198309)39:5<771::aid-jclp2270390523>3.0.co;2-u. [DOI] [PubMed] [Google Scholar] · DOI
  32. [35]Steinweg SB. Chapel Hill, NC: University of North Carolina; 1979. A comparison of the effects of reinforcement on intelligence test performance of normal and retarded children. PhD dissertation. [Google Scholar]
  33. [36]Sweet RC, Ringness TA. Variations in the intelligence test performance of referred boys of differing racial and socioeconomic backgrounds as a function of feedback or monetary reinforcement. J Sch Psychol. 1971;9:399–409. [Google Scholar]
  34. [37]Terrell F, Terrell SL, Taylor J. Effects of race of examiner and type of reinforcement on the intelligence test performance of lower-class Black children. Psychol Sch. 1980;17:270–272. doi: 10.1037//0022-006x.46.6.1538. [DOI] [PubMed] [Google Scholar] · DOI
  35. [38]Tiber N. Tallahassee, FL: Florida State University; 1963. The effects of incentives on intelligence test performance. PhD dissertation. [Google Scholar]
  36. [39]Weiss RH. Logan, UT: Utah State University; 1981. Effects of reinforcement on the IQ scores of preschool children as a function of initial IQ. PhD dissertation. [Google Scholar]
  37. [40]Willis J, Shibata B. A comparison of tangible reinforcement and feedback effects on the WPPSI I.Q. scores of nursery school children. Educ Treat Child. 1978;1:31–45. [Google Scholar]
  38. [41]Binet A, Simon T, Kite ES. The Development of Intelligence in Children (The Binet Simon Scale) Baltimore: Williams & Wilkins; 1916. [Google Scholar]
  39. [42]Thorndike EL. An Introduction to the Theory of Mental and Social Measurements. Oxford: Science Press; 1904. [Google Scholar]
  40. [43]Wintre MG, North C, Sugar LA. Psychologists‘ response to criticisms about research based on undergraduate participants: A developmental perspective. Can Psychol. 2001;42:216–225. [Google Scholar]
  41. [44]Herrnstein RJ, Murray CA. The Bell Curve: Intelligence and Class Struggle in American Life. New York: Free Press; 1994. [Google Scholar]
  42. [46]Snyderman M, Rothman S. Survey of expert opinion on intelligence and aptitude testing. Am Psychol. 1987;42:137–144. [Google Scholar]
  43. [48]Boe EE, May H, Boruch RF. Student Task Persistence in the Third International Mathematics and Science Study: A Major Source of Achievement Differences at the National, Classroom, and Student Levels. Philadelphia: Center for Research and Evaluation in Social Policy, University of Pennsylvania; 2002. Research Rep 2002-TIMSS1. [Google Scholar]
  44. [49]Haywood HC. The strange and wonderful symbiosis of motivation and cognition. Int J Cogn Ed Mediated Learn. 1992;2:186–197. [Google Scholar]
  45. [50]Borenstein M, Hedges L, Higgins J, Rothstein H. Comprehensive Meta-Analysis. Englewood, NJ: Biostat; 2005. Version 2. [Google Scholar]
  46. [52]Loeber R, Farrington DP, Stouthamer-Loeber M, Van Kammen WB. Antisocial Behavior and Mental Health Problems: Explanatory Factors in Childhood and Adolescence. Mahwah, NJ: Lawrence Erlbaum Associates; 1998. [Google Scholar]
  47. [53]Hobby KL. WISC-R Split-Half Short Form Manual. Los Angeles: Western Psychological Services; 1980. [Google Scholar]
  48. [54]Yudin LW. An abbreviated form of the WISC for use with emotionally disturbed children. J Consult Psychol. 1966;30:272–275. doi: 10.1037/h0023312. [DOI] [PubMed] [Google Scholar] · DOI
  49. [55]Silverstein AB. Notes on the reliability of Wechsler short forms. J Clin Psychol. 1990;46:194–196. doi: 10.1002/1097-4679(199003)46:2<194::aid-jclp2270460212>3.0.co;2-7. [DOI] [PubMed] [Google Scholar] · DOI
  50. [56]Lynam D, Moffitt TE, Stouthamer-Loeber M. Explaining the relation between IQ and delinquency: Class, race, test motivation, school failure, or self-control? J Abnorm Psychol. 1993;102:187–196. doi: 10.1037//0021-843x.102.2.187. [DOI] [PubMed] [Google Scholar] · DOI

Cited by

Sorted by citations

Authors
Duckworth, A. L., Quinn, P. D., Lynam, D. R., Loeber, R., & Stouthamer-Loeber, M.
Affiliations
  • University of Pennsylvania (US)
  • The University of Texas at Austin (US)
  • Purdue University West Lafayette (US)
  • University of Pittsburgh (US)
  • University of Pittsburgh
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2011-04-25
Volume
108
Issue
19
Pages
7716-7720
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1018601108
PMC
PMC3093513
OpenAlex
W2121281051
Cited by
344
References
6
Citations by year
2026: 4 · 2025: 12 · 2024: 21 · 2023: 20 · 2022: 32 · 2021: 26 · 2020: 24 · 2019: 30 · 2018: 29 · 2017: 22 · 2016: 32 · 2015: 19 · 2014: 28 · 2013: 20 · 2012: 15
FWCI
30.07
Type
journal article
Language
en
Status
Retracted
Update notices
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • SSRN Electronic Journal (submittedVersion)
Funders
  • National Institute on Aging (K01AG033182, R01AG032282)
  • National Institute of Mental Health (R01 MH-45070, MH 45070, R01MH045070)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Education, Achievement, and Giftedness
  • Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
Keywords
test-taking motivation, intelligence testing, IQ, predictive validity, material incentives, life outcomes, academic performance, criminal conviction, employment, educational attainment, adolescent boys, observer ratings, retraction
MeSH terms
Adolescent, Humans, Intelligence Tests, Male, Models, Psychological, Motivation, Bias, Test Taking Skills

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review