CoChat
Sign Up
Cited by 1,047Decision-Making and Behavioral EconomicsCognitive Abilities and TestingClinical Reasoning and Diagnostic Skills

The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks

Open access
Toplak, Maggie E., West, Richard F., Stanovich, Keith E.·2011·Memory & Cognition
DOI 10.3758/s13421-011-0104-1
Open this paper in CoChat — free accountOpen PDF

Abstract

The Cognitive Reflection Test (CRT; Frederick, 2005) is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response. In this study, we showed that the CRT is a more potent predictor of performance on a wide sample of tasks from the heuristics-and-biases literature than measures of cognitive ability, thinking dispositions, and executive functioning. Although the CRT has a substantial correlation with cognitive ability, a series of regression analyses indicated that the CRT was a unique predictor of performance on heuristics-and-biases tasks. It accounted for substantial additional variance after the other measures of individual differences had been statistically controlled. We conjecture that this is because neither intelligence tests nor measures of executive functioning assess the tendency toward miserly processing in the way that the CRT does. We argue that the CRT is a particularly potent measure of the tendency toward miserly processing because it is a performance measure rather than a self-report measure.

AI snapshot

Written from the paper’s full text.

TL;DR

The Cognitive Reflection Test (CRT) was a stronger predictor of performance on a broad battery of heuristics-and-biases and rational-thinking tasks than measures of cognitive ability, thinking dispositions, or executive functioning. It remained a unique predictor after statistically controlling for those other individual-difference measures, suggesting it taps miserly processing as a performance measure.

Key findings

  • The CRT had its highest zero-order correlation with the rational-thinking composite (*r* = .49), followed by the heuristics-and-biases composite (*r* = .42) and the cognitive ability index CA2 (*r* = .40).
  • In hierarchical regression predicting the rational-thinking composite, intelligence accounted for 17.3% of variance, executive function added 5.6%, thinking dispositions added 2.1%, and the CRT added 11.2% unique variance (*p* < .001); overall *R*² = .362.
  • For the heuristics-and-biases composite, the CRT again predicted the most unique variance (8.0%), with thinking dispositions next (4.0%). For syllogistic reasoning with belief bias, only intelligence and the CRT were significant unique predictors (5.1% and 6.4%, respectively).
  • Commonality analysis showed CRT unique variance = .108, cognitive ability unique variance = .074, and thinking dispositions unique variance = .016, with total variance explained in the rational-thinking composite = .356. CRT performance was low: mean = 0.7 items correct (*SD* = 0.93); 55.8% solved none and 6.6% solved all three.

Methods

  • A total of 346 participants (95 males, 251 females; mean age = 20.1 years, *SD* = 3.9), mostly first-year undergraduates, completed a single 2-h battery.
  • Measures included the 3-item CRT; WASI Vocabulary and Matrix Reasoning plus PASAT working memory as cognitive ability indices; 15 heuristics-and-biases tasks; syllogistic reasoning with belief bias; executive-function tasks (Trailmaking, Stroop, PASAT); and thinking-disposition scales (Actively Open-Minded Thinking, superstitious thinking, Consideration of Future Consequences).
  • Analyses used zero-order correlations, hierarchical regression, and commonality analysis.

Limitations

  • The paper states that the executive-functioning measures may have been “too thin and heterogeneous,” with only one task per construct, and that growing publicity/familiarity with CRT items could threaten its status as a performance measure. It also reports modest reliability for the heuristics-and-biases composite (split-half = .495; Cronbach’s alpha = .484).
  • Evident from the text: the sample was predominantly female, first-year undergraduates recruited at one large university, limiting generalizability; the design was correlational, so causal direction cannot be established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3758/s13421-011-0104-1

Cites

135 of 135 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 40 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The CRT was a unique predictor of performance on heuristics-and-biases tasks after other measures of individual differences were statistically controlled.

    “a series of regression analyses indicated that the CRT was a unique predictor of performance on heuristics-and-biases tasks.”
    Quote in the paperSubstantial claim
  • Findingown data

    In this study, the CRT was a more potent predictor of performance on a wide sample of tasks from the heuristics-and-biases literature than measures of cognitive ability, thinking dispositions, and executive functioning.

    “In this study, we showed that the CRT is a more potent predictor of performance on a wide sample of tasks from the heuristics-and-biases literature than measures of cognitive ability, thinking dispositions, and executive functioning.”
    Quote in the paperSubstantial claim
  • Findingown data

    The CRT accounted for substantial additional variance in heuristics-and-biases performance after other measures of individual differences had been statistically controlled.

    “It accounted for substantial additional variance after the other measures of individual differences had been statistically controlled.”
    Quote in the paperSubstantial claim
  • Findingown data

    In this study, cognitive ability correlated .40 with CRT performance.

    “and below we report a .40correlation between cognitive ability and CRT performance.”
    • Also stated in: § Results · plain
      “its correlation with CA2 (.40), the cognitive ability indicator that combined the WASI composite with working memory performance.”
    Quote in the paperSubstantial claim
  • Findingown data

    For the heuristics-and-biases task score, each of the four blocks was statistically significant (p < .001) when entered hierarchically.

    “Each of the four blocks was statistically significant (p < .001)when entered hierarchically.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 24
  • Classified as probably not a substantial claim. 40
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data

    Of the 105 possible correlations among the heuristics-and-biases tasks, 86 were in the positive direction, but only 39 were significantly so.

    “Of the 105 possible correlations among the heuristics-and-biases tasks, 86 were in the positive direction, but only 39 significantly so.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The CRT's ability to predict performance on rational-thinking tasks is not entirely due to its variance in common with cognitive ability nor due to its variance in common with executive functioning in addition to cognitive ability.

    “The results of this analysis clearly indicate that the CRT's ability to predict performance on rational-thinking tasks is not entirely due to its variance in common with cognitive abilitynor is it due to its variance in common with executive functioning in addition to cognitive ability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the analyses completed so far, the CRT was a very potent predictor and intelligence a moderate predictor.

    “In the analyses completed so far, the CRT was a very potent predictor and intelligence a moderate predictor.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The most difficult heuristics-and-biases task was the sample-size squash problem, answered correctly by only 15.6% of participants, and the easiest was the second gambler's fallacy problem, answered correctly by 92.2%.

    “The most difficult task was the sample-size squash problem, answered correctly by only 15.6% of the participants, and the easiest task was the second gambler's fallacy problem, which was answered correctly by 92.2% of the participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The CRT was the strongest correlate of the heuristics-and-biases tasks (r = .42) and was tied with CA2 (r = .36) as the most potent predictor of syllogistic reasoning with belief bias.

    “The CRT was the strongest correlate of the former (r = .42)and was tied with CA2 (r = .36)as the most potent predictor of the latter.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the syllogistic reasoning analysis, the CRT predicted the most unique variance, but the intelligence block predicted almost as much unique variance.

    “The CRT was once again the variable that predicted the most unique variance, but in this analysis the intelligence block predicted almost as much unique variance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most measures were more correlated with performance on the heuristics-and-biases tasks than with performance on syllogistic reasoning with belief bias; this was particularly true of the thinking dispositions measures (.16 to .24 versus .04 to .15).

    “Most measures were more correlated with performance on the heuristics-and-biases tasks than with performance on syllogistic reasoning with belief bias.This was particularly true of the thinking dispositions measures (.16 to .24versus .04 to .15).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The CRT displayed its highest correlation with the rational-thinking composite score (.49), followed by its correlation with the heuristics-and-biases composite (.42) and its correlation with CA2 (.40).

    “The CRT displayed its highest correlation with this composite score (.49), followed by its correlation with the heuristics-and-biases composite (.42) and its correlation with CA2 (.40), the cognitive ability indicator that combined the WASI composite with working memory performance.”
    • Also stated in: § Results · boosted
      “In terms of zero-order correlations, it is clear from Table 2 that the strongest correlate of performance on the rational-thinking composite score was, in fact, the CRT (r = .49).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In predicting the heuristics-and-biases task score, the CRT predicted the most unique variance (8.0%), and the thinking dispositions block was the next most potent unique predictor (4.0%).

    “The CRT was once again the variable that predicted the most unique variance (8.0%), but in this analysis, the thinking dispositions block was the next most potent unique predictor (4.0%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The CRT is moderately associated with both cognitive ability and rational-thinking skill.

    “The CRT is moderately associated with both cognitive ability and rational-thinking skill.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Framing Problem task was 62.7%.

    “| 11. Framing Problem | 62.7 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and WASI Matrix Reasoning was .32.

    “| 7. WASI Matrix Reasoning | .32 | .24 | .21 | .28 | (.76) | .16 | - | | | | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and Superstitious Thinking was .04.

    “| 13. Superstitious Thinking | .04 | .24 | .04 | .18 | .12 | .22 | .04 | .15 | .08 | .10 | .14 | .38 | - |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Gambler's Fallacy 2 task was 92.2%.

    “| 6. Gambler's Fallacy 2 | 92.2 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Regression to the Mean task was 40.2%.

    “| 4. Regression to the Mean | 40.2 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Covariation Detection task was 36.1%.

    “| 8. Covariation Detection | 36.1 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and Inhibition was .15.

    “| 10. Inhibition | .15 | .17 | .16 | .21 | .24 | .18 | .19 | .35 | .32 | - | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and CA2 (WASI Comp+ WM) was .40.

    “| 8. CA2 (WASI Comp+ WM) | .40 | .39 | .36 | .47 | (.81) | (.66) | (.71) | - | | | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and Actively Open-Minded Thinking was .10.

    “| 12. Actively Open-Minded Thinking | .10 | .18 | .12 | .19 | .21 | .25 | .07 | .17 | .07 | .09 | .02 | - | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and the WASI Composite was .32.

    “| 5. WASI Composite | .32 | .32 | .35 | .41 | - | | | | | | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The correlation between the Cognitive Reflection Test and Working Memory was .33.

    “| 11. Working Memory | .33 | .33 | .22 | .34 | .30 | .17 | .29 | (.80) | .41 | .36 | - | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Sample Size: Squash Problem task was 15.6%.

    “| 3. Sample Size: Squash Problem | 15.6 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Sunk Cost task was 64.2%.

    “| 14. Sunk Cost | 64.2 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Causal Base Rate task was 53.8%.

    “| 1. Causal Base Rate | 53.8 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Methodological Reasoning task was 27.2%.

    “| 9. Methodological Reasoning | 27.2 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and Set Shifting was .28.

    “| 9. Set Shifting | .28 | .15 | .12 | .17 | .27 | .11 | .30 | .40 | - | | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The correlation between the Cognitive Reflection Test and WASI Vocabulary was .17.

    “| 6. WASI Vocabulary | .17 | .24 | .32 | .35 | (.76) | - | | | | | | | |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Probability Matching task was 23.5%.

    “| 13. Probability Matching | 23.5 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Gambler's Fallacy 1 task was 69.4%.

    “| 5. Gambler's Fallacy 1 | 69.4 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Sample Size: Hospital Problem task was 28.6%.

    “| 2. Sample Size: Hospital Problem | 28.6 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Probabilistic Reasoning: Denominator Neglect task was 61.8%.

    “| 12. Probabilistic Reasoning: Denominator Neglect | 61.8 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Outcome Bias task was 72.0%.

    “| 15. Outcome Bias | 72.0 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    The correlation between the Cognitive Reflection Test and CFC was .10.

    “| 14. CFC | .10 | .16 | .15 | .19 | .26 | .23 | .16 | .21 | .06 | .04 | .03 | .21 | .12 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Conjunction Problem task was 19.1%.

    “| 7. Conjunction Problem | 19.1 |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The percentage of correct responses on the Bayesian Reasoning task was 22.3%.

    “| 10. Bayesian Reasoning | 22.3 |”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The Cognitive Reflection Test (CRT) is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response.

    “The Cognitive Reflection Test (CRT;Frederick, 2005) is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response.”
    • Also stated in: § Abstract · plain
      “The task is designed to measure the tendency to override a prepotent response alternative that is incorrect and to engage in further reflection that leads to the correct response.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Frederick, 2005 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Classic insight problems do not usually trigger an attractive alternative response, whereas the three CRT problems trigger a strong alternative response that must be overridden.

    “Classic insight problems (see Gilhooly & Fioratou, 2009;Gilhooly & Murphy, 2005) do not usually trigger an attractive alternative response.”
    • Also stated in: § Abstract · boosted
      “The three problems on the CRT are of interest to researchers working in the heuristics-and-biases tradition because a strong alternative response is initially primed and then must be overridden.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Gilhooly & Fioratou, 2009 source

      Not checked in the free run.

    • cites Gilhooly & Murphy, 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Oechssler, Roider, and Schmitz (2009) found the CRT to be related to the number of expected-value choices and the tendency to commit the conjunction fallacy.

    “Oechssler, Roider, and Schmitz (2009) found the CRT to be related to the number of expected-value choices and the tendency to commit the conjunction fallacy.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Oechssler, Roider, and Schmitz (2009) source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    All tests of intelligence or cognitive aptitude are optimal performance assessments, whereas measures of thinking dispositions are often assessed under typical performance conditions.

    “All tests of intelligence or cognitive aptitude are optimal performance assessments, whereas measures of thinking dispositions are often assessed under typical performance conditions (Ackerman & Heggestad, 1997;Cacioppo, Petty, Feinstein, & Jarvis, 1996;Norris & Ennis, 1989;Perkins, 1995;Sternberg, 2003;Zeidner & Matthews, 2000).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Ackerman & Heggestad, 1997 source

      Not checked in the free run.

    • cites Cacioppo, Petty, Feinstein, & Jarvis, 1996 source

      Not checked in the free run.

    • cites Norris & Ennis, 1989

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Obrecht, Chapman, and Gelman (2009) found no relation between CRT performance and the degree of encounter frequency bias.

    “In contrast, Obrecht, Chapman, and Gelman (2009) found no relation between CRT performance and the degree of encounter frequency bias.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Obrecht, Chapman, and Gelman (2009) source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Frederick (2005) observed that the CRT could predict performance on measures of temporal discounting, the tendency to choose high-expected-value gambles, and framing effects.

    “Frederick (2005) observed that with as few as three items, his CRT could predict performance on measures of temporal discounting, the tendency to choose high-expected-value gambles, and framing effects.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Frederick, 2005 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Frederick (2005) found that large numbers of highly select university students at MIT, Princeton, and Harvard were cognitive misers; they responded that the cost was 10 cents rather than the correct answer of 5 cents.

    “Frederick (2005) found that large numbers of highly select university students at MIT, Princeton, and Harvard were cognitive misers; they responded that the cost was 10 cents, rather than the correct answer... 5 cents.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Frederick, 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Consideration of Future Consequences (CFC) scale is a 12-item scale developed by Strathman, Gleicher, Boninger, and Scott Edwards (1994) to measure the extent to which individuals consider distant outcomes when choosing their present behavior.

    “The Consideration of Future Consequences (CFC) scale is a 12-item scale that was developed by Strathman, Gleicher, Boninger, and Scott Edwards (1994) to measure the extent to which individuals consider distant outcomes when choosing their present behavior.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Strathman, Gleicher, Boninger, and Scott Edwards (1994) source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Answering the problems on the CRT requires suppressing a prepotent natural response, which could be related to set-shifting and inhibitory processes assessed on measures of executive functioning.

    “As the bat/ball example described above illustrates, answering the problems on the CRT requires suppressing a prepotent”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kahneman, 2003 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The PASAT is a serial-addition task used to assess working memory, divided attention, and information-processing speed.

    “It is a serial-addition task used to assess working memory, divided attention, and informationprocessing speed (Gonzalez et al., 2006;Strauss, Sherman, & Spreen, 2006).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Gonzalez et al., 2006 source

      Not checked in the free run.

    • cites Strauss, Sherman, & Spreen, 2006

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The heuristics-and-biases composite score had modest reliability: split-half reliability was .495 and Cronbach's alpha was .484.

    “Thus, only modest reliability for the composite score was expected, and this was the case.The split-half reliability was .495,and Cronbach's alpha was .484.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The Actively Openminded Thinking scale had a split-half reliability of .78 and Cronbach's alpha of .81.

    “The split-half reliability of the scale (Spearman-Brown corrected) was .78,and Cronbach's alpha was .81.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Superstitious thinking has been found to predict probabilistic reasoning.

    “Superstitious thinking has been found to predict probabilistic reasoning (Kokis, Macpherson, Toplak, West, & Stanovich, 2002;Toplak, Liu, Macpherson, Toneatto, & Stanovich, 2007).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Kokis, Macpherson, Toplak, West, & Stanovich, 2002 source

      Not checked in the free run.

    • cites Toplak, Liu, Macpherson, Toneatto, & Stanovich, 2007 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Frederick (2005) reported a correlation of .44 between CRT performance and SAT total scores, and a .43 correlation between CRT scores and performance on the Wonderlic IQ test.

    “Frederick (2005) reported a correlation of .44 between CRT performance and SAT total scores, as well as a .43correlation between CRT scores and performance on the Wonderlic IQ test.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Frederick, 2005 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The superstitious thinking scale had a split-half reliability of .83 and Cronbach's alpha of .81.

    “The splithalf reliability of the scale (Spearman-Brown corrected) was .83,and Cronbach's alpha was .81.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Variance in CRT performance overlaps with both intelligence and rational-thinking ability.

    “The correlations displayed in Table 2 indicate that variance in CRT performance overlaps with both intelligence and rational-thinking ability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The CFC scale had a split-half reliability of .53 and Cronbach's alpha of .55.

    “The splithalf reliability of the scale (Spearman-Brown corrected) was .53,and Cronbach's alpha was .55.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Humans tend to be cognitive misers.

    “The processing problem is the one mentioned in our introductory discussion: that humans tend to be cognitive misers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The CRT is highly relevant to the idea of humans as cognitive misers.

    “In contrast, the CRT does seem highly relevant to the idea of humans as cognitive misers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    Many effects in the heuristics-and-biases literature are the results of the human tendency to default to miserly processing: anchoring biases, framing effects, preference reversals, nondisjunctive reasoning, myside biases, and status quo biases.

    “Many effects in the heuristicsand-biases literature are the results of the human tendency to default to miserly processing: anchoring biases, framing effects, preference reversals, nondisjunctive reasoning, myside biases, and status quo biases, to name just a few.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Intelligence and other cognitive ability tests leave out important domains of human cognition.

    “It has only recently been fully recognized that intelligence and other cognitive ability tests leave out important domains of human cognition (Stanovich, 2009b).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Stanovich, 2009b

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The CRT measures miserliness in action and is a direct measure of miserly processing rather than an indirect self-report indicator.

    “The CRT measures miserliness in action, so to speak.It is a direct measure of miserly processing rather than an indirect self-report indicator.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The CRT items were unfamiliar to the participants, so the CRT remains a performance measure rather than a self-report measure.
    • Rests on an assumption: Heuristics-and-biases tasks collectively measure a construct that can be termed rational thought.
57 of 57

  • The paper relies on this without support.

    The CRT items were unfamiliar to the participants, so the CRT remains a performance measure rather than a self-report measure.

    “Clearly, if individuals become familiar with the items, the CRT can no longer be considered a performance measure.”

  • The paper relies on this without support.

    Heuristics-and-biases tasks collectively measure a construct that can be termed rational thought.

    “Heuristics-and-biases tasks collectively measure a construct that we might term rational thought.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research has shown that populist attitudes, analytical thinking, numerical thinking, bullshit receptivity, conspiracy mentality, news checking frequency, and social media use are associated with susceptibility to misinformation.

    “we also included the following measures as covariates, which research has shown are associated with susceptibility to misinformation:”

    From Psychological inoculation improves resilience against misinformation on social media (2022) · cites this paper as [50]

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Ackerman, P. L. (1994). Intelligence, attention, and learning: Maximal and typical performance. In D. K. Detterman (Ed.), Current topics in human intelligence (Vol. 4, pp. 1–27). Norwood: Ablex.

Cited by

Sorted by citations

Related

Authors
Toplak, M. E., West, R. F., & Stanovich, K. E.
Affiliations
  • York University (CA)
  • James Madison University (US)
  • University of Toronto (CA)
Corresponding author
Toplak, Maggie E.
ORCID
Venue
Memory & Cognition
Publisher
Springer Science and Business Media LLC
Published
2011-05-04
Volume
39
Issue
7
Pages
1275-1289
ISSN
  • 0090-502X (ISSN-L)
  • 1532-5946
DOI
10.3758/s13421-011-0104-1
OpenAlex
W2081056419
Semantic Scholar
08d70bb4d91179f6660a4d9fceeb1fc2e9fb39db
Cited by
1,047
References
135
Citations by year
2026: 44 · 2025: 66 · 2024: 76 · 2023: 84 · 2022: 89 · 2021: 107 · 2020: 97 · 2019: 88 · 2018: 68 · 2017: 69 · 2016: 78 · 2015: 77 · 2014: 43 · 2013: 33 · 2012: 22
FWCI
17.96
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (tdm)
Locations
  • Memory & Cognition (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Canada Research Chairs
  • Social Sciences and Humanities Research Council of Canada
Topics
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Abilities and Testing
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
Cognitive Reflection Test, heuristics and biases, cognitive reflection, cognitive ability, executive function, thinking dispositions, individual differences, performance prediction
MeSH terms
Adult, Female, Humans, Intelligence, Male, Neuropsychological Tests, Predictive Value of Tests, Psychometrics, Regression Analysis, Thinking, Young Adult, Executive Function

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review