The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)
Open accessAbstract
The cognitive reflection test (CRT) is a short measure of a person's ability to resist intuitive response tendencies and to produce a normatively correct response, which is based on effortful reasoning. Although the CRT is a very popular measure, its psychometric properties have not been extensively investigated. A major limitation of the CRT is the difficulty of the items, which can lead to floor effects in populations other than highly educated adults. The present study aimed at investigating the psychometric properties of the CRT applying item response theory analyses (a two‐parameter logistic model) and at developing a new version of the scale (the CRT‐long), which is appropriate for participants with both lower and higher levels of cognitive reflection. The results demonstrated the good psychometric properties of the original, as well as the new scale. The validity of the new scale was also assessed by measuring correlations with various indicators of intelligence, numeracy, reasoning and decision‐making skills, and thinking dispositions. Moreover, we present evidence for the suitability of the new scale to be used with developmental samples. Finally, by comparing the performance of adolescents and young adults on the CRT and CRT‐long, we report the first investigation into the development of cognitive reflection. Copyright © 2015 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
(2000) · Cited by 4,106
Fundamentals of item response theory
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4185 · Cited by 3,414
Marginal Maximum Likelihood Estimation of Item Parameters: Application of an EM Algorithm
Bock, R. D., & Aitkin, M. (1981) · Psychometrika · doi:10.1007/bf02293801 · Cited by 2,339
Gender differences in mathematics performance: A meta-analysis.
Hyde, J. S., Fennema, E., & Lamon, S. J. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.139 · Cited by 1,732
General Performance on a Numeracy Scale among Highly Educated Samples
Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100105 · Cited by 1,497
The basics of item response theory
Baker, F. B. (1985) · Cited by 1,362
On narrow norms and vague heuristics: A reply to Kahneman and Tversky.
Gigerenzer, G. (1996) · Psychological Review · doi:10.1037//0033-295x.103.3.592 · Cited by 1,070
The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks
Toplak, M. E., West, R. F., & Stanovich, K. E. (2011) · Memory & Cognition · doi:10.3758/s13421-011-0104-1 · Cited by 1,047
Proceedings of the 35th Annual Conference of the Cognitive Science Society
Dale, R. A. C., Yu, C., Nagai, Y., Coco, M. I., & Kopp, S. (2013) · Cited by 1,006
Measuring Numeracy without a Math Test: Development of the Subjective Numeracy Scale
Fagerlin, A., Zikmund-Fisher, B. J., Ubel, P. A., Jankovic, A., Derry, H. A., & Smith, D. M. (2007) · Medical Decision Making · doi:10.1177/0272989x07304449 · Cited by 960
Assessing miserly information processing: An expansion of the Cognitive Reflection Test
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014) · Thinking & Reasoning · doi:10.1080/13546783.2013.844729 · Cited by 864
Overcoming intuition: Metacognitive difficulty activates analytic reasoning.
Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.136.4.569 · Cited by 844
Local Dependence Indexes for Item Pairs Using Item Response Theory
Chen, W.-H., & Thissen, D. (1997) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165285 · Cited by 806
Embretson, S. E., & Diehl, K. A. (2000) · Encyclopedia of psychology, Vol. 4. · doi:10.1037/10519-153 · Cited by 595
Analytic cognitive style predicts religious and paranormal belief
Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (2012) · Cognition · doi:10.1016/j.cognition.2012.03.003 · Cited by 544
Cokely, E. T., & Kelley, C. M. (2009) · Judgment and Decision Making · doi:10.1017/s193029750000067x · Cited by 427
Bock, R. D., & Aitkin, M. (1982) · Psychometrika · doi:10.1007/bf02294168 · Cited by 415
Development and Testing of an Abbreviated Numeracy Scale: A Rasch Analysis Approach
Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1751 · Cited by 392
STANDARDIZATION OF PROGRESSIVE MATRICES, 1938
RAVEN, J. C. (1941) · British Journal of Medical Psychology · doi:10.1111/j.2044-8341.1941.tb00316.x · Cited by 389
Woodcock-Johnson Tests of Achievement
(2018) · The SAGE Encyclopedia of Intellectual and Developmental Disorders · doi:10.4135/9781483392271.n542 · Cited by 381
Modern Psychometrics: The Science of Psychological Assessment
Rust, J. N., & Golombok, S. E. (1989) · Cited by 378
Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012) · Journal of Behavioral Decision Making · doi:10.1002/bdm.752 · Cited by 344
Kokis, J. V., Macpherson, R., Toplak, M. E., West, R. F., & Stanovich, K. E. (2002) · Journal of Experimental Child Psychology · doi:10.1016/s0022-0965(02)00121-2 · Cited by 274
Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach
Campitelli, G., & Gerrans, P. (2014) · Memory & Cognition · doi:10.3758/s13421-013-0367-9 · Cited by 259
Math Performance in Stressful Situations
Beilock, S. L. (2008) · Current Directions in Psychological Science · doi:10.1111/j.1467-8721.2008.00602.x · Cited by 258
Analytic and Heuristic Processing Influences on Adolescent Reasoning and Decision-Making
Klaczynski, P. A. (2001) · Child Development · doi:10.1111/1467-8624.00319 · Cited by 230
Bats, balls, and substitution sensitivity: cognitive misers are no happy fools
De Neys, W., Rossi, S., & Houdé, O. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0384-5 · Cited by 201
Multivariate Statistical Methods: A First Course
Borror, C. M. (1998) · Journal of Quality Technology · doi:10.1080/00224065.1998.11979845 · Cited by 181
Morsanyi, K., Busdraghi, C., & Primi, C. (2014) · Behavioral and Brain Functions · doi:10.1186/1744-9081-10-31 · Cited by 144
Self-citationThinking about conditionals: A study of individual differences
Evans, J. S. B. T., Handley, S. J., Neilens, H., & Over, D. E. (2007) · Memory & Cognition · doi:10.3758/bf03193509 · Cited by 133
The conjunction fallacy and the many meanings of and
Hertwig, R., Benz, B., & Krauß, S. (2008) · Cognition · doi:10.1016/j.cognition.2008.06.008 · Cited by 132
The metacognitive advantage of deliberative thinkers: A dual-process perspective on overconfidence.
Mata, A., Ferreira, M. B., & Sherman, S. J. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0033640 · Cited by 114
Klaczynski, P. A. (2001) · Journal of Applied Developmental Psychology · doi:10.1016/s0193-3973(01)00085-5 · Cited by 107
Analytic thinking: do you feel like it?
Thompson, V. L., & Morsanyi, K. (2012) · Mind & Society · doi:10.1007/s11299-012-0100-6 · Cited by 106
Self-citationExplanation Fiends and Foes: How Mechanistic Detail Determines Understanding and Preference
Fernbach, P. M., Sloman, S. A., Louis, R. D. S., & Shube, J. N. (2012) · Journal of Consumer Research · doi:10.1086/667782 · Cited by 97
Gauffroy, C., & Barrouillet, P. (2009) · Developmental Review · doi:10.1016/j.dr.2009.09.002 · Cited by 83
Morsanyi, K., Devine, A., Nobes, A., & Szűcs, D. (2013) · Developmental Science · doi:10.1111/desc.12048 · Cited by 72
Self-citationKlaczynski, P. A., & Cottrell, J. M. (2004) · Thinking & Reasoning · doi:10.1080/13546780442000042 · Cited by 63
Evans, J. S. B. T. (2013) · Thinking & Reasoning · doi:10.1080/13546783.2013.845605 · Cited by 60
The Cognitive Reflection Test: how much more than Numerical Ability?
Welsh, M. B., Burns, N. R., & Delfabbro, P. (2013) · eScholarship (California Digital Library) · Cited by 60
Chiesi, F., Primi, C., & Morsanyi, K. (2011) · Thinking & Reasoning · doi:10.1080/13546783.2011.598401 · Cited by 59
Self-citationChiesi, F., Ciancaleoni, M., Galli, S., & Primi, C. (2012) · Psychological Assessment · doi:10.1037/a0027830 · Cited by 49
Self-citationLeavitt, V. M., & Weber, E. (2017) · Springer eBooks · doi:10.1007/978-3-319-56782-2_1870-2 · Cited by 36
Cognitive reflection test and behavioral biases in Malaysia
Albaity, M. S. A., Rahman, M., & Shahidul, I. (2014) · Judgment and Decision Making · doi:10.1017/s1930297500005507 · Cited by 28
Cited by
Investigating an alternate form of the cognitive reflection test
Thomson, K. S., & Oppenheimer, D. M. (2016) · Judgment and Decision Making · doi:10.1017/s1930297500007622 · Cited by 609
Calvillo, D. P., Ross, B. J., Garcia, R. J. B., Smelter, T. J., & Rutchick, A. M. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550620940539 · Cited by 514
Byrd, N., & Conway, P. (2019) · Cognition · doi:10.1016/j.cognition.2019.06.007 · Cited by 77
Applying novel technologies and methods to inform the ontology of self-regulation
Eisenberg, I. W., Bissett, P. G., Canning, J. R., Dallery, J., Enkavi, A. Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A. I., Greene, M. A., Kiernan, M., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.014 · Cited by 61
Pictures and repeated exposure increase perceived accuracy of news headlines
Smelter, T. J., & Calvillo, D. P. (2020) · Applied Cognitive Psychology · doi:10.1002/acp.3684 · Cited by 58
Metadata
- Authors
- Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J.
- Venue
- Journal of Behavioral Decision Making
- Publisher
- Wiley
- Published
- 2015-06-14
- Volume
- 29
- Issue
- 5
- Pages
- 453-469
- ISSN
- 0894-3257 (ISSN-L)
- 1099-0771
- OpenAlex
- W2150324751
- Cited by
- 303
- References
- 46
- FWCI
- 9.88
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.