The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Investigating an alternate form of the cognitive reflection test
Open accessAbstract
Much research in cognitive psychology has focused on the tendency to conserve limited cognitive resources. The CRT is the predominant measure of such miserly information processing, and also predicts a number of frequently studied decision-making traits (such as belief bias and need for cognition). However, many subjects from common subject populations have already been exposed to the questions, which might add considerable noise to data. Moreover, the CRT has been shown to be confounded with numeracy. To increase the pool of available questions and to try to address numeracy confounds, we developed and tested the CRT-2. CRT-2 questions appear to rely less on numeracy than the original CRT but appear to measure closely related constructs in other respects. Crucially, substantially fewer subjects from Amazon’s Mechanical Turk have been previously exposed to CRT-2 questions. Though our primary purpose was investigating the CRT-2, we also found that belief bias questions appear suitable as an additional source of new items. Implications and remaining measurement challenges are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2011) · Cited by 14,317
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Tests for comparing elements of a correlation matrix
Steiger, J. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.87.2.245 · Cited by 4,621
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007) · Child Development · doi:10.1111/j.1467-8624.2007.00995.x · Cited by 3,784
On the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913
Self-discipline outdoes IQ in predicting academic performance of adolescents.
Duckworth, A. L., & Seligman, M. E. P. (2005) · Psychological Science · doi:10.1111/j.1467-9280.2005.01641.x · Cited by 2,253
General Performance on a Numeracy Scale among Highly Educated Samples
Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001) · Medical Decision Making · doi:10.1177/0272989x0102100105 · Cited by 1,497
Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01720.x · Cited by 1,180
How numeracy influences risk comprehension and medical decision making.
Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F. (2009) · Psychological Bulletin · doi:10.1037/a0017327 · Cited by 1,168
The Role of Numeracy in Understanding the Benefit of Screening Mammography
Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-11-199712010-00003 · Cited by 1,130
The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks
Toplak, M. E., West, R. F., & Stanovich, K. E. (2011) · Memory & Cognition · doi:10.3758/s13421-011-0104-1 · Cited by 1,047
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Assessing miserly information processing: An expansion of the Cognitive Reflection Test
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014) · Thinking & Reasoning · doi:10.1080/13546783.2013.844729 · Cited by 864
Social heuristics shape intuitive cooperation
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 720
Bivariate median splits and spurious statistical significance.
Maxwell, S. E., & Delaney, H. D. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.113.1.181 · Cited by 665
Misleading Heuristics and Moderated Multiple Regression Models
Irwin, J. R., & McClelland, G. H. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.1.100.18835 · Cited by 611
Cognitive abilities and behavioral biases
Oechssler, J., Roider, A., & Schmitz, P. W. (2009) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2009.04.018 · Cited by 491
Cokely, E. T., & Kelley, C. M. (2009) · Judgment and Decision Making · doi:10.1017/s193029750000067x · Cited by 427
Negative Consequences of Dichotomizing Continuous Predictor Variables
Irwin, J. R., & McClelland, G. H. (2003) · Journal of Marketing Research · doi:10.1509/jmkr.40.3.366.19237 · Cited by 422
West, R. F., Toplak, M. E., & Stanovich, K. E. (2008) · Journal of Educational Psychology · doi:10.1037/a0012842 · Cited by 416
Reflection and Reasoning in Moral Judgment
Paxton, J. M., Ungar, L., & Greene, J. D. (2012) · Cognitive Science · doi:10.1111/j.1551-6709.2011.01210.x · Cited by 414
Development and Testing of an Abbreviated Numeracy Scale: A Rasch Analysis Approach
Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1751 · Cited by 392
Beyond Comprehension: The Role of Numeracy in Judgments and Decisions
Peters, E. (2012) · Current Directions in Psychological Science · doi:10.1177/0963721411429960 · Cited by 368
Baron, J., Scott, S., Fincher, K., & Emlen Metz, S. (2015) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2014.09.003 · Cited by 346
Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012) · Journal of Behavioral Decision Making · doi:10.1002/bdm.752 · Cited by 344
Kirkpatrick, L. A., & Epstein, S. (1992) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.63.4.534 · Cited by 334
Johnson-Laird, P. N., & Bara, B. G. (1984) · Cognition · doi:10.1016/0010-0277(84)90035-0 · Cited by 327
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 2×
“A recent analysis using capturerecapture analysis found that the average lab samples from a pool of about 7,300 MTurk workers (so a lab’s reach in practice is much smaller than the population of 50,000 advertised by Amazon; Stewart et al., 2015).”
BackgroundMethodPrimi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J. (2016) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1883 · Cited by 303
Journal of Behavioral Decision Making
(2006) · Journal of Behavioral Decision Making · doi:10.1002/(issn)1099-0771 · Cited by 279
Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach
Campitelli, G., & Gerrans, P. (2014) · Memory & Cognition · doi:10.3758/s13421-013-0367-9 · Cited by 259
Decision‐making Competence, Executive Functioning, and General Cognitive Abilities
Del Missier, F., Mäntylä, T., & de Bruin, W. B. (2012) · Journal of Behavioral Decision Making · doi:10.1002/bdm.731 · Cited by 248
Practice Effects on Commonly Used Measures of Executive Function Across Twelve Months
Basso, M. R., Bornstein, R. A., & Lang, J. M. (1999) · The Clinical Neuropsychologist · doi:10.1076/clin.13.3.283.1743 · Cited by 242
Making believers out of computers
Levesque, H. J. (1986) · Artificial Intelligence · doi:10.1016/0004-3702(86)90068-8 · Cited by 228
The belief-bias effect in the production and evaluation of logical conclusions
Markovits, H., & Nantel, G. (1989) · Memory & Cognition · doi:10.3758/bf03199552 · Cited by 217
Bringing meaning to numbers: The impact of evaluative categories on decisions.
Peters, E., Dieckmann, N. F., Västfjäll, D., Mertz, C. K., Slovic, P., & Hibbard, J. H. (2009) · Journal of Experimental Psychology: Applied · doi:10.1037/a0016978 · Cited by 203
Developing a tool for measuring the decision-making competence of older adults.
Finucane, M. L., & Gullion, C. M. (2010) · Psychology and Aging · doi:10.1037/a0019106 · Cited by 202 · cited 2×
“…questions were drawn from the work of others, including Markovits and Nantel (1989), De Neys and Franssens (2009), Johnson-Laird and Bara (1984), Finucane and Gullion (2010), Toplak and Stanovich (2002), Bockenhölt (2012), and Levesque (1986, 1989). whether the problems might be used to create…”
BackgroundMethodInfluentialBehavioral biases and cognitive reflection
Hoppe, E. I., & Kusterer, D. J. (2011) · Economics Letters · doi:10.1016/j.econlet.2010.11.015 · Cited by 199
Cognitive reflection vs. calculation in decision making
Sinayev, A., & Peters, E. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00532 · Cited by 198
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · cited 2×
“, 1999) and attenuate effect sizes (Chandler et al., 2015).”
BackgroundToplak, M. E., West, R. F., & Stanovich, K. E. (2014) · Developmental Psychology · doi:10.1037/a0034910 · Cited by 170 · cited 6×
“As in Toplak et al. (2014a), the syllogisms were adapted from Markovits and Nantel (1989; see Appendix C for items).”
BackgroundMethodInfluentialLogic and the Complexity of Reasoning
Levesque, H. J. (1989) · Philosophical Logic and Artificial Intelligence · doi:10.1007/978-94-009-2448-2_4 · Cited by 168
The diminishing criterion model for metacognitive regulation of time investment.
Ackerman, R. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0035098 · Cited by 152
Toplak, M. E., & Stanovich, K. E. (2002) · Journal of Educational Psychology · doi:10.1037/0022-0663.94.1.197 · Cited by 145
Belief inhibition during thinking: Not always winning but at least taking part
Neys, W. D., & Franssens, S. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.009 · Cited by 126
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Multiple Regression Analyses in Clinical Child and Adolescent Psychology
Jaccard, J. J., Guilamo‐Ramos, V., Johansson, M., & Bouris, A. M. (2006) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3503_11 · Cited by 117
The adolescent brain: Learning, reasoning, and decision making.
(2011) · American Psychological Association eBooks · doi:10.1037/13493-000 · Cited by 107
The persistence of the fluency–confidence association in problem solving
Ackerman, R., & Zalmanov, H. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0305-z · Cited by 102
Royzman, E. B., Landy, J. F., & Leeman, R. F. (2014) · Cognitive Science · doi:10.1111/cogs.12136 · Cited by 98
Probability matching and strategy availability
J. Koehler, D., & James, G. (2010) · Memory & Cognition · doi:10.3758/mc.38.6.667 · Cited by 85 · cited 1×
“…2011), Wason selection task performance, (Toplak et al., 2014a), using and endorsing maximizing strategies on probabilistic prediction tasks (Koehler & James, 2010), and a reduction in the degree to which people underweight sample size and standard deviation in making pairwise comparisons…”
MethodIntuitivet tests: Lay use of statistical information
Obrecht, N. A., Chapman, G. B., & Gelman, R. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03193104 · Cited by 84
The Cognitive-Miser Response Model: Testing for Intuitive and Deliberate Reasoning
Böckenholt, U. (2012) · Psychometrika · doi:10.1007/s11336-012-9251-y · Cited by 82
The Cognitive Reflection Test: how much more than Numerical Ability?
Welsh, M. B., Burns, N. R., & Delfabbro, P. (2013) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 73
Dichotomization, Partial Correlation, and Conditional Independence
Vargha, A., Rudas, T., Delaney, H. D., & Maxwell, S. E. (1996) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986021003264 · Cited by 68
An encounter frequency account of how experience affects likelihood estimation
Obrecht, N. A., Chapman, G. B., & Gelman, R. (2009) · Memory & Cognition · doi:10.3758/mc.37.5.632 · Cited by 54
Negative Consequences of Dichotomizing Continuous Predictor VariablesPreprint
McClelland, G. H., & Irwin, J. R. (2003) · SSRN Electronic Journal · Cited by 45
Graffeo, M., Polonio, L., & Bonini, N. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00844 · Cited by 41
Research works printed but not linked: 3
- 10.1037/0033-2909.113.1.181 · doi:10.1037/0033-2909.113.1.181 · Paper page
- Postrel, 2006, Would you take a bird in the hand, or a 75% chance of two in the bush?, The New York Times.
- 10.1037/0022-3514.63.4.534 · doi:10.1037/0022-3514.63.4.534 · Paper page
Cited by
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020) · Psychological Science · doi:10.1177/0956797620939054 · Cited by 1,715
Pennycook, G., & Rand, D. G. (2020) · Journal of Personality · doi:10.1111/jopy.12476 · Cited by 879
OSMUNDSEN, M., BOR, A., VAHLSTRUP, P. B., BECHMANN, A., & PETERSEN, M. B. (2021) · American Political Science Review · doi:10.1017/s0003055421000290 · Cited by 614
Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines
Bago, B., Rand, D. G., & Pennycook, G. (2019) · doi:10.31234/osf.io/29b4j · Cited by 593
Metadata
- Authors
- Thomson, K. S., & Oppenheimer, D. M.
- Venue
- Judgment and Decision Making
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-01
- Volume
- 11
- Issue
- 1
- Pages
- 99-113
- ISSN
- 1930-2975 (ISSN-L)
- OpenAlex
- W2302521008
- Cited by
- 609
- References
- 60
- FWCI
- 27.49
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.