The DOI matches its Crossref record; 202 of the 202 references it cites were found in a registry or an index.
Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods.
Abstract
Since its establishment, psychology has struggled to find valid methods for studying thoughts and subjective experiences. Thirty years ago, Ericsson and Simon (1980) proposed that participants can give concurrent verbal expression to their thoughts (think aloud) while completing tasks without changing objectively measurable performance (accuracy). In contrast, directed requests for concurrent verbal reports, such as explanations or directions to describe particular kinds of information, were predicted to change thought processes as a consequence of the need to generate this information, thus altering performance. By comparing performance of concurrent verbal reporting conditions with their matching silent control condition, Ericsson and Simon found several studies demonstrating that directed verbalization was associated with changes in performance. In contrast, the lack of effects of thinking aloud was merely suggested by a handful of experimental studies. In this article, Ericsson and Simon's model is tested by a meta-analysis of 94 studies comparing performance while giving concurrent verbalizations to a matching condition without verbalization. Findings based on nearly 3,500 participants show that the "think-aloud" effect size is indistinguishable from zero (r = -.03) and that this procedure remains nonreactive even after statistically controlling additional factors such as task type (primarily visual or nonvisual). In contrast, procedures that entail describing or explaining thoughts and actions are significantly reactive, leading to higher performance than silent control conditions. All verbal reporting procedures tend to increase times to complete tasks. These results suggest that think-aloud should be distinguished from other methods in future studies. Theoretical and practical implications are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
James, W. (1913) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,226
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317 · cited 1×
“Wilson highlighted the role of introspection in the analysis of reasons for feelings and attitudes and even retracted some earlier claims by Nisbett and Wilson (1977): “To the extent that people’s responses are caused by the conscious self, they have privileged access to the actual causes of these…”
BackgroundStatistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Newell, A. (1972) · Cited by 6,222
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539
Essentials of Psychological Testing
Rosner, B., & Cronbach, L. J. (1960) · The American Journal of Psychology · doi:10.2307/1419921 · Cited by 4,513
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Self-citationMeta-Analytic Procedures for Social Research.
Metcalfe, A., & Rosenthal, R. (1994) · The Statistician · doi:10.2307/2348600 · Cited by 4,249
Decision making: A psychological analysis of conflict, choice, and commitment
(1978) · Evaluation and Program Planning · doi:10.1016/0149-7189(78)90042-3 · Cited by 3,946
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Self-citationThe Cambridge Handbook of Expertise and Expert Performance
(2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816796 · Cited by 3,158
Ericsson, K. A. (2004) · Academic Medicine · doi:10.1097/00001888-200410001-00022 · Cited by 2,988 · cited 1×
“…of the cognitive processes mediating experts’ superior performance in other domains (Ericsson, 2006; Ericsson & Williams, 2007), such as medicine (Ericsson, 2004, 2007), sport (Ward, Hodges, Starkes, & Williams, 2004), Scrabble (Tuffiash, Roring, & Ericsson, 2007), and music (Lehmann & Grüber,…”
BackgroundSelf-citationDuncker, K. (1945) · Psychological Monographs · doi:10.1037/h0093599 · Cited by 2,938
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
(2014) · doi:10.4324/9781315799681 · Cited by 2,174
EXPERT AND EXCEPTIONAL PERFORMANCE: Evidence of Maximal Adaptation to Task Constraints
Ericsson, K. A., & Lehmann, A. C. (1996) · Annual Review of Psychology · doi:10.1146/annurev.psych.47.1.273 · Cited by 2,139
Self-citationA Fail-Safe N for Effect Size in Meta-Analysis
Orwin, R. G. (1983) · Journal of Educational Statistics · doi:10.2307/1164923 · Cited by 2,064 · cited 1×
“Rosenthal (1979) and Orwin (1983) proposed fail-safe formulas that estimate the number of null studies that must have been overlooked to render an effect nonsignificant.”
BackgroundThe construction of preference.
Slovic, P. (1995) · American Psychologist · doi:10.1037/0003-066x.50.5.364 · Cited by 2,009
Cognitive Interviewing: A Tool for Improving Questionnaire Design
Willis, G. (2004) · Cited by 1,934
Parametric measures of effect size.
Rosenthal, R. (1994) · Cited by 1,847
A History of Experimental Psychology
Ruckmick, C. A., & Boring, E. G. (1930) · The American Journal of Psychology · doi:10.2307/1414897 · Cited by 1,808
Psychology as the behaviorist views it.
Watson, J. B. (1994) · Psychological Review · doi:10.1037//0033-295x.101.2.248 · Cited by 1,735
Meta-analysis of experiments with matched groups or repeated measures designs
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.170 · Cited by 1,693
Strangers to ourselves: discovering the adaptive unconscious
(2003) · Choice Reviews Online · doi:10.5860/choice.40-3712 · Cited by 1,645
Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00051.x · Cited by 1,629
Research synthesis: the practice of cognitive interviewing
Beatty, P. C., & Willis, G. B. (2007) · Public Opinion Quarterly · doi:10.1093/poq/nfm006 · Cited by 1,615
Thinking too much: Introspection can reduce the quality of preferences and decisions.
Wilson, T. D., & Schooler, J. W. (1991) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.60.2.181 · Cited by 1,417
On making the right choice: the deliberation-without-attention effect.
Dijksterhuis, A., Bos, M. W., Nordgren, L. F., & van Baaren, R. B. (2006) · Science · doi:10.1126/science.1121629 · Cited by 1,159
On the conflict between logic and belief in syllogistic reasoning
Evans, J. S. B. T., Barston, J. L., & Pollard, P. (1983) · Memory & Cognition · doi:10.3758/bf03196976 · Cited by 840
Exploring the Articulatory Loop
Baddeley, A., Lewis, V., & Vallar, G. (1984) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748408402157 · Cited by 836
Think Different: The Merits of Unconscious Thought in Preference Development and Decision Making.
Dijksterhuis, A. (2004) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.87.5.586 · Cited by 836
Verbal overshadowing of visual memories: Some things are better left unsaid
Schooler, J. W., & Engstler-Schooler, T. Y. (1990) · Cognitive Psychology · doi:10.1016/0010-0285(90)90003-m · Cited by 794
On the Relationship between Task Performance and Associated Verbalizable Knowledge
Berry, D. C., & Broadbent, D. E. (1984) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748408402156 · Cited by 759
Eliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
The power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 2×
“Several authors (Cohn & Becker, 2003; Hedges & Pigott, 2001; Muncer, Craigie, & Holmes, 2003) have noted that power is often ignored in meta-analysis.”
BackgroundMethodThoughts beyond words: When language overshadows insight.
Schooler, J. W., Ohlsson, S., & Brooks, K. (1993) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.122.2.166 · Cited by 694
Effects of schema acquisition and rule automation on mathematical problem-solving transfer.
Cooper, G., & Sweller, J. (1987) · Journal of Educational Psychology · doi:10.1037/0022-0663.79.4.347 · Cited by 688
Learning from worked-out examples: A study on individual differences
RENKL, A. (1997) · Cognitive Science · doi:10.1016/s0364-0213(99)80017-2 · Cited by 679
Protocol analysis: Verbal reports as data, Rev. ed.
Ericsson, K. A., & Simon, H. A. (1993) · Cited by 655
Self-citationGilhooly, K. J., Fioratou, E., Anthony, S. H., & Wynn, V. (2007) · British Journal of Psychology · doi:10.1111/j.2044-8295.2007.tb00467.x · Cited by 643
Self-Knowledge: Its Limits, Value, and Potential for Improvement
Wilson, T. D., & Dunn, E. W. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.141954 · Cited by 624
The validity of verbal protocols
Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989) · Memory & Cognition · doi:10.3758/bf03202637 · Cited by 543
Meta-Analyses of Fakability Estimates: Implications for Personality Measurement
Viswesvaran, C., & Ones, D. S. (1999) · Educational and Psychological Measurement · doi:10.1177/00131649921969802 · Cited by 529
Just, M. A., & Carpenter, P. A. (1985) · Psychological Review · doi:10.1037//0033-295x.92.2.137 · Cited by 509
How meta-analysis increases statistical power.
Cohn, L. D., & Becker, B. J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.243 · Cited by 498
Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003) · Journal of Educational Psychology · doi:10.1037/0022-0663.95.4.774 · Cited by 491
van den Haak, M., De Jong, M., & Jan Schellens, P. (2003) · Behaviour & Information Technology · doi:10.1080/0044929031000 · Cited by 486
Stochastically dependent effect sizes.
Gleser, L. J., & Olkin, I. (1994) · Cited by 485
Ericsson, K. A. (2006) · The Cambridge Handbook of Expertise and Expert Performance · doi:10.1017/cbo9780511816796.013 · Cited by 441
Self-citationContext, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence
Harrison, D. A., McLaughlin, M. E., & Coalter, T. M. (1996) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1996.0103 · Cited by 429
Cokely, E. T., & Kelley, C. M. (2009) · Judgment and Decision Making · doi:10.1017/s193029750000067x · Cited by 427
Outcome dependency and attention to inconsistent information.
Erber, R., & Fiske, S. T. (1984) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.47.4.709 · Cited by 412
A Comparison of Concurrent and Retrospective Verbal Protocol Analysis
Kuusela, H., & Paul, P. (2000) · The American Journal of Psychology · doi:10.2307/1423365 · Cited by 408
Boring, E. G. (1953) · Psychological Bulletin · doi:10.1037/h0090793 · Cited by 402
Spatial Visualization in Physics Problem Solving
Kozhevnikov, M., Motes, M. A., & Hegarty, M. (2007) · Cognitive Science · doi:10.1080/15326900701399897 · Cited by 400
The Psychology of Human Thought
Smith, E. E., & Sternberg, R. J. (2020) · Medical Entomology and Zoology · Cited by 387
Attention, Awareness, and Foreign Language Behavior
Leow, R. P. (2001) · Language Learning · doi:10.1111/j.1467-1770.2001.tb00016.x · Cited by 372
Zeitschrift für experimentelle und angewandte Psychologie
Psychologie, D. G. F. (1994) · Cited by 356
The road to excellence: deliberate practice and the development of expertise.
Ward, P., Hodges, N. J., Starkes, J. L., & Williams, M. A. (2007) · High Ability Studies · doi:10.1080/13598130701709715 · Cited by 353
Chrysikou, E. G., & Weisberg, R. W. (2005) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.31.5.1134 · Cited by 346
Irrelevant thoughts, emotional mood states, and cognitive task performance
Seibert, P. S., & Ellis, H. C. (1991) · Memory & Cognition · doi:10.3758/bf03199574 · Cited by 326
Using protocol analysis to evaluate the usability of a commercial web site
Benbunan-Fich, R. (2001) · Information & Management · doi:10.1016/s0378-7206(01)00085-4 · Cited by 324
Baumeister, R. F., & Masicampo, E. J. (2010) · Psychological Review · doi:10.1037/a0019393 · Cited by 301
LEARNERS' USES OF TWO TYPES OF WRITTEN FEEDBACK ON A L2 WRITING REVISION TASK
Sachs, R., & Polio, C. (2007) · Studies in Second Language Acquisition · doi:10.1017/s0272263107070039 · Cited by 293
A study of the effects of verbalization on problem solving.
Gagné, R. M., & Smith, E. C. (1962) · Journal of Experimental Psychology · doi:10.1037/h0048703 · Cited by 290
Students making sense of informational text: Relations between processing and representation
Coté, N., Goldman, S. R., & Saul, E. U. (1998) · Discourse Processes · doi:10.1080/01638539809545019 · Cited by 276
Ericsson, K. A. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02946.x · Cited by 271
Self-citationMetacognition and problem solving: A process-oriented approach.
Berardi-Coletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. (1995) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.21.1.205 · Cited by 269
Do applicants fake? An examination of the frequency of applicant faking behavior
Griffith, R. L., Chmielowski, T., & Yoshita, Y. (2007) · Personnel Review · doi:10.1108/00483480710731310 · Cited by 240
A meta‐analysis of the verbal overshadowing effect in face identification
Meissner, C. A., & Brigham, J. C. (2001) · Applied Cognitive Psychology · doi:10.1002/acp.728 · Cited by 236
Inflexibility of experts—Reality or myth? Quantifying the Einstellung effect in chess masters
Bilalić, M., McLeod, P., & Gobet, F. (2008) · Cognitive Psychology · doi:10.1016/j.cogpsych.2007.02.001 · Cited by 229
We talk, therefore we think? A cultural analysis of the effect of talking on thinking.
Kim, H. S. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.83.4.828 · Cited by 213
Effects of introspecting about reasons: Inferring attitudes from accessible thoughts.
Wilson, T. D., Hodges, S. D., & LaFleur, S. J. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.1.16 · Cited by 208
Similarity- versus rule-based categorization
Smith, E. E., & Sloman, S. A. (1994) · Memory & Cognition · doi:10.3758/bf03200864 · Cited by 202
Hypotheses about relations between scaled variables in the learning of probabilistic inference tasks
Brehmer, B. (1974) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(74)90002-6 · Cited by 200
Bannert, M., & Mengelkamp, C. (2008) · Metacognition and Learning · doi:10.1007/s11409-007-9009-6 · Cited by 198
Vickery, C. (2001) · Archives of Clinical Neuropsychology · doi:10.1016/s0887-6177(99)00058-x · Cited by 195
Differentiating insight from non-insight problems
Gilhooly, K., & Murphy, P. (2005) · Thinking & Reasoning · doi:10.1080/13546780442000187 · Cited by 192
Spann, M., & Skiera, B. (2009) · Journal of Forecasting · doi:10.1002/for.1091 · Cited by 190
Does having to justify one's judgments change the nature of the judgment process?
Hagafors, R., & Brehmer, B. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90122-8 · Cited by 180
Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?
Hertzum, M., Hansen, K. D., & Andersen, H. H. K. (2009) · Behaviour & Information Technology · doi:10.1080/01449290701773842 · Cited by 177
The effects of tasks on integrating information from multiple documents.
Cerdán, R., & Vidal-Abarca, E. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.1.209 · Cited by 177
Editorial–2: Computer assisted learning
Harden, R. M. (1986) · Medical Teacher · doi:10.3109/01421598609036839 · Cited by 176
The Effect of Common and Unique Features in Consumer Choice
Dhar, R., & Sherman, S. J. (1996) · Journal of Consumer Research · doi:10.1086/209477 · Cited by 170
Observational learning as a function of symbolization and incentive set.
Bandura, A., Grusec, J. E., & Menlove, F. L. (1966) · PubMed · Cited by 169
Duncker, K. (1926) · The Pedagogical Seminary and Journal of Genetic Psychology · doi:10.1080/08856559.1926.10533052 · Cited by 168
The use of verbal protocols as data: An analysis of insight in the candle problem
Fleck, J. I., & Weisberg, R. W. (2004) · Memory & Cognition · doi:10.3758/bf03196876 · Cited by 162
The behavioristic interpretation of consciousness. I.
Lashley, K. S. (1923) · Psychological Review · doi:10.1037/h0073839 · Cited by 158
At a Loss From Words: Verbal Overshadowing of Perceptual Memories
Schooler, J. W., Fiore, S. M., & Brandimonte, M. A. (1997) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60505-8 · Cited by 157
The influence of retrieval processes in verbal overshadowing
Meissner, C. A., Brigham, J. C., & Kelley, C. M. (2001) · Memory & Cognition · doi:10.3758/bf03195751 · Cited by 157
Is thinking merely the action of language mechanisms?
Watson, J. B. (2009) · British Journal of Psychology · doi:10.1348/000712608x336095 · Cited by 155
Estimation strategies and the judgment of event frequency.
Brown, N. R. (1995) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.21.6.1539 · Cited by 153
Acker, F. (2008) · Judgment and Decision Making · doi:10.1017/s1930297500000863 · Cited by 152
DeShon, R. P., Chan, D. K., & Weissbein, D. A. (1995) · Intelligence · doi:10.1016/0160-2896(95)90023-3 · Cited by 151
Decision Making: An Interdisciplinary Inquiry.
Meindl, J. R., Ungson, G. R., & Braunstein, D. N. (1984) · Administrative Science Quarterly · doi:10.2307/2393045 · Cited by 146
Systematic Mathematical Errors and Cognitive Load
Ayres, P. (2001) · Contemporary Educational Psychology · doi:10.1006/ceps.2000.1051 · Cited by 145
The Proper Protocol: Validity and Completeness of Verbal Reports
Wilson, T. D. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00621.x · Cited by 143
Skill in Map Reading and Memory for Maps
Gilhooly, K. J., Wood, M. J., Kinnear, P. R., & Green, C. (1988) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748808402284 · Cited by 142
Thinking; An Introduction to Its Experimental Psychology
Ogden, R. M., & Humphrey, G. (1953) · The American Journal of Psychology · doi:10.2307/1417991 · Cited by 141
LEARNERS' INTERPRETATIONS OF RECASTS
Carpenter, H. S., Jeon, K.-J., MacGregor, D., & Mackey, A. (2006) · Studies in Second Language Acquisition · doi:10.1017/s0272263106060104 · Cited by 140
Laing, S. P., & Kamhi, A. G. (2002) · Journal of Learning Disabilities · doi:10.1177/00222194020350050401 · Cited by 136
van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2004) · Interacting with Computers · doi:10.1016/j.intcom.2004.07.007 · Cited by 135
L2 Reading on the Web: Strategies for Accessing Information in Hypermedia
Chun, D. M. (2001) · Computer Assisted Language Learning · doi:10.1076/call.14.5.367.5775 · Cited by 131
Meta-Analysis of Bidirectional Relations in Personality-Job Performance Research
Tett, R. P., Jackson, D. N., Rothstein, M. G., & Reddon, J. R. (1999) · Human Performance · doi:10.1207/s15327043hup1201_1 · Cited by 128
Evaluating the usability of a mobile guide: The influence of location, participants and resources
Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S., & Davies, J. M. (2004) · Behaviour and Information Technology · doi:10.1080/01449290512331319030 · Cited by 127
Ericsson, K. A., & WILLIAMS, A. M. (2007) · Journal of Experimental Psychology Applied · doi:10.1037/1076-898x.13.3.115 · Cited by 126
Self-citationThink, blink or sleep on it? The impact of modes of thought on complex decision making
Newell, B. R., Wong, K. Y., Cheung, J., & Rakow, T. (2008) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210802215202 · Cited by 118
The validity of verbal reports in children's subtraction.
Robinson, K. M. (2001) · Journal of Educational Psychology · doi:10.1037//0022-0663.93.1.211 · Cited by 116
Ericsson, K. A. (2002) · Applied Cognitive Psychology · doi:10.1002/acp.925 · Cited by 113
Self-citationWindows on comprehension: Reading comprehension processes as revealed by two think-aloud procedures.
Crain-Thoreson, C., Lippman, M. Z., & McClendon-Magnuson, D. (1997) · Journal of Educational Psychology · doi:10.1037/0022-0663.89.4.579 · Cited by 112
Predicting Wimbledon 2005 tennis results by mere player name recognition
Scheibehenne, B., & Bröder, A. (2007) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2007.05.006 · Cited by 104
Predicting the World Cup 2002 in soccer: Performance and confidence of experts and non-experts
Andersson, P., Edman, J., & Ekman, M. (2005) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2005.03.004 · Cited by 103
Cognitive load and the imagination effect
Leahy, W., & Sweller, J. (2004) · Applied Cognitive Psychology · doi:10.1002/acp.1061 · Cited by 102
Predicting Soccer Matches After Unconscious and Conscious Thought as a Function of Expertise
Dijksterhuis, A., Bos, M. W., van der Leij, A. R., & van Baaren, R. B. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02451.x · Cited by 102
Pachur, T., & Biele, G. (2006) · Acta Psychologica · doi:10.1016/j.actpsy.2006.07.002 · Cited by 99
What underlies the deficit in reported recollective experience in old age?
Perfect, T. J., & Dasgupta, Z. R. R. (1997) · Memory & Cognition · doi:10.3758/bf03211329 · Cited by 98
Explanation and Verbalization in a Computer-Assisted Search Task
Berry, D. C., & Broadbent, D. E. (1987) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748708401804 · Cited by 95
The Effects of Aging and Dual Task Demands on Language Production
Kemper, S. J., Schmalzried, R., Herman, R. E., Leedahl, S. N., & Mohankumar, D. (2008) · Aging Neuropsychology and Cognition · doi:10.1080/13825580802438868 · Cited by 94
Exceptional memorizers: made, not born
Ericsson, K. A. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00103-7 · Cited by 94
Self-citationImplicit learning in problem solving: The role of working memory capacity.
Reber, P. J., & Kotovsky, K. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.2.178 · Cited by 93
The Influence of Author Visibility on High School Students Solving a Historical Problem
Paxton, R. J. (2002) · Cognition and Instruction · doi:10.1207/s1532690xci2002_3 · Cited by 93
Why do words hurt? Content, process, and criterion shift accounts of verbal overshadowing
Chin, J. M., & Schooler, J. W. (2008) · European Journal of Cognitive Psychology · doi:10.1080/09541440701728623 · Cited by 92
Binary choice under instructions to select versus reject
Meloy, M. G., & Russo, J. E. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2003.12.002 · Cited by 92
Directed search through autobiographical memory
Whitten, W. B., & Leonard, J. M. (1981) · Memory & Cognition · doi:10.3758/bf03202351 · Cited by 90
Chrysikou, E. G. (2006) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.32.4.935 · Cited by 90
Loxterman, J. A., Beck, I. L., & McKeown, M. G. (1994) · Reading Research Quarterly · doi:10.2307/747784 · Cited by 90
Concurrent Verbalizations, Pedagogical Conditions, and Reactivity: Two CALL Studies
Sanz, C., Lin, H., Lado, B., Bowden, H. W., & Stafford, C. A. (2009) · Language Learning · doi:10.1111/j.1467-9922.2009.00500.x · Cited by 89
The effect of conscious controlled verbalization cognitive strategy on transfer in problem solving
Ahlum-Heath, M. E., & Vesta, F. J. D. (1986) · Memory & Cognition · doi:10.3758/bf03197704 · Cited by 89
Evaluation of different modalities of verbalization in a sorting task
Hoc, J., & Leplat, J. (1983) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(83)80011-x · Cited by 88
Wilson, T. D., & Kraft, D. (1993) · Personality and Social Psychology Bulletin · doi:10.1177/0146167293194006 · Cited by 87
An experimental analysis of the mechanisms of a memory skill.
Ericsson, K. A., & Polson, P. G. (1988) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.14.2.305 · Cited by 86
Self-citationAn information-processing model of Duncker's Candle Problem
Weisberg, R. W., & Suls, J. M. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90014-5 · Cited by 80
The Role of Gestures in Mental Animation
Hegarty, M., Mayer, S. A., Kriz, S., & Keehner, M. (2005) · Spatial Cognition and Computation · doi:10.1207/s15427633scc0504_3 · Cited by 77
Motivated memory: Dissociation between performance data and subjective reports
Nilsson, L.-G. (1987) · Psychological Research · doi:10.1007/bf00308685 · Cited by 76
Ericsson, K. A. (2003) · Journal of Consciousness Studies · Cited by 76
Self-citationGonzalez-Vallejo, C. C., Lassiter, G. D., Bellezza, F. S., & Lindberg, M. J. (2008) · Review of General Psychology · doi:10.1037/a0013134 · Cited by 75
Meissner, C. A., Sporer, S. L., & Susa, K. J. (2008) · European Journal of Cognitive Psychology · doi:10.1080/09541440701728581 · Cited by 73
Tuffiash, M., Roring, R. W., & Ericsson, K. A. (2007) · Journal of Experimental Psychology Applied · doi:10.1037/1076-898x.13.3.124 · Cited by 72
Self-citationTASK TYPE AND REACTIVITY OF VERBAL REPORTS IN SLA: A First Look at a L2 Task Other Than Reading
Bowles, M. A. (2008) · Studies in Second Language Acquisition · doi:10.1017/s0272263108080492 · Cited by 71
Is self‐explanation while solving problems helpful? The case of analogical problem‐solving
Neuman, Y., & Schwarz, B. B. (1998) · British Journal of Educational Psychology · doi:10.1111/j.2044-8279.1998.tb01271.x · Cited by 70
Planning Processes and Age in the Five-disc Tower of London Task
Gilhooly, K. J., Phillips, L. H., Wynn, V., Logie, R. H., & Sala, S. D. (1999) · Thinking & Reasoning · doi:10.1080/135467899393977 · Cited by 70
The role of instruction and verbalization in improving performance on complex search tasks
Berry, D. C., & Broadbent, D. E. (1990) · Behaviour and Information Technology · doi:10.1080/01449299008924235 · Cited by 69
Response selection strategies and realism of confidence judgments
Allwood, C. M., & Montgomery, H. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90029-x · Cited by 65
The Effects of Concurrent Verbalization on Choice Processing
Biehal, G. J., & Chakravarti, D. (1989) · Journal of Marketing Research · doi:10.1177/002224378902600107 · Cited by 65
Skimming the Surface: Verbal Overshadowing of Analogical Retrieval
Lane, S. M., & Schooler, J. W. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00747.x · Cited by 64
Sucharowski, W. (1996) · VS Verlag für Sozialwissenschaften eBooks · doi:10.1007/978-3-322-95664-4 · Cited by 64
Biggs, S. F., Rosman, A. J., & Sergenian, G. K. (1993) · Journal of Behavioral Decision Making · doi:10.1002/bdm.3960060303 · Cited by 63
Fox, M. C., & Charness, N. (2010) · Aging, Neuropsychology, and Cognition · doi:10.1080/13825580903042668 · Cited by 61
Self-citationUnconscious-Thought Effects Take Place Off-Line, Not On-Line
Strick, M., Dijksterhuis, A., & van Baaren, R. B. (2010) · Psychological Science · doi:10.1177/0956797610363555 · Cited by 60
Roberts, V., & Fels, D. I. (2005) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.11.001 · Cited by 56
Verbalization and problem solving: Insight and spatial factors
Gilhooly, K. J., Fioratou, E., & Henretty, N. (2009) · British Journal of Psychology · doi:10.1348/000712609x422656 · Cited by 56
Culture and the cognitive and neuroendocrine responses to speech.
Kim, H. S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.32 · Cited by 53
Effect of Eliciting Verbal Reports of Thinking on Critical Thinking Test Performance
Norris, S. P. (1990) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1990.tb00733.x · Cited by 52
The effect of verbalization of cognitive processes on clinical decision making
Henry, S. B., Lebreck, D. B., & Holzemer, W. L. (1989) · Research in Nursing & Health · doi:10.1002/nur.4770120309 · Cited by 51
Generating Thinking-Aloud Protocols: Impact on the Narrative Writing of College Students
Ransdell, S. (1995) · The American Journal of Psychology · doi:10.2307/1423102 · Cited by 51
Blanc, N., Kendeou, P., van den Broek, P., & Brouillet, D. (2008) · Discourse Processes · doi:10.1080/01638530701792784 · Cited by 50
Methodological pitfalls of the Unconscious Thought paradigm
Waroquier, L., Marchiori, D. R., Klein, O., & Axel Cleeremans, B. (2009) · Judgment and Decision Making · doi:10.1017/s1930297500001169 · Cited by 49
It's Time We Brought Introspection Out of the Closet
Locke, E. A. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01090.x · Cited by 49
Strategy use on standardized reading comprehension tests.
Cordón, L. A., & Day, J. D. (1996) · Journal of Educational Psychology · doi:10.1037/0022-0663.88.2.288 · Cited by 48
WILLIAMS, A. M., & Davids, K. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200589 · Cited by 47
Dependent Correlations in Meta-Analysis: The Case of Heterogeneous Dependence
Cheung, S. F., & Chan, D. K. ‐. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408315263 · Cited by 47
Ziegler, M. D., Schmidt-Atzert, L., Bühner, M., & Krumm, S. (2007) · Psychology science · Cited by 45
The effects of concurrent verbalization on performance in a dynamic systems task
McGeorge, P., & Burton, A. M. (1989) · British Journal of Psychology · doi:10.1111/j.2044-8295.1989.tb02335.x · Cited by 45
The influence of verbal protocol methods on children's mental computation
Cooney, J. B., & Ladd, S. F. (1992) · Learning and Individual Differences · doi:10.1016/1041-6080(92)90004-x · Cited by 44
Schweiger, D. M. (1983) · Academy of Management Journal · doi:10.2307/256146 · Cited by 44
Effects of Concurrent Verbalization on a Time-Critical, Dynamic Decision-Making Task
Dickson, J. M., McLennan, J., & Omodei, M. M. (2000) · The Journal of General Psychology · doi:10.1080/00221300009598580 · Cited by 43
Rossomondo, A. (2007) · Studies in Second Language Acquisition · doi:10.1017/s0272263107070027 · Cited by 40
Thinking aloud during problem solving: Facilitation effects
Short, E. J., Evans, S. W., Friebert, S. E., & Schatschneider, C. W. (1991) · Learning and Individual Differences · doi:10.1016/1041-6080(91)90011-o · Cited by 38
Kobayashi, K. (2007) · Educational Psychology · doi:10.1080/01443410601104171 · Cited by 38
Verbalization, experimenter presence, and problem solving.
Davis, J. H., Carey, M. H., & Foxman, P. (1968) · Journal of Personality and Social Psychology · doi:10.1037/h0025519 · Cited by 38
Mental accounting and the process of multiattribute choice
Ranyard, R., & Nabi, D. A. (1993) · Acta Psychologica · doi:10.1016/0001-6918(93)90025-m · Cited by 37
The Thinking Aloud Technique and Problem Solving Ability
Flaherty, E. G. (1975) · The Journal of Educational Research · doi:10.1080/00220671.1975.10884753 · Cited by 36
Hu, Y., Ericsson, K. A., Yang, D., & Lu, C. (2009) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0017395 · Cited by 33
Self-citationOvert and covert verbalization in problem solving
Wilder, L., & Harvey, D. J. (1971) · Speech Monographs · doi:10.1080/03637757109375707 · Cited by 32
Some Evidence on the Effect of Verbalization on Process: A Methodological Note
Anderson, M. J. (1985) · Journal of Accounting Research · doi:10.2307/2490841 · Cited by 31
The influence of articulation, self‐monitoring ability, and sensitivity to others on creativity
de Vet, A. J., & Dreu, C. K. W. D. (2006) · European Journal of Social Psychology · doi:10.1002/ejsp.386 · Cited by 30
Meta-Analysis and Power: Some Suggestions for the Use of Power in Research Synthesis
Muncer, S., Craigie, M. A., & Holmes, J. (2003) · Understanding Statistics · doi:10.1207/s15328031us0201_01 · Cited by 29
Learning user interfaces by exploration
Mul, S. D., & van Oostendorp, H. (1996) · Acta Psychologica · doi:10.1016/0001-6918(95)00060-7 · Cited by 28
College Students and Algebra Story Problems: Strategies for Identifying Relevant Information
Cook, J. L. (2006) · Reading Psychology · doi:10.1080/02702710600640198 · Cited by 28
Blank-trial probes and introtacts in human discrimination learning.
Karpf, D., & Levine, M. (1971) · Journal of Experimental Psychology · doi:10.1037/h0031339 · Cited by 27
Learning by exploration: Thinking aloud while exploring an information system
van Oostendorp, H., & Mul, S. D. (1999) · Instructional Science · doi:10.1007/bf00897322 · Cited by 27
Fox, M. C., Roring, R. W., & Mitchum, A. L. (2008) · Intelligence · doi:10.1016/j.intell.2008.08.002 · Cited by 26
Self-citationThe influence of verbalization on problem‐solving
Stinessen, L. (1985) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.1985.tb01173.x · Cited by 26
Bower, A. C., & King, W. L. (1967) · Psychonomic Science · doi:10.3758/bf03332288 · Cited by 25
Assessment of imagery during covert modeling of assertive behavior
Kazdin, A. E. (1976) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(76)90002-1 · Cited by 25
An Implementation of Protocol Analysis and the Silent Dog Method in the Area of Behavioral Safety
Alvero, A. M., & Austin, J. (2006) · The Analysis of Verbal Behavior · doi:10.1007/bf03393027 · Cited by 22
Comparing of feedback-collection and think-aloud methods in program comprehension studies
Karahasanović, A., Hinkel, U. N., Sjberg, D., & Thomas, R. C. (2009) · Behaviour and Information Technology · doi:10.1080/01449290701682761 · Cited by 22
The role of verbal processing at different stages of recognition memory for faces
Nakabayashi, K., & Burton, A. M. (2008) · The European Journal of Cognitive Psychology · doi:10.1080/09541440801946174 · Cited by 22
Rewarded remembering: dissociations between self-rated motivation and memory performance
NGAOSUVAN, L., & MÄNTYLÄ, T. (2005) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.2005.00462.x · Cited by 19
Move Evaluation as a Predictor and Moderator of Success in Solutions to Well-Structured Problems
Davies, S. P. (2000) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755943 · Cited by 19
Attractiveness of decision alternatives characterized by numerical and non‐numerical information
Svenson, O., & KARLSSON, G. (1986) · Scandinavian Journal of Psychology · doi:10.1111/j.1467-9450.1986.tb01189.x · Cited by 17
A correlational study of two reasoning problems.
Brunk, L., Collister, E. G., Swift, C., & Stayton, S. (1958) · Journal of Experimental Psychology · doi:10.1037/h0046171 · Cited by 14
Cited by
Handbook of Self-Regulation of Learning and Performance
(2017) · Routledge eBooks · doi:10.4324/9781315697048 · Cited by 1,816
Measuring Risk Literacy: The Berlin Numeracy Test
Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001819 · Cited by 745
Navigating Into the Future or Driven by the Past
Seligman, M. E. P., Railton, P., Baumeister, R. F., & Sripada, C. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691612474317 · Cited by 720
Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes
Leijten, M., & Van Waes, L. (2013) · Written Communication · doi:10.1177/0741088313491692 · Cited by 513
The think aloud method: what is it and how do I use it?
Eccles, D. W., & Arsal, G. (2017) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2017.1331501 · Cited by 421
Metadata
- Authors
- Fox, M. C., Ericsson, K. A., & Best, R.
- Venue
- Psychological Bulletin
- Published
- 2011-03
- Volume
- 137
- Issue
- 2
- Pages
- 316-344
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2077979711
- Cited by
- 424
- References
- 202
- FWCI
- 10.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- Florida State University Foundation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.