The DOI matches its Crossref record; 202 of the 202 references it cites were found in a registry or an index.
Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods.
Abstract
Since its establishment, psychology has struggled to find valid methods for studying thoughts and subjective experiences. Thirty years ago, Ericsson and Simon (1980) proposed that participants can give concurrent verbal expression to their thoughts (think aloud) while completing tasks without changing objectively measurable performance (accuracy). In contrast, directed requests for concurrent verbal reports, such as explanations or directions to describe particular kinds of information, were predicted to change thought processes as a consequence of the need to generate this information, thus altering performance. By comparing performance of concurrent verbal reporting conditions with their matching silent control condition, Ericsson and Simon found several studies demonstrating that directed verbalization was associated with changes in performance. In contrast, the lack of effects of thinking aloud was merely suggested by a handful of experimental studies. In this article, Ericsson and Simon's model is tested by a meta-analysis of 94 studies comparing performance while giving concurrent verbalizations to a matching condition without verbalization. Findings based on nearly 3,500 participants show that the "think-aloud" effect size is indistinguishable from zero (r = -.03) and that this procedure remains nonreactive even after statistically controlling additional factors such as task type (primarily visual or nonvisual). In contrast, procedures that entail describing or explaining thoughts and actions are significantly reactive, leading to higher performance than silent control conditions. All verbal reporting procedures tend to increase times to complete tasks. These results suggest that think-aloud should be distinguished from other methods in future studies. Theoretical and practical implications are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
James, W. (1913) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,226
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317 · cited 1×
“Wilson highlighted the role of introspection in the analysis of reasons for feelings and attitudes and even retracted some earlier claims by Nisbett and Wilson (1977): “To the extent that people’s responses are caused by the conscious self, they have privileged access to the actual causes of these…”
BackgroundStatistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology · doi:10.2307/2063712 · Cited by 6,834
Newell, A. (1972) · Cited by 6,222
Cited by
Handbook of Self-Regulation of Learning and Performance
(2017) · Routledge eBooks · doi:10.4324/9781315697048 · Cited by 1,816
Measuring Risk Literacy: The Berlin Numeracy Test
Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001819 · Cited by 745
Navigating Into the Future or Driven by the Past
Seligman, M. E. P., Railton, P., Baumeister, R. F., & Sripada, C. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691612474317 · Cited by 720
Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes
Leijten, M., & Van Waes, L. (2013) · Written Communication · doi:10.1177/0741088313491692 · Cited by 513
The think aloud method: what is it and how do I use it?
Eccles, D. W., & Arsal, G. (2017) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2017.1331501 · Cited by 421
Godfroid, A., Boers, F., & Housen, A. (2013) · Studies in Second Language Acquisition · doi:10.1017/s0272263113000119 · Cited by 328
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · Science · doi:10.1126/science.adq1814 · Cited by 354
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Baumeister, R. F., & Masicampo, E. J. (2010) · Psychological Review · doi:10.1037/a0019393 · Cited by 301
Lalmas, M., O’Brien, H., & Yom-Tov, E. (2015) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02289-0 · Cited by 192
When asking the question changes the ultimate answer: Metamemory judgments change memory
Mitchum, A. L., Kelley, C. M., & Fox, M. C. (2016) · Journal of Experimental Psychology: General · doi:10.1037/a0039923 · Cited by 148
Self-citationA meta-analysis and systematic review of reactivity to judgements of learning
Double, K. S., Birney, D. P., & Walker, S. A. (2018) · Memory · doi:10.1080/09658211.2017.1404111 · Cited by 120
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“However, prior work has pointed out that the set-up of a think-aloud study can cause people to behave differently than they would otherwise [9, 34, 90].”
User Experiences of Well-Being Chatbots
Kettle, L., & Lee, Y.-C. (2024) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208231162453 · Cited by 45
Temporal Assessment of Self-Regulated Learning by Mining Students’ Think-Aloud Protocols
Lim, L., Bannert, M., van der Graaf, J., Molenaar, I., Fan, Y., Kilgour, J., Moore, J., & Gašević, D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.749749 · Cited by 44
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · doi:10.31234/osf.io/xcwdn · Cited by 28
Blech, C., Gaschler, R., & Bilalić, M. (2020) · Thinking & Reasoning · doi:10.1080/13546783.2019.1685001 · Cited by 26
When and how seductive details harm learning. a study using cued retrospective reporting
Bender, L., Renkl, A., & Eitel, A. (2021) · Applied Cognitive Psychology · doi:10.1002/acp.3822 · Cited by 22
Glogger-Frey, I., Treier, A.-K., & Renkl, A. (2022) · Instructional Science · doi:10.1007/s11251-022-09577-6 · Cited by 19
Open Minded: Searching for Truth about the Unconscious Mind
Newell, B. R., & Shanks, D. R. (2023) · doi:10.7551/mitpress/14922.001.0001 · Cited by 17
Unawareness of Attitudes, Their Environmental Causes, and Their Behavioral Effects
Gawronski, B., & Corneille, O. (2025) · Annual Review of Psychology · doi:10.1146/annurev-psych-051324-031037 · Cited by 13
Lang, F., Kammerer, Y., Stürmer, K., & Gerjets, P. (2021) · European Journal of Psychology of Education · doi:10.1007/s10212-020-00471-8 · Cited by 12
Metadata
- Authors
- Fox, M. C., Ericsson, K. A., & Best, R.
- Venue
- Psychological Bulletin
- Published
- 2011-03
- Volume
- 137
- Issue
- 2
- Pages
- 316-344
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2077979711
- Cited by
- 424
- References
- 202
- FWCI
- 10.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- Florida State University Foundation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.