The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture
Open accessThe DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Like many other areas of science, experimental psychology is affected by a “replication crisis” that is causing concern in many fields of research. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals, and institutions. Here, I argue that if proposed solutions are to be effective, we also need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effect of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding; specific programmes directed at inoculating against cognitive biases; adoption of Registered Reports to encourage more critical reflection in planning studies; and using methods such as triangulation and “pre mortem” evaluation of study design to foster a culture of dialogue and criticism.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · cited 2×
“A clue to the persistence of underpowered psychology studies comes from early work by Tversky and Kahneman (1971, 1974).”
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
REMEMBERING: A STUDY IN EXPERIMENTAL AND SOCIAL PSYCHOLOGY
Bartlett, F. C., & Burt, C. (1933) · British Journal of Educational Psychology · doi:10.1111/j.2044-8279.1933.tb02913.x · Cited by 7,475
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Perception of the speech code.
Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967) · Psychological Review · doi:10.1037/h0020279 · Cited by 3,722
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 1×
“A clue to the persistence of underpowered psychology studies comes from early work by Tversky and Kahneman (1971, 1974).”
BackgroundPromoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 1×
“First, there have been calls for greater openness and transparency (Nosek et al., 2015).”
BackgroundSiddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Research works printed but not linked: 64
- Alexander S. (2019). 5-HTTLPR: A pointed review. Retrieved from https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/
- Babbage C. (1830). Reflections on the decline of science in England, and on some of its causes. London, England: B. Fellowes. Republished by Franklin Classics.
- 10.1098/rsos.170193 · DOI · Paper page
- Barber T. X. (1976). Pitfalls in human research. New York, NY: Pergamon Press.
- 10.1037/h0026724 · DOI · Paper page
- 10.1227/01.NEU.0000255391.92785.ED · doi:10.1227/01.neu.0000255391.92785.ed
- Bem D. J. (2004). Writing the empirical journal article. In Darley J. M., Zanna M. P., Roediger H. L. (Eds.), The compleat academic: A career guide (2nd ed., pp. 185–219). Washington, DC: American Psychological Association.
- 10.1111/j.1601-183X.2007.00360.x · DOI · Paper page
- 10.1080/14640748908402356 · DOI
- 10.1126/science.2403694 · DOI · Paper page
- 10.1038/d41586-019-02674-6 · DOI · Paper page
- Dawkins R. (1976). The selfish gene. Oxford, UK: Oxford University Press.
- 10.1126/science.1194140 · DOI · Paper page
- 10.1038/s41598-019-42515-0 · DOI
- 10.1017/S0033291718001873 · DOI · Paper page
- 10.1016/0001-6918(79)90017-9 · DOI · Paper page
- 10.2466/pr0.1970.27.3.835 · DOI
- 10.1136/bmj.b2680 · DOI · Paper page
- Hahn U., Harris A. J. L. (2014). What does it mean to be biased: Motivated reasoning and rationality. In Ross B. (Ed.), The psychology of learning and motivation (Vol. 61, pp. 41–101). San Diego, CA: Elsevier Academic Press.
- 10.1002/(SICI)1099-0992(199603)26:2<201::AID-EJSP745>3.0.CO;2-J · DOI
- 10.1177/00222194050380030401 · DOI
- 10.1016/j.cortex.2016.03.019 · DOI · Paper page
- Hossenfelder S. (2018). Lost in math: How beauty leads physics astray. New York, NY: Basic Books.
- International Committee of Medical Journal Editors. (2019). Recommendations: Clinical trials. Retrieved from http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html
- 10.1007/s10802-013-9741-0 · DOI
- 10.1038/nmeth.2699 · DOI
- 10.31234/osf.io/shm8v · doi:10.31234/osf.io/shm8v
- 10.1017/9781139696647 · DOI
- 10.1177/0023677218767478 · DOI
- 10.1037/bul0000142 · DOI · Paper page
- Levenson T. (2015). The hunt for Vulcan: . . . And how Albert Einstein destroyed a planet, discovered relativity, and deciphered the universe. New York, NY: Random House.
- Mahoney M. J. (1976). Scientist as subject: The psychological imperative. Cambridge, MA: Ballinger.
- 10.1177/2372732215600886 · DOI · Paper page
- 10.1093/ntr/ntx081 · DOI · Paper page
- 10.1136/bmj.295.6599.656 · DOI
- 10.7554/eLife21451.001 · doi:10.7554/elife21451.001
- 10.2307/27757473 · DOI
- 10.1111/infa.12186 · DOI · Paper page
- 10.1037/0033-295X.101.4.608 · doi:10.1037/0033-295x.101.4.608 · Paper page
- 10.1177/1745691618755704 · DOI · Paper page
- Perfors A. F., Navarro D. J. (2009). Confirmation bias is rational when hypotheses are sparse. In Taatgen N., van Rijn H. (Eds.), Proceedings of the 31st annual conference of the cognitive science society (pp. 2471–2476). Austin, TX: Cognitive Science Society.
- 10.1007/s11948-017-9886-2 · DOI
- 10.1101/059188 · DOI
- Popper K. (1959). The logic of scientific discovery. London, England: Routledge. (Originally work published 1934)
- 10.15252/embj.201570010 · DOI
- Rayleigh L. (1885). Presidential address at the 54th meeting of the British Association for the Advancement of Science, Montreal, August/September 1884. London, England: John Murray.
- 10.1037/pspi0000081 · DOI · Paper page
- 10.1111/ejn.13610 · DOI · Paper page
- 10.1111/1467-8624.00417 · DOI · Paper page
- 10.1111/j.1467-8721.2006.00452.x · DOI
- 10.1258/jrsm.99.4.178 · doi:10.1258/jrsm.99.4.178 · Paper page
- 10.1177/1745691616652875 · DOI
- 10.1080/00031305.2018.1505657 · DOI
- 10.1080/00031305.2015.1081619 · DOI
- 10.1016/j.biopsych.2017.09.029 · DOI
- 10.1523/JNEUROSCI.0888-19.2019 · DOI
- 10.1017/S0025727300002994 · doi:10.1017/s0025727300002994
- 10.1002/rrq.160 · DOI · Paper page
- Van den Broek P. (1994). Comprehension and memory of narrative texts: Inferences and coherence. In Gernsbacher M. A. (Ed.), Handbook of psycholinguistics (pp. 539–588). San Diego, CA: Academic Press.
- 10.31234/osf.io/cdgyh · DOI
- 10.1016/0010-0277(93)90015-N · DOI
- 10.1136/bmj.d4741 · DOI · Paper page
- 10.1080/17470216008416717 · DOI · Paper page
- 10.1038/nrn2776 · DOI · Paper page
Cited by
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178
Elsherif, M. M., Middleton, S. L., Phan, J. M., Azevedo, F., Iley, B. J., Grose-Hodge, M., Tyler, S. L., Kapp, S. K., Gourdon-Kanhukamwe, A., Grafton-Clarke, D., et al. (2022) · doi:10.31222/osf.io/k7a9p · Cited by 91
Generalization Bias in Science
Peters, U., Krauss, A., & Braganza, O. (2022) · Cognitive Science · doi:10.1111/cogs.13188 · Cited by 63
“Some internal factors such as cognitive biases have been discussed as well (Bishop, 2020; Munafò et al., 2020).”
BackgroundHardwicke, T. E., Szűcs, D., Thibault, R. T., Crüwell, S., van den Akker, O. R., Nuijten, M. B., & Ioannidis, J. P. A. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211040837 · Cited by 44
“…preexisting beliefs) or motivated reasoning (constructing and evaluating arguments according to what is desirable rather than what is rationally justifiable), may partly explain researchers’ tendency to ignore or dismiss the replication evidence (Bishop, 2019; Kunda, 1990; Nickerson, 1998).”
Background
Metadata
- Authors
- Bishop, D. V.
- Venue
- Quarterly Journal of Experimental Psychology
- Publisher
- SAGE Publications
- Published
- 2019-11-14
- Volume
- 73
- Issue
- 1
- Pages
- 1-19
- ISSN
- 0033-555X (ISSN-L)
- 1747-0218
- 1747-0226
- OpenAlex
- W2989011475
- Cited by
- 108
- References
- 34
- FWCI
- 14.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Funders
- H2020 European Research Council (694189)
- Wellcome Trust (082498)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. https://app.cochat.ai/papers/10.1177/1747021819886519
- The paper
- Bishop, D. V. (2020). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology. https://doi.org/10.1177/1747021819886519
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.