CoChat

The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture

Open access
Bishop, Dorothy VM · 2020 · Quarterly Journal of Experimental Psychology
Cited by 108

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

Abstract

Like many other areas of science, experimental psychology is affected by a “replication crisis” that is causing concern in many fields of research. Approaches to tackling this crisis include better training in statistical methods, greater transparency and openness, and changes to the incentives created by funding agencies, journals, and institutions. Here, I argue that if proposed solutions are to be effective, we also need to take into account human cognitive constraints that can distort all stages of the research process, including design and execution of experiments, analysis of data, and writing up findings for publication. I focus specifically on cognitive schemata in perception and memory, confirmation bias, systematic misunderstanding of statistics, and asymmetry in moral judgements of errors of commission and omission. Finally, I consider methods that may help mitigate the effect of cognitive constraints: better training, including use of simulations to overcome statistical misunderstanding; specific programmes directed at inoculating against cognitive biases; adoption of Registered Reports to encourage more critical reflection in planning studies; and using methods such as triangulation and “pre mortem” evaluation of study design to foster a culture of dialogue and criticism.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/1747021819886519

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 64

  1. Alexander S. (2019). 5-HTTLPR: A pointed review. Retrieved from https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/
  2. Babbage C. (1830). Reflections on the decline of science in England, and on some of its causes. London, England: B. Fellowes. Republished by Franklin Classics.
  3. 10.1098/rsos.170193 · DOI · Paper page
  4. Barber T. X. (1976). Pitfalls in human research. New York, NY: Pergamon Press.
  5. 10.1037/h0026724 · DOI · Paper page
  6. 10.1227/01.NEU.0000255391.92785.ED · doi:10.1227/01.neu.0000255391.92785.ed
  7. Bem D. J. (2004). Writing the empirical journal article. In Darley J. M., Zanna M. P., Roediger H. L. (Eds.), The compleat academic: A career guide (2nd ed., pp. 185–219). Washington, DC: American Psychological Association.
  8. 10.1111/j.1601-183X.2007.00360.x · DOI · Paper page
  9. 10.1080/14640748908402356 · DOI
  10. 10.1126/science.2403694 · DOI · Paper page
  11. 10.1038/d41586-019-02674-6 · DOI · Paper page
  12. Dawkins R. (1976). The selfish gene. Oxford, UK: Oxford University Press.
  13. 10.1126/science.1194140 · DOI · Paper page
  14. 10.1038/s41598-019-42515-0 · DOI
  15. 10.1017/S0033291718001873 · DOI · Paper page
  16. 10.1016/0001-6918(79)90017-9 · DOI · Paper page
  17. 10.2466/pr0.1970.27.3.835 · DOI
  18. 10.1136/bmj.b2680 · DOI · Paper page
  19. Hahn U., Harris A. J. L. (2014). What does it mean to be biased: Motivated reasoning and rationality. In Ross B. (Ed.), The psychology of learning and motivation (Vol. 61, pp. 41–101). San Diego, CA: Elsevier Academic Press.
  20. 10.1002/(SICI)1099-0992(199603)26:2<201::AID-EJSP745>3.0.CO;2-J · DOI
  21. 10.1177/00222194050380030401 · DOI
  22. 10.1016/j.cortex.2016.03.019 · DOI · Paper page
  23. Hossenfelder S. (2018). Lost in math: How beauty leads physics astray. New York, NY: Basic Books.
  24. International Committee of Medical Journal Editors. (2019). Recommendations: Clinical trials. Retrieved from http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html
  25. 10.1007/s10802-013-9741-0 · DOI
  26. 10.1038/nmeth.2699 · DOI
  27. 10.31234/osf.io/shm8v · doi:10.31234/osf.io/shm8v
  28. 10.1017/9781139696647 · DOI
  29. 10.1177/0023677218767478 · DOI
  30. 10.1037/bul0000142 · DOI · Paper page
  31. Levenson T. (2015). The hunt for Vulcan: . . . And how Albert Einstein destroyed a planet, discovered relativity, and deciphered the universe. New York, NY: Random House.
  32. Mahoney M. J. (1976). Scientist as subject: The psychological imperative. Cambridge, MA: Ballinger.
  33. 10.1177/2372732215600886 · DOI · Paper page
  34. 10.1093/ntr/ntx081 · DOI · Paper page
  35. 10.1136/bmj.295.6599.656 · DOI
  36. 10.7554/eLife21451.001 · doi:10.7554/elife21451.001
  37. 10.2307/27757473 · DOI
  38. 10.1111/infa.12186 · DOI · Paper page
  39. 10.1037/0033-295X.101.4.608 · doi:10.1037/0033-295x.101.4.608 · Paper page
  40. 10.1177/1745691618755704 · DOI · Paper page
  41. Perfors A. F., Navarro D. J. (2009). Confirmation bias is rational when hypotheses are sparse. In Taatgen N., van Rijn H. (Eds.), Proceedings of the 31st annual conference of the cognitive science society (pp. 2471–2476). Austin, TX: Cognitive Science Society.
  42. 10.1007/s11948-017-9886-2 · DOI
  43. 10.1101/059188 · DOI
  44. Popper K. (1959). The logic of scientific discovery. London, England: Routledge. (Originally work published 1934)
  45. 10.15252/embj.201570010 · DOI
  46. Rayleigh L. (1885). Presidential address at the 54th meeting of the British Association for the Advancement of Science, Montreal, August/September 1884. London, England: John Murray.
  47. 10.1037/pspi0000081 · DOI · Paper page
  48. 10.1111/ejn.13610 · DOI · Paper page
  49. 10.1111/1467-8624.00417 · DOI · Paper page
  50. 10.1111/j.1467-8721.2006.00452.x · DOI
  51. 10.1258/jrsm.99.4.178 · doi:10.1258/jrsm.99.4.178 · Paper page
  52. 10.1177/1745691616652875 · DOI
  53. 10.1080/00031305.2018.1505657 · DOI
  54. 10.1080/00031305.2015.1081619 · DOI
  55. 10.1016/j.biopsych.2017.09.029 · DOI
  56. 10.1523/JNEUROSCI.0888-19.2019 · DOI
  57. 10.1017/S0025727300002994 · doi:10.1017/s0025727300002994
  58. 10.1002/rrq.160 · DOI · Paper page
  59. Van den Broek P. (1994). Comprehension and memory of narrative texts: Inferences and coherence. In Gernsbacher M. A. (Ed.), Handbook of psycholinguistics (pp. 539–588). San Diego, CA: Academic Press.
  60. 10.31234/osf.io/cdgyh · DOI
  61. 10.1016/0010-0277(93)90015-N · DOI
  62. 10.1136/bmj.d4741 · DOI · Paper page
  63. 10.1080/17470216008416717 · DOI · Paper page
  64. 10.1038/nrn2776 · DOI · Paper page

Cited by

See all 108

Metadata

Authors
Bishop, D. V.
Venue
Quarterly Journal of Experimental Psychology
Published
2019-11-14
Volume
73
Issue
1
Pages
1-19
ISSN
  • 0033-555X (ISSN-L)
  • 1747-0218
  • 1747-0226
OpenAlex
W2989011475
Cited by
108
References
34
FWCI
14.83
Type
journal article
Language
en
Open access
hybrid
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Funders
  • H2020 European Research Council (694189)
  • Wellcome Trust (082498)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. https://app.cochat.ai/papers/10.1177/1747021819886519
The paper
Bishop, D. V. (2020). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology. https://doi.org/10.1177/1747021819886519

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.