Psychologists update their beliefs about effect sizes after replication studies
Open accessPublished version of a preprint
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Self-correction-a key feature distinguishing science from pseudoscience-requires that scientists update their beliefs in light of new evidence. However, people are often reluctant to change their beliefs. We examined belief updating in action by tracking research psychologists' beliefs in psychological effects before and after the completion of four large-scale replication projects. We found that psychologists did update their beliefs; they updated as much as they predicted they would, but not as much as our Bayesian model suggests they should if they trust the results. We found no evidence that psychologists became more critical of replications when it would have preserved their pre-existing beliefs. We also found no evidence that personal investment or lack of expertise discouraged belief updating, but people higher on intellectual humility updated their beliefs slightly more. Overall, our results suggest that replication studies can contribute to self-correction within psychology, but psychologists may underweight their evidentiary value.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Acknowledging the fallibility of scientific results via replication studies can help scientists to revise their beliefs about evidence for particular scientific phenomena 149 .”
From Predictors and consequences of intellectual humility (2022) · cites this paper as [149]
Citations
Cites
(2011) · Cited by 14,317
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022 · cited 1×
“For a scientific field to be self-correcting, at least three conditions are necessary: (1) empirical results must be subject to replication, and members of the field must (2) evaluate those replication efforts and (3) adjust their beliefs in response to new evidence.”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Lord, C. G., Ross, L., & Lepper, M. R. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.11.2098 · Cited by 4,583
Science as Social Knowledge: Values and Objectivity in Scientific Inquiry
Longino, H. E. (1990) · doi:10.1515/9780691209753 · Cited by 3,092
Main Trends in Recent Philosophy: Two Dogmas of Empiricism
Quine, W. V. (1951) · The Philosophical Review · doi:10.2307/2181906 · Cited by 3,029
van Orman Quine, W. (1976) · Can Theories be Refuted? · doi:10.1007/978-94-010-1863-0_2 · Cited by 2,709
The serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717
Research works printed but not linked: 3
- Luchins, A. S. in The Order of Presentation in Persuasion (ed. C. Hovland) (Yale Univ. Press, 1957).
- Munro, G. D., Leary, S. P. & Lasane, T. P. Between a rock and a hard place: biased assimilation of scientific information in the face of commitment. North Am. J. Psychol. 6, 431–444 (2004).
- Jost, A. Die Assoziationsfestigkeit in ihrer Abha ̈ngigkeit von der Verteilung der Wiederholungen [The strength of associations in their dependence on the distribution of repetitions]. Z. Psychol. Physiol. Si. 16, 436–472 (1897).
Cited by
Reflexivity in quantitative research: A rationale and beginner's guide
Jamieson, M. K., Govaart, G. H., & Pownall, M. (2023) · Social and Personality Psychology Compass · doi:10.1111/spc3.12735 · Cited by 398
Predictors and consequences of intellectual humility
Porter, T., Elnakouri, A., Meyers, E. A., Shibayama, T., Jayawickreme, E., & Grossmann, I. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00081-9 · Cited by 248
A meta-analysis of correction effects in science-relevant misinformation
Chan, M. S., & Albarracín, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01623-8 · Cited by 174
Developing an open science ‘mindset’
Hagger, M. S. (2021) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2021.2012474 · Cited by 56
Altenmüller, M. S., Nuding, S., & Gollwitzer, M. (2021) · Public Understanding of Science · doi:10.1177/09636625211022181 · Cited by 47
Metadata
- Authors
- McDiarmid, A. D., Tullett, A. M., Whitt, C. M., Vazire, S., Smaldino, P. E., & Stephens, J. E.
- Affiliations
- University of Alabama (US)
- The University of Melbourne (AU)
- University of California, Merced (US)
- Corresponding author
- McDiarmid, Alex D.
- Tullett, Alexa M.
- Venue
- Nature Human Behaviour
- Published
- 2021-11-22
- Volume
- 5
- Issue
- 12
- Pages
- 1663-1673
- ISSN
- 2397-3374 (ISSN-L)
- OpenAlex
- W3215782000
- Cited by
- 16
- References
- 56
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 7 · 2022: 2 · 2021: 5
- FWCI
- 2.02
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Human Behaviour (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- belief updating, replication study, effect size, self-correction, Bayesian updating, intellectual humility, personal investing, trust in research, replication crisis
- MeSH terms
- Humans, Psychology, Research, Statistics as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Psychologists update their beliefs about effect sizes after replication studies. https://app.cochat.ai/papers/10.1038/s41562-021-01220-7
- The paper
- McDiarmid, A. D., Tullett, A. M., Whitt, C. M., Vazire, S., Smaldino, P. E., & Stephens, J. E. (2021). Psychologists update their beliefs about effect sizes after replication studies. Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01220-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.