CoChat

Replication, Communication, and the Population Dynamics of Scientific Discovery

Open access

The DOI matches its Crossref record; 39 of the 43 references it cites were found in a registry or an index.

Abstract

Many published research results are false (Ioannidis, 2005), and controversy continues over the roles of replication and publication policy in improving the reliability of research. Addressing these problems is frustrated by the lack of a formal framework that jointly represents hypothesis formation, replication, publication bias, and variation in research quality. We develop a mathematical model of scientific discovery that combines all of these elements. This model provides both a dynamic model of research as well as a formal framework for reasoning about the normative structure of science. We show that replication may serve as a ratchet that gradually separates true hypotheses from false, but the same factors that make initial findings unreliable also make replications unreliable. The most important factors in improving the reliability of research are the rate of false positives and the base rate of true hypotheses, and we offer suggestions for addressing each. Our results also bring clarity to verbal debates about the communication of research. Surprisingly, publication bias is not always an obstacle, but instead may have positive impacts-suppression of negative novel findings is often beneficial. We also find that communication of negative replications may aid true discovery even when attempts to replicate have diminished power. The model speaks constructively to ongoing debates about the design and conduct of science, focusing analysis and discussion on precise, internally consistent models, as well as highlighting the importance of population dynamics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0136088

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig 1 Population dynamics of replication.
Fig 1 Population dynamics of replication.CC BY · PubMed Central
Fig 2 Effects of base rate, replication, power, false-positives, and communication on the probability that an hypothesis with a given tally is true. The two clusters illustrate difference scenarios. The blue trends, each labeled with its tally value, show precision as it varies by the parameter on each horizontal axis. The numbers indicate the tally of a curve. Dashed curves are tallies of an even number. The vertical hairlines show the parameter values held constant across panels within the same cluster.
Fig 2 Effects of base rate, replication, power, false-positives, and communication on the probability that an hypothesis with a given tally is true. The two clusters illustrate difference scenarios. The blue trends, each labeled with its tally value, show precision as it varies by the parameter on each horizontal axis. The numbers indicate the tally of a curve. Dashed curves are tallies of an even number. The vertical hairlines show the parameter values held constant across panels within the same cluster.CC BY · PubMed Central
Fig 3 Replication and communication as epistemological chromatography. Precision is indicated in blue, sensitivity in orange, and specificity in gray.
Fig 3 Replication and communication as epistemological chromatography. Precision is indicated in blue, sensitivity in orange, and specificity in gray.CC BY · PubMed Central
Fig 4 Targeted replication effort. In all three plots, tallies marked for targeted replication are shown by the shaded region. Precision is indicated in blue, sensitivity in orange, and specificity in gray. Baseline parameters set to b = 0.001, α = 0.05, r = 0.1, r T = 0.5, c N− = 0, c R− = c R+ = 1. Dashed curves display steady-state without targeted replication, r T = 0. (a) High power setting, 1 − β = 0.8. (b) Low power setting, 1 − β = 0.6. (c) Low power, 1 − β = 0.6, and including tally s = 0 in the target.
Fig 4 Targeted replication effort. In all three plots, tallies marked for targeted replication are shown by the shaded region. Precision is indicated in blue, sensitivity in orange, and specificity in gray. Baseline parameters set to b = 0.001, α = 0.05, r = 0.1, r T = 0.5, c N− = 0, c R− = c R+ = 1. Dashed curves display steady-state without targeted replication, r T = 0. (a) High power setting, 1 − β = 0.8. (b) Low power setting, 1 − β = 0.6. (c) Low power, 1 − β = 0.6, and including tally s = 0 in the target.CC BY · PubMed Central
Fig 5 Differential power and replication dynamics. Precision is indicated in blue, sensitivity in orange, and specificity in gray. (a) Low power initial studies (1 − β = 0.6, α = 0.2) but high power replications (1 − β R = 0.8, α R = 0.05). (b) High power initial studies (1 − β = 0.8, α = 0.05) but low power replications (1 − β R = 0.5, α R = 0.05). (c) and (d) as in (a) and (b), respectively, but only 10% of negative replications are communicated.
Fig 5 Differential power and replication dynamics. Precision is indicated in blue, sensitivity in orange, and specificity in gray. (a) Low power initial studies (1 − β = 0.6, α = 0.2) but high power replications (1 − β R = 0.8, α R = 0.05). (b) High power initial studies (1 − β = 0.8, α = 0.05) but low power replications (1 − β R = 0.5, α R = 0.05). (c) and (d) as in (a) and (b), respectively, but only 10% of negative replications are communicated.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. R Levins, 1966, The Strategy of Model Building in Population Biology, American Scientist, 54
  2. American Political Science Association Task Force on Public Engagement. Increasing the credibility of political science research: A proposal for journal reforms; 2014.

Cited by

See all 117

Metadata

Authors
McElreath, R., & Smaldino, P. E.
Affiliations
University of California, Davis (US)
Corresponding author
McElreath, Richard
Venue
PLOS ONE
Published
2015-08-26
Volume
10
Issue
8
Pages
e0136088
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2127007930
Cited by
117
References
43
Citations by year
2026: 2 · 2025: 5 · 2024: 5 · 2023: 11 · 2022: 8 · 2021: 15 · 2020: 15 · 2019: 13 · 2018: 29 · 2017: 6 · 2016: 6 · 2015: 2
FWCI
28.18
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • University of California, Davis
  • Society for Personality and Social Psychology
Topics
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
replication, publication bias, false positives, research reliability, population dynamics, hypothesis generation, research dissemination, negative results, publication policy
MeSH terms
Communication, Humans, Models, Theoretical, Population Dynamics, Research Design, Science, Reproducibility of Results, Publication Bias, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.