CoChat

Estimating the reproducibility of psychological science

Open access
Open Science Collaboration · 2015 · Science

Published version of a preprint

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; 20 of the 57 sources its claims cite were checked for support.

Abstract

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

Overview

CoChat Summary

TL;DR

A large collaborative replication of 100 published psychology studies found that replication effects were about half the magnitude of original effects, and only 36% of replications had significant results versus 97% of originals. Replication success was more consistently related to the strength of the original evidence than to characteristics of the replication teams or implementation.

Key findings

  • Original effects (M = .396, SD = .193) were reliably larger than replication effects (M = .198, SD = .255); 82 of 97 study pairs (82.8%) showed a stronger original effect.
  • 97% of original studies had significant results (p < .05), but only 36% of replications did; 47% of original effect sizes fell within the 95% CI of the replication effect size, 38% were subjectively rated as replicated, and 70% of meta-analytic combinations were significant if no bias in original results was assumed.
  • Reproducibility was stronger for cognitive than social psychology: 20 of 38 (53%) cognitive effects replicated by p < .05 versus 14 of 53 (26%) social effects; main/simple effects replicated more often (23/50, 46%) than interaction effects (8/37, 22%).
  • Replication success correlated with original p-value, original effect size, and replication power, but not consistently with perceived importance, team expertise, or self-assessed replication quality.

Methods

  • The Open Science Collaboration conducted 100 replications of studies published in 2008 in three psychology journals: Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition; 270 contributing authors completed the replications.
  • Replication teams followed a structured, pre-registered protocol: selecting a key effect, contacting original authors for materials, writing and registering an analysis plan, obtaining review, conducting high-powered replications (average 92% power to detect the original effect size), and archiving data and materials.
  • Analyses converted effects to correlation coefficients where possible and evaluated reproducibility with five indicators: significance testing, comparison of original vs. replication effect sizes, whether the original effect size fell within the replication 95% CI, meta-analytic combination, and subjective “Did it replicate?” ratings.

Limitations

  • The paper states that it is unknown how far findings extend to other psychology subdisciplines or other disciplines; not all eligible studies were replicated, resource-intensive studies were less likely to be included, and the impact of selection bias is unknown.
  • The paper notes no single indicator sufficiently describes replication success, and meta-analytic estimates assume no publication bias in original results; if original effects are inflated by publication, selection, or reporting biases, replication estimates may better reflect true effects.
  • The text makes evident that the sample is limited to 100 completed replications from 2008 articles in three journals, and that unanticipated differences in sample, setting, or procedure could alter observed effect magnitudes despite efforts to maintain fidelity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1126/science.aac4716

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

8 Valid · 0 Not valid · 10 Not checked · 144 Own results and interpretations

60 more claims

Create a free account to see all 162 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 65
  • It may state two separate claims as one. 49
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

25 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentχ²(128) = 155.83reported p = .048 · recomputed p = .048

    “χ 2 (128) = 155.83, p = .048”

  • Consistentχ²(1) = 59.1reported p < .001 · recomputed p < .001

    “χ 2 (1) = 59.1, p < .001”

  • Consistentt(98) = -8.22reported p < .001 · recomputed p < .001

    “t (98) = ­8.22, p < .001”

  • Consistentt(96) = 9.33reported p < .001 · recomputed p < .001

    “t (96) = 9.33, p < .001”

  • ConsistentQ(73) = 310.45reported p < .001 · recomputed p < .001

    “Q (73) = 310.45, p < .001”

5 of 22 · the rest aren’t loaded

17 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 45 references

Research works printed but not linked: 2

  1. 6. W. C. Salmon, in Introduction to the Philosophy of Science, M. H. Salmon Ed. (Hackett Publishing Company, Inc., Indianapolis, 1999) pp. 7­41.
  2. 25. Open Science Collaboration, in Implementing Reproducible Computational Research (A Volume in The R Series) , V. Stodden, F. Leisch, R. Peng, Eds. (Taylor & Francis, New York, 2014) pp. 299­323.

Cited by

Items 6,201–6,300 of 8,626

Metadata

Authors
Open Science Collaboration
Corresponding author
Open Science Collaboration
Venue
Science
Published
2015-08-28
Volume
349
Issue
6251
Pages
aac4716-aac4716
ISSN
  • 0036-8075 (ISSN-L)
  • 1095-9203
OpenAlex
W1897139626
Cited by
8,515
References
39
Citations by year
2026: 414 · 2025: 532 · 2024: 698 · 2023: 757 · 2022: 783 · 2021: 906 · 2020: 1001 · 2019: 989 · 2018: 938 · 2017: 800 · 2016: 607 · 2015: 79
FWCI
736.90
Type
journal article
Language
en
Locations
  • Science (publishedVersion)
  • Research portal (Tilburg University) (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, other-oa)
  • UvA-DARE (University of Amsterdam) (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
  • VU Research Portal (publishedVersion)
  • University of Groningen research database (University of Groningen / Centre for Information Technology) (acceptedVersion, other-oa)
  • Repository of the Academy's Library (Library of the Hungarian Academy of Sciences) (acceptedVersion)
  • London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion)
  • Discovery Research Portal (University of Dundee) (submittedVersion, other-oa)
  • Keele Research Repository (Keele University) (submittedVersion, cc-by)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
  • Pure (Coventry University) (submittedVersion, other-oa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Winchester Research Repository (submittedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
  • ANU Open Research (Australian National University) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • TU/e Research Portal (submittedVersion)
  • BOA (University of Milano-Bicocca) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Max Planck Digital Library (submittedVersion)
  • Coventry University Open Collections (Coventry university) (submittedVersion)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
  • Research Portal (King's College London) (submittedVersion)
  • publish.UP (University of Potsdam) (submittedVersion)
  • Lirias (KU Leuven) (submittedVersion)
  • Faculty of Philosophy (University of Belgrade) (submittedVersion)
  • Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
  • Bristol Research (University of Bristol) (submittedVersion)
  • UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by-nc-nd)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
  • University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, other-oa)
  • The Open Collections (Coventry University) (submittedVersion)
  • Research Publications (Maastricht University) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
Funders
  • Laura and John Arnold Foundation
  • Center for Open Science
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
Keywords
reproducibility, replication, statistical significance, confidence intervals, publication bias, quality of evidence, psychology research
MeSH terms
Periodicals as Topic, Psychology, Publishing, Research, Research Design, Reproducibility of Results, Confidence Intervals, Publication Bias, Behavioral Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.