A glass half full interpretation of the replicability of psychological science
Open accessAbstract
A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported in the paper, mass and social media adhered to the simple narrative that only 36% of the studies replicated their original results. Here we show that 77% of the replication effect sizes reported were within a prediction interval based on the original effect size. In this light, the results of Reproducibility Project: Psychology can be viewed as a positive result for the scientific process.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Leek, Patil & Peng (2015) found that 75% of the RPP replications were within the prediction intervals and 77% if they included the two replications with effect sizes larger than the upper bound.
“they found that 75% of the replications were within the prediction intervals and 77% if they included the two replications with effect sizes larger than the upper bound of the prediction intervals.”
From Evaluating replicability of laboratory experiments in economics (2016) · cites this paper as Leek, Patil & Peng (2015)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Self-citationThe Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963 · see citing works
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M. T., Fruyt, F. D., Houwer, J. D., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R. M., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 818 · see citing works
A comment on replication,<i>P</i>‐values and evidence
Goodman, S. N. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110705 · Cited by 309 · see citing works
Reproducible Epidemiologic Research
Peng, R. D., Dominici, F., & Zeger, S. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj093 · Cited by 292 · see citing works
Self-citationStatistics: P values are just the tip of the iceberg
Leek, J. T., & Peng, R. D. (2015) · Nature · doi:10.1038/520612a · Cited by 211 · see citing works
Self-citationAn estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179 · see citing works
Self-citationOver half of psychology studies fail reproducibility test
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.18248 · Cited by 148 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [4][4] Ronald A Fisher. Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika , pages 507-521, 1915. -
- [7][7] John PA Ioannidis. Contradicted and initially stronger effects in highly cited clinical research. Jama , 294(2):218-228, 2005. -
Cited by
Sorted by citations
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · see citing works
Predicting the Replicability of Social Science Lab ExperimentsPreprint
Altmejd, A., Dreber, A., Forsell, E., Ho, T., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · doi:10.31222/osf.io/zamry · Cited by 23 · see citing works
- Authors
- Leek, J. T., Patil, P., & Peng, R. D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1509.08968
- OpenAlex
- W2284286170
- Cited by
- 1
- References
- 6
- Citations by year
- 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- replicability, Reproducibility Project: Psychology, prediction intervals, scientific method, media coverage of science, psychology research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
