CoChat

New Statistical Metrics for Multisite Replication Projects

Open access
Mathur, Maya B., VanderWeele, Tyler J. · 2020 · Journal of the Royal Statistical Society Series A: Statistics in Society
Cited by 28

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Abstract

Summary: Increasingly, researchers are attempting to replicate published original studies by using large, multisite replication projects, at least 134 of which have been completed or are on going. These designs are promising to assess whether the original study is statistically consistent with the replications and to reassess the strength of evidence for the scientific effect of interest. However, existing analyses generally focus on single replications; when applied to multisite designs, they provide an incomplete view of aggregate evidence and can lead to misleading conclusions about replication success. We propose new statistical metrics representing firstly the probability that the original study's point estimate would be at least as extreme as it actually was, if in fact the original study were statistically consistent with the replications, and secondly the estimated proportion of population effects agreeing in direction with the original study. Generalized versions of the second metric enable consideration of only meaningfully strong population effects that agree in direction, or alternatively that disagree in direction, with the original study. These metrics apply when there are at least 10 replications (unless the heterogeneity estimate τ^=0, in which case the metrics apply regardless of the number of replications). The first metric assumes normal population effects but appears robust to violations in simulations; the second is distribution free. We provide R packages (Replicate and MetaUtility).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/rssa.12572

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 39

  1. Association for Psychological Science, 2018, Ongoing replication projects
  2. Baumeister, 2016, Misguided effort with elusive implications, Perspect. Psychol. Sci., 11, 574, 10.1177/1745691616652878 · DOI · Paper page
  3. Bouwmeester, 2017, Registered replication report: Rand, Greene, and Nowak (2012), Perspect. Psychol. Sci., 12, 527, 10.1177/1745691617693624 · DOI · Paper page
  4. Carpenter, 2000, Bootstrap confidence intervals: when, which, what?: A practical guide for medical statisticians, Statist. Med., 19, 1141, 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F · doi:10.1002/(sici)1097-0258(20000515)19:9<1141::aid-sim479>3.0.co;2-f
  5. Cheung, 2016, Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro, & Hannon (2002), Perspect. Psychol. Sci., 11, 750, 10.1177/1745691616664694 · DOI · Paper page
  6. Cohen, 1977, Statistical Power Analysis for the Behavioral Sciences
  7. Cova, 2018, Estimating the reproducibility of experimental philosophy, Rev. Philos. Psychol., 14, 1
  8. Crandall, 2016, On the scientific superiority of conceptual replications for scientific progress, J. Exptl Socl Psychol., 66, 93, 10.1016/j.jesp.2015.10.002 · DOI · Paper page
  9. Davison, 1997, Bootstrap Methods and Their Application, 10.1017/CBO9780511802843 · DOI · Paper page
  10. Ebersole, 2018, Many Labs 5: Can conducting formal peer review in advance improve reproducibility?
  11. Eerland, 2016, Registered replication report: Hart & Albarracín (2011), Perspect. Psychol. Sci., 11, 158, 10.1177/1745691615605826 · DOI · Paper page
  12. Efron, 1987, Better bootstrap confidence intervals, J. Am. Statist. Ass., 82, 171, 10.1080/01621459.1987.10478410 · DOI · Paper page
  13. Fisher, 2015, Robumeta: an R-package for robust variance estimation in meta-analysis
  14. Francis, 2012, The psychology of replication and replication in psychology, Perspect. Psychol. Sci., 7, 585, 10.1177/1745691612459520 · DOI · Paper page
  15. Gadbury, 2000, Unit-treatment interaction and its practical consequences, Biometrics, 56, 882, 10.1111/j.0006-341X.2000.00882.x · DOI
  16. Gadbury, 2001, Evaluating subject-treatment interaction when comparing two treatments, J. Biopharm. Statist., 11, 313, 10.1081/BIP-120008851 · doi:10.1081/bip-120008851
  17. Hagger, 2016, A multilab preregistered replication of the ego-depletion effect, Perspect. Psychol. Sci., 11, 546, 10.1177/1745691616652873 · DOI · Paper page
  18. Hardy, 1998, Detecting and describing heterogeneity in meta-analysis, Statist. Med., 17, 841, 10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D · DOI · Paper page
  19. Heckman, 1997, Making the most out of programme evaluations and social experiments: accounting for heterogeneity in programme impacts, Rev. Econ. Stud., 64, 487, 10.2307/2971729 · DOI · Paper page
  20. Hedges, 1980, Vote-counting methods in research synthesis, Psychol. Bull., 88, 359, 10.1037/0033-2909.88.2.359 · DOI · Paper page
  21. Lakens, 2013, Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs, Front. Psychol., 4, 10.3389/fpsyg.2013.00863 · DOI · Paper page
  22. Longford, 1999, Selection bias and treatment heterogeneity in clinical trials, Statist. Med., 18, 1467, 10.1002/(SICI)1097-0258(19990630)18:12<1467::AID-SIM149>3.0.CO;2-H · DOI · Paper page
  23. Lynch, 2015, Reflections on the Replication Corner: In praise of conceptual replications, Int. J. Res. Marktng, 32, 333, 10.1016/j.ijresmar.2015.09.006 · DOI
  24. Mathur, 2019, New metrics for meta-analyses of heterogeneous effects, Statist. Med., 38, 1336, 10.1002/sim.8057 · DOI · Paper page
  25. Mathur, 2020, Robust metrics and sensitivity analyses for meta-analyses of heterogeneous effects, Epidemiology, 31, 356, 10.1097/EDE.0000000000001180 · DOI · Paper page
  26. Monin, 2016, Be careful what you wish for: Commentary on Ebersole et al. (2016), J. Exptl Socl Psychol., 67, 95, 10.1016/j.jesp.2016.01.007 · DOI
  27. Monin, 2014, Commentaries and rejoinder on Klein et al. (2014), Socl Psychol., 45, 299, 10.1027/1864-9335/a000202 · DOI · Paper page
  28. Paule, 1982, Consensus values and weighting factors, J. Res. Natn. Bur. Stand., 87, 377, 10.6028/jres.087.022 · DOI · Paper page
  29. Pettigrew, 2008, How does intergroup contact reduce prejudice?: Meta-analytic tests of three mediators, Eur. J. Socl Psychol., 38, 922, 10.1002/ejsp.504 · DOI · Paper page
  30. Riley, 2011, Interpretation of random effects meta-analyses, Br. Med. J., 342, 10.1136/bmj.d549 · DOI · Paper page
  31. Schweinsberg, 2016, The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline, J. Exptl Socl Psychol., 66, 55, 10.1016/j.jesp.2015.10.001 · DOI · Paper page
  32. Schweinsberg, 2018, The Pipeline Project 2
  33. Shapiro, 1972, An approximate analysis of variance test for normality, J. Am. Statist. Ass., 67, 215, 10.1080/01621459.1972.10481232 · DOI · Paper page
  34. Simons, 2014, An introduction to Registered Replication Reports at Perspectives on Psychological Science, Perspect. Psychol. Sci., 9, 552, 10.1177/1745691614543974 · DOI
  35. Stewart, 2012, Statistical analysis of individual participant data meta-analyses: a comparison of methods and recommendations for practice, PLOS One, 7, 10.1371/journal.pone.0046042 · DOI · Paper page
  36. Sutton, 2000, Methods for Meta-analysis in Medical Research
  37. Verhagen, 2014, Bayesian tests to quantify the result of a replication attempt, J. Exptl Psychol., 143, 1457, 10.1037/a0036731 · DOI · Paper page
  38. Viechtbauer, 2010, Conducting meta-analyses in r with the metafor package, J. Statist. Softwr., 36
  39. Wang, 2019, A simple method to estimate prediction intervals and predictive distributions: summarizing meta-analyses beyond means and confidence intervals, Res. Synth. Meth., 10, 255, 10.1002/jrsm.1345 · DOI · Paper page

Cited by

Metadata

Authors
Mathur, M. B., & VanderWeele, T. J.
Venue
Journal of the Royal Statistical Society Series A: Statistics in Society
Published
2020-05-21
Volume
183
Issue
3
Pages
1145-1166
ISSN
  • 0964-1998 (ISSN-L)
  • 1467-985X
OpenAlex
W2776998249
Cited by
28
References
24
FWCI
2.94
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.