New Statistical Metrics for Multisite Replication Projects
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Summary: Increasingly, researchers are attempting to replicate published original studies by using large, multisite replication projects, at least 134 of which have been completed or are on going. These designs are promising to assess whether the original study is statistically consistent with the replications and to reassess the strength of evidence for the scientific effect of interest. However, existing analyses generally focus on single replications; when applied to multisite designs, they provide an incomplete view of aggregate evidence and can lead to misleading conclusions about replication success. We propose new statistical metrics representing firstly the probability that the original study's point estimate would be at least as extreme as it actually was, if in fact the original study were statistically consistent with the replications, and secondly the estimated proportion of population effects agreeing in direction with the original study. Generalized versions of the second metric enable consideration of only meaningfully strong population effects that agree in direction, or alternatively that disagree in direction, with the original study. These metrics apply when there are at least 10 replications (unless the heterogeneity estimate τ^=0, in which case the metrics apply regardless of the number of replications). The first metric assumes normal population effects but appears robust to violations in simulations; the second is distribution free. We provide R packages (Replicate and MetaUtility).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An analysis of variance test for normality (complete samples)
SHAPIRO, S. S., & WILK, M. B. (1965) · Biometrika · doi:10.1093/biomet/52.3-4.591 · Cited by 23,480
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
A Re-Evaluation of Random-Effects Meta-Analysis
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2008.00552.x · Cited by 2,569 · cited 1×
“Additionally, sensitivity analyses could be conducted by meta-analysing the replication study estimates under distribution-free approaches (Fisher and Tipton, 2015; Hedges et al., 2010) or under more flexible distributional assumptions (see Higgins et al. (2009) for a review).”
MethodRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 2×
“Additionally, sensitivity analyses could be conducted by meta-analysing the replication study estimates under distribution-free approaches (Fisher and Tipton, 2015; Hedges et al., 2010) or under more flexible distributional assumptions (see Higgins et al. (2009) for a review).”
MethodMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · cited 8×
“…parameters via random-effects meta-analysis, there are many possible choices of heterogeneity estimator, and it is important to choose one that is known to perform well for the effect measure of choice, particularly when the number of replication studies is relatively small (Veroniki et al., 2016).”
BackgroundMethodInfluentialEquivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 1×
“This approach is similar to equivalence testing and minimal effects testing, which compare a point estimate with null values other than 0 (Lakens et al., 2018).”
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · cited 3×
“…2014) has proposed analytic approaches for a single replication of a single study or designs in which numerous original studies across a discipline or domain are each replicated once (here termed ‘one-to-one designs’), as in Open Science Collaboration (2015) and in Camerer et al. (2016).”
BackgroundNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · cited 1×
“…supplementing direct replications with ‘conceptual replications’ that assess the same theory as the original study, but using a different operationalization (Crandall and Sherman (2016), Lynch et al. (2015) and Monin et al. (2014); see also dissent by Nosek et al. (2012) and Simons (2014)).”
BackgroundInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Moral credentials and the expression of prejudice.
Monin, B., & Miller, D. T. (2001) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.81.1.33 · Cited by 1,086 · cited 3×
“Returning to moral credentialling, the theory under investigation predicts that credentialling opportunities would increase subsequent attitudes that are consistent with prejudice; however, other theories suggest that credentialling opportunities might sometimes decrease such attitudes by prompting self-consistency or…”
BackgroundResult
Research works printed but not linked: 39
- Association for Psychological Science, 2018, Ongoing replication projects
- Baumeister, 2016, Misguided effort with elusive implications, Perspect. Psychol. Sci., 11, 574, 10.1177/1745691616652878 · DOI · Paper page
- Bouwmeester, 2017, Registered replication report: Rand, Greene, and Nowak (2012), Perspect. Psychol. Sci., 12, 527, 10.1177/1745691617693624 · DOI · Paper page
- Carpenter, 2000, Bootstrap confidence intervals: when, which, what?: A practical guide for medical statisticians, Statist. Med., 19, 1141, 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F · doi:10.1002/(sici)1097-0258(20000515)19:9<1141::aid-sim479>3.0.co;2-f
- Cheung, 2016, Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro, & Hannon (2002), Perspect. Psychol. Sci., 11, 750, 10.1177/1745691616664694 · DOI · Paper page
- Cohen, 1977, Statistical Power Analysis for the Behavioral Sciences
- Cova, 2018, Estimating the reproducibility of experimental philosophy, Rev. Philos. Psychol., 14, 1
- Crandall, 2016, On the scientific superiority of conceptual replications for scientific progress, J. Exptl Socl Psychol., 66, 93, 10.1016/j.jesp.2015.10.002 · DOI · Paper page
- Davison, 1997, Bootstrap Methods and Their Application, 10.1017/CBO9780511802843 · DOI · Paper page
- Ebersole, 2018, Many Labs 5: Can conducting formal peer review in advance improve reproducibility?
- Eerland, 2016, Registered replication report: Hart & Albarracín (2011), Perspect. Psychol. Sci., 11, 158, 10.1177/1745691615605826 · DOI · Paper page
- Efron, 1987, Better bootstrap confidence intervals, J. Am. Statist. Ass., 82, 171, 10.1080/01621459.1987.10478410 · DOI · Paper page
- Fisher, 2015, Robumeta: an R-package for robust variance estimation in meta-analysis
- Francis, 2012, The psychology of replication and replication in psychology, Perspect. Psychol. Sci., 7, 585, 10.1177/1745691612459520 · DOI · Paper page
- Gadbury, 2000, Unit-treatment interaction and its practical consequences, Biometrics, 56, 882, 10.1111/j.0006-341X.2000.00882.x · DOI
- Gadbury, 2001, Evaluating subject-treatment interaction when comparing two treatments, J. Biopharm. Statist., 11, 313, 10.1081/BIP-120008851 · doi:10.1081/bip-120008851
- Hagger, 2016, A multilab preregistered replication of the ego-depletion effect, Perspect. Psychol. Sci., 11, 546, 10.1177/1745691616652873 · DOI · Paper page
- Hardy, 1998, Detecting and describing heterogeneity in meta-analysis, Statist. Med., 17, 841, 10.1002/(SICI)1097-0258(19980430)17:8<841::AID-SIM781>3.0.CO;2-D · DOI · Paper page
- Heckman, 1997, Making the most out of programme evaluations and social experiments: accounting for heterogeneity in programme impacts, Rev. Econ. Stud., 64, 487, 10.2307/2971729 · DOI · Paper page
- Hedges, 1980, Vote-counting methods in research synthesis, Psychol. Bull., 88, 359, 10.1037/0033-2909.88.2.359 · DOI · Paper page
- Lakens, 2013, Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs, Front. Psychol., 4, 10.3389/fpsyg.2013.00863 · DOI · Paper page
- Longford, 1999, Selection bias and treatment heterogeneity in clinical trials, Statist. Med., 18, 1467, 10.1002/(SICI)1097-0258(19990630)18:12<1467::AID-SIM149>3.0.CO;2-H · DOI · Paper page
- Lynch, 2015, Reflections on the Replication Corner: In praise of conceptual replications, Int. J. Res. Marktng, 32, 333, 10.1016/j.ijresmar.2015.09.006 · DOI
- Mathur, 2019, New metrics for meta-analyses of heterogeneous effects, Statist. Med., 38, 1336, 10.1002/sim.8057 · DOI · Paper page
- Mathur, 2020, Robust metrics and sensitivity analyses for meta-analyses of heterogeneous effects, Epidemiology, 31, 356, 10.1097/EDE.0000000000001180 · DOI · Paper page
- Monin, 2016, Be careful what you wish for: Commentary on Ebersole et al. (2016), J. Exptl Socl Psychol., 67, 95, 10.1016/j.jesp.2016.01.007 · DOI
- Monin, 2014, Commentaries and rejoinder on Klein et al. (2014), Socl Psychol., 45, 299, 10.1027/1864-9335/a000202 · DOI · Paper page
- Paule, 1982, Consensus values and weighting factors, J. Res. Natn. Bur. Stand., 87, 377, 10.6028/jres.087.022 · DOI · Paper page
- Pettigrew, 2008, How does intergroup contact reduce prejudice?: Meta-analytic tests of three mediators, Eur. J. Socl Psychol., 38, 922, 10.1002/ejsp.504 · DOI · Paper page
- Riley, 2011, Interpretation of random effects meta-analyses, Br. Med. J., 342, 10.1136/bmj.d549 · DOI · Paper page
- Schweinsberg, 2016, The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline, J. Exptl Socl Psychol., 66, 55, 10.1016/j.jesp.2015.10.001 · DOI · Paper page
- Schweinsberg, 2018, The Pipeline Project 2
- Shapiro, 1972, An approximate analysis of variance test for normality, J. Am. Statist. Ass., 67, 215, 10.1080/01621459.1972.10481232 · DOI · Paper page
- Simons, 2014, An introduction to Registered Replication Reports at Perspectives on Psychological Science, Perspect. Psychol. Sci., 9, 552, 10.1177/1745691614543974 · DOI
- Stewart, 2012, Statistical analysis of individual participant data meta-analyses: a comparison of methods and recommendations for practice, PLOS One, 7, 10.1371/journal.pone.0046042 · DOI · Paper page
- Sutton, 2000, Methods for Meta-analysis in Medical Research
- Verhagen, 2014, Bayesian tests to quantify the result of a replication attempt, J. Exptl Psychol., 143, 1457, 10.1037/a0036731 · DOI · Paper page
- Viechtbauer, 2010, Conducting meta-analyses in r with the metafor package, J. Statist. Softwr., 36
- Wang, 2019, A simple method to estimate prediction intervals and predictive distributions: summarizing meta-analyses beyond means and confidence intervals, Res. Synth. Meth., 10, 255, 10.1002/jrsm.1345 · DOI · Paper page
Cited by
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Self-citationMany Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920958687 · Cited by 105
Self-citationResearch into evidence-based psychological interventions needs a stronger focus on replicability
Niemeyer, H., Knaevelsrud, C., van Aert, R. C. M., & Ehring, T. (2023) · Clinical Psychology in Europe · doi:10.32872/cpe.9997 · Cited by 3
“Is the effect size of the original study consistent with the replications in a multisite replication project (Mathur & VanderWeele, 2020)?”
Metadata
- Authors
- Mathur, M. B., & VanderWeele, T. J.
- Venue
- Journal of the Royal Statistical Society Series A: Statistics in Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-05-21
- Volume
- 183
- Issue
- 3
- Pages
- 1145-1166
- ISSN
- 0964-1998 (ISSN-L)
- 1467-985X
- OpenAlex
- W2776998249
- Cited by
- 28
- References
- 24
- FWCI
- 2.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: New Statistical Metrics for Multisite Replication Projects. https://app.cochat.ai/papers/10.1111/rssa.12572
- The paper
- Mathur, M. B., & VanderWeele, T. J. (2020). New Statistical Metrics for Multisite Replication Projects. Journal of the Royal Statistical Society Series A: Statistics in Society. https://doi.org/10.1111/rssa.12572
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.