Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Open accessPublished version of a preprint
Abstract
Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ebersole et al. found that replicability of original findings did not improve with revised vs original replication protocols.
“they found that the replicability of original findings did not improve with the revised vs. original replication protocols”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Ebersole et al., 2020
Heterogeneity is not uncommon, even if it is relatively small, in multisite replications in which heterogeneity can be directly estimated.
“Such heterogeneity is not uncommon, even if it is relatively small, in multisite replications in which heterogeneity can be directly estimated (Ebersole et al., 2020; Klein et al., 2018)”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Ebersole et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772 · see citing works
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521 · see citing works
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314 · cited 7× · see citing works
“(2014), 36 of 97 findings (37%) in the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015)1, 11 of 18 findings (61%) in Camerer et al. (2016), 3 of 10 findings (30%) in Ebersole et al. (2016), 29 of 37 findings (78%) in Cova et al. (2018), 13 of 21 findings (62%) in Camerer et al.”
BackgroundResultInfluentialSelf-citationInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Self-citationConsequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2× · see citing works
“The initial finding might have been a false positive, reflecting either a “normal” Type I error or one made more likely by selectively reporting positive results and ignoring null results (Greenwald, 1975; Rosenthal, 1979; Sterling, 1959) or by employing flexibility in analytic decisions and reporting (Gelman & Loken,…”
BackgroundConstraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986 · see citing works
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · see citing works
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963 · see citing works
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Self-citationThe Effect of Implied Orientation Derived from Verbal Context on Picture Recognition
Stanfield, R. A., & Zwaan, R. A. (2001) · Psychological Science · doi:10.1111/1467-9280.00326 · Cited by 836 · see citing works
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
The Value of Believing in Free Will
Vohs, K. D., & Schooler, J. W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02045.x · Cited by 659 · see citing works
Shnabel, N., & Nadler, A. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.116 · Cited by 618 · see citing works
The alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597 · see citing works
Registered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 574 · see citing works
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474 · see citing works
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, M. B. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 474 · see citing works
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468 · see citing works
Self-citationWhy do implicit and explicit attitude tests diverge? The role of structural fit.
Payne, B. K., Burkley, M. A., & Stokes, M. B. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.16 · Cited by 401 · see citing works
Nosek, B. A., & Errington, T. M. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000691 · Cited by 401 · see citing works
Self-citationA Simple Approximation for Calculating Sample Sizes for Comparing Independent Proportions
Fleiss, J. L., Tytun, A., & Ury, H. K. (1980) · Biometrics · doi:10.2307/2529990 · Cited by 394 · see citing works
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388 · see citing works
Detecting the Snake in the Grass
LoBue, V., & DeLoache, J. S. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02081.x · Cited by 382 · see citing works
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331 · see citing works
Self-citationInterpreting the predictions of prediction markets
Manski, C. F. (2006) · Economics Letters · doi:10.1016/j.econlet.2006.01.004 · Cited by 330 · see citing works
Murray, S., Derrick, J. L., Leder, S., & Holmes, J. G. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.3.429 · Cited by 255 · see citing works
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243 · see citing works
Nosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242 · see citing works
Self-citationvan Dijk, E., van Kleef, G. A., Steinel, W., & van Beest, I. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.600 · Cited by 216 · see citing works
Förster, J., Liberman, N., & Kuschel, S. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.579 · Cited by 207 · see citing works
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C., Bahník, Š., Barnett‐Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 189 · see citing works
Self-citationEstimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
Why people are reluctant to tempt fate.
Risen, J. L., & Gilovich, T. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.95.2.293 · Cited by 138 · see citing works
Conceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 125 · see citing works
Klein, R. A., Cook, C. L., Ebersole, C. R., et al. (2019) · doi:10.31234/osf.io/vef2c · Cited by 125 · see citing works
Self-citationNoah, T., Schul, Y., & Mayo, R. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000121 · Cited by 110 · see citing works
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y.-L., Nosek, B. A., Johannesson, M., & Dreber, A. (2018) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 97 · see citing works
Self-citationIncreasing and decreasing motor and cognitive output: A model of general action and inaction goals.
Albarracín, D., Handley, I. M., Noguchi, K., McCulloch, K. C., Li, H., Leeper, J., Brown, R. D., Earl, A. N., & Hart, W. (2008) · Journal of Personality and Social Psychology · doi:10.1037/a0012833 · Cited by 93 · see citing works
Reflection on the Smiling Registered Replication Report
Strack, F. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616674460 · Cited by 78 · see citing works
Where Do We Look During Potentially Offensive Behavior?
Crosby, J. R., Monin, B., & Richardson, D. C. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02072.x · Cited by 72 · see citing works
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R., & Xu, M.-R. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 64 · see citing works
A Perspectivist Approach to Theory Construction
McGuire, W. J. (2004) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0802_11 · Cited by 58 · see citing works
Schwarz, N., & Strack, F. (2014) · Cited by 36 · see citing works
New Statistical Metrics for Multisite Replication Projects
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12572 · Cited by 29 · see citing works
Self-citationPetty, R. E., & Cacioppo, J. T. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.12.007 · Cited by 23 · see citing works
Many Labs 5: Registered Replication of Vohs and Schooler (2008), Experiment 1
Buttrick, N. R., Aczél, B., Aeschbach, L. F., Bakos, B. E., Brühlmann, F., Claypool, H. M., Hüffmeier, J., Kovács, M., Schuepfer, K., Szécsi, P., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920917931 · Cited by 22 · see citing works
Self-citationMany Labs 5: Registered Replication of LoBue and DeLoache (2008)
Lazarević, L. B., Purić, D., Žeželj, I. L., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920953350 · Cited by 10 · see citing works
Self-citationInvestigating Object Orientation Effects Across 14 LanguagesPreprint
Chen, S.-C., Szabelska, A., Chartier, C. R., Kekecs, Z., Lynott, D., Bernabeu, P., Jones, B. C., DeBruine, L. M., Levitan, C., Werner, K. M., et al. (2018) · doi:10.31234/osf.io/t2pjv · Cited by 9 · see citing works
Self-citationMany Labs 5: Registered Replication of Albarracín et al. (2008), Experiment 5
Chartier, C. R., Arnal, J., Arrow, H., Bloxsom, N. G., Bonfiglio, D. B. V., Brumbaugh, C. C., Corker, K. S., Ebersole, C. R., Garinther, A., Giessner, S. R., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920945963 · Cited by 8 · see citing works
Self-citationMany Labs 5: Registered Replication of Payne, Burkley, and Stokes (2008), Study 4
Ebersole, C. R., Andrighetto, L., Casini, E., Chiorri, C., Rosa, A. D., Domaneschi, F., Ferguson, I. R., Fryberger, E., Giacomantonio, M., Grahe, J. E., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919885609 · Cited by 8 · see citing works
Self-citationMany Labs 5: Registered Replication of Shnabel and Nadler (2008), Study 4
Baranski, E. N., Baskin, E., Coary, S. P., Ebersole, C. R., Krueger, L. E., Lazarević, L. B., Miller, J. K., Orlić, A., Penner, M. R., Purić, D., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920917334 · Cited by 7 · see citing works
Self-citationMany Labs 5: Registered Replication of Förster, Liberman, and Kuschel’s (2008) Study 1
IJzerman, H., Ropovik, I., Ebersole, C. R., Tidwell, N., Markiewicz, Ł., de Lima, T. J. S., Wolf, D., Novak, S. A., Collins, W. M., Menon, M., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920916513 · Cited by 6 · see citing works
Self-citationMathur, M. B., Bart-Plange, D.-J., Aczél, B., Bernstein, M. H., Ciunci, A. M., Ebersole, C. R., Falcão, F., Ashbaugh, K., Hilliard, R. A., Jern, A., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918785165 · Cited by 5 · see citing works
Self-citationMany Labs 5: Registered Replication of Crosby, Monin, and Richardson (2008)
Rabagliati, H., Corley, M. M. B., Dering, B. R., Hancock, P., King, J. P. J., Levitan, C., Loy, J. E., & Millen, A. E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919870737 · Cited by 4 · see citing works
Self-citationMany Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)
Skorb, L., Aczél, B., Bakos, B. E., Feinberg, L., Hałasa, E., Kauff, M., Kovács, M., Krasuska, K., Kuchno, K., Manfredi, D., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920927643 · Cited by 4 · see citing works
Self-citationMany Labs 5: Registered Replication of Albarracín et al. (2008), Experiment 7
Corker, K. S., Arnal, J., Bonfiglio, D. B. V., Curran, P. G., Chartier, C. R., Chopik, W. J., Guadagno, R. E., Kimbrough, A. M., Schmidt, K., & Wiggins, B. J. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920925750 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Center for Open Science. (n.d.). Registered Reports: Peer review before results are known to align scientific values and practices . Retrieved from http://cos.io/rr/
Cited by
Sorted by citations
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Self-citationInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
Self-citationKlein, R. A., Cook, C. L., Ebersole, C. R., Vitiello, C., Nosek, B. A., Hilgard, J., Ahn, P. H., Brady, A. J., Chartier, C. R., Christopherson, C. D., et al. (2022) · Collabra Psychology · doi:10.1525/collabra.35271 · Cited by 193 · see citing works
Self-citationFlake, J. K. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1898962 · Cited by 58 · see citing works
Sommet, N., Weissman, D. L., Cheutin, N., & Elliot, A. (2022) · doi:10.31219/osf.io/xhe3u · Cited by 36 · see citing works
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 34 · see citing works
“In addition, recent investigations failed to show evidence for the claim that researchers’ expertise and diligence generally produce larger or more precise effect sizes and higher replication success (Ebersole et al., 2020; Protzko & Schooler, 2020).”
Result
- Authors
- Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., Corker, K. S., Corley, M., Hartshorne, J. K., IJzerman, H., Lazarević, L. B., Rabagliati, H., Ropovik, I., Aczel, B., Aeschbach, L. F., Andrighetto, L., Arnal, J. D., Arrow, H., Babincak, P., Bakos, B. E., Baník, G., Baskin, E., Belopavlović, R., Bernstein, M. H., Białek, M., Bloxsom, N. G., Bodroža, B., Bonfiglio, D. B. V., Boucher, L., Brühlmann, F., Brumbaugh, C. C., Casini, E., Chen, Y., Chiorri, C., Chopik, W. J., Christ, O., Ciunci, A. M., Claypool, H. M., Coary, S., Čolić, M. V., Collins, W. M., Curran, P. G., Day, C. R., Dering, B., Dreber, A., Edlund, J. E., Falcão, F., Fedor, A., Feinberg, L., Ferguson, I. R., Ford, M., Frank, M. C., Fryberger, E., Garinther, A., Gawryluk, K., Ashbaugh, K., Giacomantonio, M., Giessner, S. R., Grahe, J. E., Guadagno, R. E., Hałasa, E., Hancock, P. J. B., Hilliard, R. A., Hüffmeier, J., Hughes, S., Idzikowska, K., Inzlicht, M., Jern, A., Jiménez-Leal, W., Johannesson, M., Joy-Gaba, J. A., Kauff, M., Kellier, D. J., Kessinger, G., Kidwell, M. C., Kimbrough, A. M., King, J. P. J., Kolb, V. S., Kołodziej, S., Kovacs, M., Krasuska, K., Kraus, S., Krueger, L. E., Kuchno, K., Lage, C. A., Langford, E. V., Levitan, C. A., de Lima, T. J. S., Lin, H., Lins, S., Loy, J. E., Manfredi, D., Markiewicz, Ł., Menon, M., Mercier, B., Metzger, M., Meyet, V., Millen, A. E., Miller, J. K., Montealegre, A., Moore, D. A., Muda, R., Nave, G., Nichols, A. L., Novak, S. A., Nunnally, C., Orlić, A., Palinkas, A., Panno, A., Parks, K. P., Pedović, I., Pękala, E., Penner, M. R., Pessers, S., Petrović, B., Pfeiffer, T., Pieńkosz, D., Preti, E., Purić, D., Ramos, T., Ravid, J., Razza, T. S., Rentzsch, K., Richetin, J., Rife, S. C., Rosa, A. D., Rudy, K. H., Salamon, J., Saunders, B., Sawicki, P., Schmidt, K., Schuepfer, K., Schultze, T., Schulz-Hardt, S., Schütz, A., Shabazian, A. N., Shubella, R. L., Siegel, A., Silva, R., Sioma, B., Skorb, L., de Souza, L. E. C., Steegen, S., Stein, L. A. R., Sternglanz, R. W., Stojilović, D., Storage, D., Sullivan, G. B., Szaszi, B., Szecsi, P., Szöke, O., Szuts, A., Thomae, M., Tidwell, N. D., Tocco, C., Torka, A.-K., Tuerlinckx, F., Vanpaemel, W., Vaughn, L. A., Vianello, M., Viganola, D., Vlachou, M., Walker, R. J., Weissgerber, S. C., Wichman, A. L., Wiggins, B. J., Wolf, D., Wood, M. J., Zealley, D., Žeželj, I., Zrubka, M., & Nosek, B. A.
- Affiliations
- University of Virginia (US)
- Stanford University (US)
- University of Houston (US)
- Ashland University (US)
- Grand Valley State University (US)
- University of Edinburgh (GB)
- Boston College (US)
- Institut Universitaire de France (FR)
- University of Belgrade (RS)
- University of Prešov (SK)
- Institute of Psychology, ELTE Eötvös Loránd University
- University of Basel (CH)
- University of Genoa (IT)
- McDaniel College (US)
- University of Oregon (US)
- Institute of Psychology, Faculty of Arts, University of Presov
- Eötvös Loránd University (HU)
- Saint Joseph's University (US)
- University of Novi Sad (RS)
- University of Rhode Island (US)
- University of Wrocław (PL)
- Nova Southeastern University (US)
- Queens College, CUNY (US)
- University of Milano-Bicocca (IT)
- Harvard University (US)
- Michigan State University (US)
- FernUniversität in Hagen (DE)
- Miami University (US)
- Loyola University Chicago (US)
- Coventry University (GB)
- University of Stirling (GB)
- Stockholm School of Economics (SE)
- Rochester Institute of Technology (US)
- Universidade do Porto (PT)
- HUN-REN Centre for Ecological Research (HU)
- Virginia Commonwealth University (US)
- Loyola Marymount University (US)
- Pacific Lutheran University (US)
- Kozminski University (PL)
- Rose–Hulman Institute of Technology (US)
- Sapienza University of Rome (IT)
- Erasmus University Rotterdam (NL)
- Maria Curie-Skłodowska University (PL)
- TU Dortmund University (DE)
- Ghent University (BE)
- University of Toronto (CA)
- Department of Psychology, Universidad de los Andes
- MSH Medical School Hamburg – University of Applied Sciences and Medical University (DE)
- University of Pennsylvania (US)
- Brigham Young University - Idaho (US)
- University of Utah (US)
- The University of Texas at Dallas (US)
- Fort Lewis College (US)
- East Texas A&M University (US)
- Pontifícia Universidade Católica do Rio de Janeiro (BR)
- Occidental College (US)
- Department of Social and Work Psychology, University of Brasília
- University of California, Irvine (US)
- Willamette University (US)
- Cornell University (US)
- University of California, Berkeley (US)
- Duy Tan University (VN)
- Hofstra University (US)
- European University of Rome (IT)
- University of Nis (RS)
- Western Kentucky University (US)
- KU Leuven (BE)
- Massey University (NZ)
- Psychologische Hochschule Berlin (DE)
- Murray State University (US)
- University of Padua (IT)
- Department of Psychology, Brigham Young University–Idaho
- University of Dundee (GB)
- Southern Illinois University Carbondale (US)
- Leibniz ScienceCampus Primate Cognition (DE)
- Institute of Psychology, University of Göttingen
- University of Bamberg (DE)
- Cultivate Labs, Chicago, Illinois
- Department of Psychology, University of Fortaleza
- Belgrade, Serbia
- University of Denver (US)
- Diploma Hochschule (DE)
- Ithaca College (US)
- University of Kassel (DE)
- University of Winchester (GB)
- Department of Psychology, University of Amsterdam
- Center for Open Science (US)
- Corresponding author
- Ebersole, Charles R.
- ORCID
- Ebersole, Charles R.
- Mathur, Maya B.
- Baranski, Erica
- Bart-Plange, Diane-Jo
- Buttrick, Nicholas R.
- Corker, Katherine S.
- Corley, Martin
- Hartshorne, Joshua K.
- IJzerman, Hans
- Lazarević, Ljiljana B.
- Rabagliati, Hugh
- Ropovik, Ivan
- Aeschbach, Lena F.
- Andrighetto, Luca
- Arnal, Jack D.
- Arrow, Holly
- Bakos, Bence E.
- Baník, Gabriel
- Baskin, Ernest
- Belopavlović, Radomir
- Bernstein, Michael H.
- Białek, Michał
- Bodroža, Bojana
- Boucher, Leanne
- Brühlmann, Florian
- Brumbaugh, Claudia C.
- Casini, Erica
- Chen, Yiling
- Chiorri, Carlo
- Chopik, William J.
- Christ, Oliver
- Claypool, Heather M.
- Coary, Sean
- Čolić, Marija V.
- Collins, W. Matthew
- Curran, Paul G.
- Dering, Benjamin
- Dreber, Anna
- Edlund, John E.
- Falcão, Filipe
- Fedor, Anna
- Ford, Máire
- Frank, Michael C.
- Garinther, Alexander
- Gawryluk, Katarzyna
- Giacomantonio, Mauro
- Giessner, Steffen R.
- Guadagno, Rosanna E.
- Hancock, Peter J. B.
- Hüffmeier, Joachim
- Hughes, Sean
- Idzikowska, Katarzyna
- Inzlicht, Michael
- Johannesson, Magnus
- Joy-Gaba, Jennifer A.
- Kauff, Mathias
- Kellier, Danielle J.
- Kidwell, Mallory C.
- King, Josiah P. J.
- Kołodziej, Sabina
- Krasuska, Karolina
- Krueger, Lacy E.
- Levitan, Carmel A.
- Lin, Hause
- Lins, Samuel
- Loy, Jia E.
- Manfredi, Dylan
- Markiewicz, Łukasz
- Mercier, Brett
- Millen, Ailsa E.
- Miller, Jeremy K.
- Montealegre, Andres
- Moore, Don A.
- Muda, Rafał
- Nave, Gideon
- Nichols, Austin Lee
- Orlić, Ana
- Panno, Angelo
- Pedović, Ivana
- Petrović, Boban
- Pfeiffer, Thomas
- Preti, Emanuele
- Purić, Danka
- Ravid, Jonathan
- Rentzsch, Katrin
- Richetin, Juliette
- Rife, Sean C.
- Rosa, Anna Dalla
- Salamon, Janos
- Saunders, Blair
- Sawicki, Przemysław
- Schmidt, Kathleen
- Schultze, Thomas
- Schütz, Astrid
- Siegel, Adam
- Steegen, Sara
- Stein, L. A. R.
- Stojilović, Darko
- Storage, Daniel
- Sullivan, Gavin Brent
- Szuts, Attila
- Thomae, Manuela
- Torka, Ann-Kathrin
- Tuerlinckx, Francis
- Vanpaemel, Wolf
- Vaughn, Leigh Ann
- Vianello, Michelangelo
- Viganola, Domenico
- Vlachou, Maria
- Weissgerber, Sophia C.
- Wichman, Aaron L.
- Wolf, Daniel
- Wood, Michael J.
- Žeželj, Iris
- Nosek, Brian A.
- Venue
- Advances in Methods and Practices in Psychological Science
- Publisher
- SAGE Publications
- Published
- 2020-09
- Volume
- 3
- Issue
- 3
- Pages
- 309-331
- ISSN
- 2515-2459 (ISSN-L)
- 2515-2467
- OpenAlex
- W2936257342
- Cited by
- 105
- References
- 62
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 14 · 2023: 19 · 2022: 30 · 2021: 13 · 2020: 7 · 2019: 1
- FWCI
- 10.52
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Advances in Methods and Practices in Psychological Science (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, other-oa)
- OSF Preprints (OSF Preprints) (acceptedVersion)
- Journal of Media Literacy Education (submittedVersion, cc-by)
- NSUWorks (Nova Southeastern University) (submittedVersion)
- Rose-Hulman Scholar (Rose–Hulman Institute of Technology) (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- HAL (submittedVersion)
- Ghent University Academic Bibliography (Ghent University) (submittedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- edoc (University of Basel) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- CINECA IRIS Institutial Research Information System (University of Genoa) (submittedVersion, other-oa)
- Lirias (submittedVersion)
- GoeScholar The Publication Server of the Georg-August-Universität Göttingen (Georg-August-Universität Göttingen) (submittedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Faculty of Philosophy (University of Belgrade) (submittedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Padua Research Archive (University of Padova) (submittedVersion, other-oa)
- Funders
- Laura and John Arnold Foundation
- Association for Psychological Science
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- replication study, replicability, Reproducibility Project: Psychology, replication failure, preregistration, effect size estimation, cumulative evidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
