A Bayesian Perspective on the Reproducibility Project: Psychology
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We revisit the results of the recent Reproducibility Project: Psychology by the Open Science Collaboration. We compute Bayes factors-a quantity that can be used to express comparative evidence for an hypothesis but also for the null hypothesis-for a large subset (N = 72) of the original papers and their corresponding replication attempts. In our computation, we take into account the likely scenario that publication bias had distorted the originally published results. Overall, 75% of studies gave qualitatively similar results in terms of the amount of evidence provided. However, the evidence was often weak (i.e., Bayes factor < 10). The majority of the studies (64%) did not provide strong evidence for either the null or the alternative hypothesis in either the original or the replication, and no replication attempts provided strong evidence in favor of the null. In all cases where the original paper provided strong evidence but the replication did not (15%), the sample size in the replication was smaller than the original. Where the replication provided strong evidence but the original did not (10%), the replication sample size was larger. We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. We further conclude that traditional sample sizes are insufficient and that a more widespread adoption of Bayesian methods is desirable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
In the citing paper
“Second, and more importantly, because how reproducibility ought to be measured is the subject of a growing methodological and philosophical debate”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1972) · Cited by 17,806
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“Studies that do not meet the criteria 57 are relegated to lab file drawers [9].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…research designs undermine the credibility of statistically significant results in addition to increasing the probability of nonsignificant ones (see (Button et al., 2013) and the references therein for a detailed argument); furthermore, low-powered studies generally provide only small amounts of…”
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“These two weaknesses of our field have conspired 285 to create a literature that is rife with false alarms [27].”
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 1×
“Our prior expectation regarding the 82 effect size under HA is represented by a normal distribution centered on zero with 83 variance equal to 1 (this is a unit information prior, which carries a weight equivalent to 84 approximately one observation [17]).”
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · cited 1×
“306 Decades of reliance on orthodox statistical inference—which is known to overstate 307 the evidence against a null hypothesis [29–32]—have obfuscated the widespread problem 308 of small samples in psychological studies in general and in replication studies specifically.”
Research works printed but not linked: 1
- Jeffreys H. Theory of Probability. 1st ed Oxford, UK: Oxford University Press; 1939.
Cited by
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
“…(although the contribution 3Bayesian statistics might even be used to compare the weight of evidence for competing (null and alternative) hypotheses (Braver et al. 2014, Etz & Vandekerckhove 2016). www.annualreviews.org • Systematic Reviews 9.9 Review in Advance first posted on August 8, 2018.”
BackgroundInfluentialCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Makowski, D., Ben-Shachar, M., & Lüdecke, D. (2019) · Journal of Open Source Software · doi:10.21105/joss.01541 · Cited by 1,483
“…accuracy (particularly in noisy data and small samples), the possibility of incorporating prior knowledge into the analysis, and the intuitive interpretation of results (Andrews & Baguley, 2013; Etz & Vandekerckhove, 2016; Kruschke, 2010; Kruschke, Aguinis, & Joo, 2012; Wagenmakers et al., 2017).”
BackgroundPower Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
“At the same time, it is becoming clear that the definition of replication success may be less straightforward than assumed at first (Amrhein, Korner-Nievergelt, & Roth, 2017; Etz & Vandekerckhove, 2016).”
BackgroundIndices of Effect Existence and Significance in the Bayesian Framework
Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02767 · Cited by 983
“The Bayesian framework is quickly gaining popularity among psychologists and neuroscientists (Andrews and Baguley, 2013), for reasons such as flexibility, better accuracy in noisy data and small samples, less proneness to type I errors, the possibility of introducing prior knowledge into the analysis and the intuitive…”
BackgroundMethodReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
“First, they fail to account for the fact that replication attempts may be inconclusive, rather than definitive successes or failures (Etz & Vandekerckhove 2016, Simonsohn 2015c, Verhagen & Wagenmakers 2014).”
BackgroundResultDetecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Quintana, D. S., & Williams, D. R. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1761-4 · Cited by 408
“As we agree that Bayes factors should be interpreted in light of the research context [26] and wish to highlight the direct interpretability of Bayes factors, we do not characterize the”
BackgroundContextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
“According to a Bayesian reanalysis of the Reproducibility Project, one pair of authors argued that “the apparent discrepancy between the original set of results and the outcome of the Reproducibility Project can be explained adequately by the combination of deleterious publication practices and weak standards of evide…”
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
The psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science
Makowski, D. (2018) · The Journal of Open Source Software · doi:10.21105/joss.00470 · Cited by 379
Four reasons to prefer Bayesian analyses over significance testing
Dienes, Z., & Mclatchie, N. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1266-z · Cited by 354
“Another discussion worth having is whether this is good enough level of evidence; would it better to default to 6 (Cf. Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2015) or maybe 10 (Cf. Etz & Vandekerckhove, 2016)?”
BackgroundMethodThe relation between statistical power and inference in fMRI
Cremers, H. R., Wager, T. D., & Yarkoni, T. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184923 · Cited by 349
“…simply been overestimated, and that the true effect may be too small to consistently replicate without acquiring much larger samples (for a discussion see reference [62] and see [63] for an Bayesian perspective of the reproducibility project in psychology that illustrates the problem empirically).”
BackgroundA tutorial on Bayes Factor Design Analysis using an informed prior
Stefan, A. M., Gronau, Q. F., Schönbrodt, F. D., & Wagenmakers, E.-J. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-01189-8 · Cited by 284
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
“Looking at these numbers the replication crisis does not seem surprising: using NHST very high FRP can be expected even with modestly high H0:H1 odds and moderate bias (Etz and Vandekerckhove, 2016).”
BackgroundSchloss, P. D. (2018) · mBio · doi:10.1128/mbio.00525-18 · Cited by 280
“Solutions to these problems include preregistering data analysis plans (34–36), justifying sample sizes based on power calculations (10, 11, 37), and using Bayesian frameworks that allow prior knowledge of the system to influence the interpretation of new results (39, 40).”
BackgroundAcceptance of immersive head-mounted virtual reality in older adults
Huygelier, H., Schraepen, B., van Ee, R., Vanden Abeele, V., & Gillebert, C. R. (2019) · Scientific Reports · doi:10.1038/s41598-019-41200-6 · Cited by 299
Marginally Significant Effects as Evidence for Hypotheses: Changing Attitudes Over Four Decades
Pritschet, L., Powell, D., & Horne, Z. (2016) · Psychological Science · doi:10.1177/0956797616645672 · Cited by 268
“This apparent relaxation of the .05 criterion suggests a potentially troublesome state of affairs, as even significant p values may constitute weak or inconclusive evidence against the null hypothesis (e.g., Etz & Vandekerckhove, 2016; Good, 1992).”
BackgroundTen common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238
A tutorial on testing hypotheses using the Bayes factor.
Hoijtink, H., Mulder, J., van Lissa, C., & Gu, X. (2019) · Psychological Methods · doi:10.1037/met0000201 · Cited by 234
Nguyen, M.-H., Le, T.-T., Nguyen, H.-K. T., Ho, M.-T., Nguyen, H. T. T., & Vuong, Q.-H. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18073681 · Cited by 229
“However, regarding the properties of psychological research, we did try to optimize accuracy through our chosen methodology, Bayesian analysis [57].”
MethodIntroduction to Bayesian Inference for Psychology
Etz, A., & Vandekerckhove, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1262-3 · Cited by 195
Self-citationThe importance of sample size for reproducibility of tDCS effects
Minarik, T., Berger, B., Althaus, L., Bader, V., Biebl, B., Brotzeller, F., Fusban, T., Hegemann, J., Jesteadt, L., Kalweit, L., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00453 · Cited by 179
“For instance, the replication rate in psychological studies seems higher than originally reported (Open Science Collaboration, 2015) when Bayes statistics are used for data analysis (Etz and Vandekerckhove, 2016).”
MethodKelter, R. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00980-6 · Cited by 167
Metastudies for robust tests of theory
Baribault, B., Donkin, C., Little, D. R., Trueblood, J. S., Oravecz, Z., van Ravenzwaaij, D., White, C. N., De Boeck, P., & Vandekerckhove, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708285114 · Cited by 148
Self-citationLakens, D., & Etz, A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693058 · Cited by 145
“Since these binomial probabilities ignore information (i.e., they treat all p .05 the same, regardless of whether p ¼ .049 or p ¼ .00001), a formal Bayesian analysis is the best approach when interested in quantifying one’s posterior belief in a hypothesis (e.g., see Etz & Vandekerckhove, 2016).”
BackgroundSelf-citationSaunders, B., Milyavskaya, M., Etz, A., Randles, D., & Inzlicht, M. (2018) · Collabra: Psychology · doi:10.1525/collabra.134 · Cited by 132
“Bayes factors can be interpreted “as the degree to which the data sway our belief from one to the other hypothesis” (Etz & Vandekerckhove, 2016, p. 4).”
BackgroundSelf-citationThe Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 244
Dougherty, M. R. P., & Horne, Z. (2022) · Royal Society Open Science · doi:10.1098/rsos.220334 · Cited by 160
“All else being equal, studies that provide stronger statistical evidence are more likely to produce replicable effects [54], and hold promise for greater scientific impact because they are more likely to be observable across a wide variety of contexts and settings [55,56].”
BackgroundCamerer, C., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · doi:10.31235/osf.io/4hmb6 · Cited by 117
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 114
“For instance, the mean power achieved in the Psychology Reproducibility Project (Open Science Collaboration, 2015) was estimated to be 0.92, and the researchers expected to detect around 89 effects as statistically significant based on the 100 they tested, assuming these effects were true.”
MethodInfluentialHow to become a Bayesian in eight easy steps: An annotated reading list
Etz, A., Gronau, Q. F., Dablander, F., Edelsbrunner, P. A., & Baribault, B. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1317-5 · Cited by 112
“…Bayes factors, with values between 1 and 3, between 3 and 10, and greater than 10 generally taken to indicate inconclusive, weak, and strong evidence, respectively (see Jeffreys, 1961; Wagenmakers, 2007; Etz & Vandekerckhove, 2016), although different researchers may set different benchmarks.”
Self-citationAn Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology
de Ruiter, J. P., & Albert, S. (2017) · Research on Language and Social Interaction · doi:10.1080/08351813.2017.1262050 · Cited by 126
“…statistics (see, e.g., Dienes, 2011; Wagenmakers, 2007) in combination with a higher threshold for the minimal amount of required evidence (e.g., Etz & Vandekerckhove, 2016) would largely remedy this problem but are still controversial because of the reluctance of many researchers to specify…”
Backgroundvan der Zee, T., & Reich, J. (2018) · AERA Open · doi:10.1177/2332858418787466 · Cited by 134
“A large-scale effort to replicate 100 studies in the social sciences found that fewer than 50% of studies replicated (Open Science Collaboration, 2012, 2015; see also Etz & Vandekerckhove, 2016).”
BackgroundCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“A Bayesian analysis of the RPP project also highlighted that the evidential value of many of the original studies (and some of the replication studies) was too weak to support robust inferences (Etz & Vandekerckhove 2016).”
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95
Replications in Comparative Cognition: What Should We Expect and How Can We Improve?
Farrar, B., Boeckle, M., & Clayton, N. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.01.02.2020 · Cited by 89
“However, Etz and Vandekerckhove (2016) used a Bayesian analysis to suggest that “75% of [replication] studies gave qualitatively similar results [to the original studies]”, but also noted that “the majority of the studies (64”
Opportunities for increased reproducibility and replicability of developmental neuroimaging
Klapwijk, E. T., van den Bos, W., Tamnes, C. K., Raschle, N. M., & Mills, K. L. (2020) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2020.100902 · Cited by 91
“…on all-or-nothing significance testing (e.g., Wasserstein et al., 2019) Use of confidence intervals (e.g., Cumming, 2013) Bayesian modeling (e.g., Etz and Vandekerckhove, 2016) Behavior change interventions (see Norris and O’Connor, 2019) Scientist’s personal concerns (e.g., risk of being…”
BackgroundBradley, S. H., DeVito, N., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C., & Gill, P. J. (2020) · Journal of the Royal Society of Medicine · doi:10.1177/0141076820956799 · Cited by 112
“Some scientists suggest that the threshold of significance for p values should be reduced or abandoned altogether, that greater emphasis be placed on effect sizes and confidence intervals, or that Bayesian statistics should be used to provide greater discriminatory utility when appropriate.(42,43) The availability of…”
BackgroundBallantyne, N., Celniker, J. B., & Dunning, D. A. (2022) · Social Epistemology · doi:10.1080/02691728.2022.2146469 · Cited by 90
“Like lay inquirers, psychologists often risk generalizing from small samples (Etz and Vandekerckhove 2016; Simmons, Nelson, and Simonsohn 2011), seeking confirmatory evidence (Scheel, Mitchell, and Lakens 2021), and researching in ideologically homogeneous environments (Duarte et al. 2015).”
BackgroundJensen, S. N., Ivarsson, A., Fallby, J., Dankers, S., & Elbe, A. (2018) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2018.02.008 · Cited by 97
Making Replication MainstreamPreprint
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, B. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4tg9c · Cited by 92
Self-citationFinkel, E. J., Eastwick, P. W., & Reis, H. T. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000075 · Cited by 84
“Among the many contributions of the evidentiary value movement is an intensive emphasis on quantifying replicability, with scholars offering thoughtful discussions of how to quantify replication success and failure (e.g., Braver, Thoemmes, & Rosenthal, 2014; Etz & Vandekerckhove, 2016).”
BackgroundStatistical analyses for studying replication: Meta-analytic perspectives.
Hedges, L. V., & Schauer, J. M. (2019) · Psychological Methods · doi:10.1037/met0000189 · Cited by 69
“This is a common concern in many analyses of replications (e.g., Etz & Vandekerckhove, 2016, or van Aert & van Assen, 2017).”
BackgroundMethodA review of some emergent quantitative analyses in sport and exercise psychology
Myers, N. D., Ntoumanis, N., Gunnell, K. E., Gucciardi, D. F., & Lee, S. (2017) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2017.1317356 · Cited by 72
“…Jamil, & Wagenmakers, 2016), sequential hypothesis testing (Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, in press), analysis of single-subject designs (de Vries, Hartogs, & Morey, 2015), mixture modelling (Depaoli, 2013), and reproducibility efforts (Etz & Vandekerckhove, 2016).”
Pensgaard, A. M., Ivarsson, A., Nilstad, A., Solstad, B. E., & Steffen, K. (2018) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2017-000317 · Cited by 79
“We conducted the descriptive analyses using the JASP software package.(27) Data on player demographics are presented as means±SDs, including ranges.”
MethodPerceived crisis and reforms: Issues, explanations, and remedies.
De Boeck, P., & Jeon, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000154 · Cited by 64
More Than One Replication Study Is Needed for Unambiguous Tests of Replication
Hedges, L. V., & Schauer, J. M. (2019) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998619852953 · Cited by 74
“The analytic methods they used (and their conclusions) were almost immediately challenged (see, e.g., Etz & Vandekerckhove, 2016; Gilbert, King, Pettigrew, & Wilson, 2016; Hartgerink, Wicherts, & van Assen, 2017; van Aert & van Assen, 2017).”
BackgroundMethodInfluentialGagné, N., & Franzen, L. (2023) · Swiss Psychology Open · doi:10.5334/spo.34 · Cited by 79
A critical reexamination of doing arithmetic nonconsciously
Moors, P., & Hesselmann, G. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1292-x · Cited by 66
Bayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 61
“Both Etz and Vandekerckhove [22] and van Aert and van Assen [21] conclude that for many RPP findings no strong conclusions can be drawn on the magnitude of the underlying true effect size.”
InfluentialThe Effects of Disgust on Moral Judgments: Testing Moderators
Johnson, D. J., Wortman, J., Cheung, F., Hein, M., Lucas, R. E., Donnellan, M. B., Ebersole, C. R., & Narr, R. K. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616654211 · Cited by 62
“Based on the guidelines from Etz and Vandekerckhove (2016), BFs between 1 and 3, between 3 and 10, and larger than 10 are interpreted as ambiguous, moderate, and strong support, respectively.”
MethodHirsch, P., Koch, I., & Karbach, J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220150 · Cited by 74
Holczer, A., Németh, V. L., Vékony, T., Vécsei, L., Klivényi, P., & Must, A. (2020) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2020.00179 · Cited by 52
“For instance, the “Bayesian revolution” can add meaningful tools to revisit the results (Etz and Vandekerckhove, 2016).”
BackgroundA Bayesian bird's eye view of ‘Replications of important results in social psychology’
Marsman, M., Schönbrodt, F. D., Morey, R. D., Yao, Y., Gelman, A., & Wagenmakers, E.-J. (2017) · Royal Society Open Science · doi:10.1098/rsos.160426 · Cited by 50
Estimating the evidential value of significant results in psychological science
Aczél, B., Pálfi, B., & Szászi, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0182651 · Cited by 56
Open Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50
Li, C., Ivarsson, A., Lam, L. T.-M., & Sun, J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00665 · Cited by 67
Too Good to be False: Nonsignificant Results Revisited
Hartgerink, C. H. J., Wicherts, J. M., & van Assen, M. A. L. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.71 · Cited by 45
“Etz and Vandekerckhove (2016) reanalyzed the RPP at the level of individual effects, using Bayesian models incorporating publication bias.”
BackgroundMethodImproving the Replicability of Psychological Science Through Pedagogy
Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., Keil, M., Lampinen, A. K., Raposo, S., Reynolds, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917740427 · Cited by 48
“Next, following Etz and Vandekerckhove (2016), we compared Bayes factors between each original study and its replication study.”
BackgroundMethodFlexible yet fair: blinding analyses in experimental psychology
Dutilh, G., Sarafoglou, A., & Wagenmakers, E. (2019) · Synthese · doi:10.1007/s11229-019-02456-7 · Cited by 55
“For instance, the Open Science Collaboration (2015) showed that out of 100 replication studies, only 39 supported the conclusions that were drawn in the original article (but see Etz and Vandekerckhove 2016; see also Camerer et al. 2018).”
Practical Methodological Reform Needs Good Theory
Gervais, W. M. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620977471 · Cited by 56
“Bayes factors yield strong evidence in support of an alternative hypothesis (Etz & Vandekerckhove, 2016).”
BackgroundDiscounting model selection with area‐based measures: A case for numerical integration
Gilroy, S. P., & Hantula, D. A. (2018) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.318 · Cited by 41
Gordon, M., Viganola, D., Dreber, A., Johannesson, M., & Pfeiffer, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248780 · Cited by 44
Hardwicke, T. E., Szűcs, D., Thibault, R. T., Crüwell, S., van den Akker, O. R., Nuijten, M. B., & Ioannidis, J. P. A. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211040837 · Cited by 44
“…other potentially larger samples to aid interpretative clarity; for example, the extent to which the results of the largescale Reproducibility Project in Psychology (RPP; Open Science Collaboration, 2015) actually contradicted the original studies has been contested (Etz & Vandekerckhove, 2016).”
ResultWhen and Why to Replicate: As Easy as 1, 2, 3?
Field, S. M., Hoekstra, R., Bringmann, L. F., & van Ravenzwaaij, D. (2019) · Collabra Psychology · doi:10.1525/collabra.218 · Cited by 56
“Throughout this paper, we will use a relatively diffuse default prior distribution for effect size to reflect the fact that we do not possess strong prior information (see also Etz & Vandekerckhove, 2016).”
BackgroundMethodSaylors, R. G., & Trafimow, D. (2020) · Organizational Research Methods · doi:10.1177/1094428119893452 · Cited by 54
“Bayesians have suggested different levels for what counts as a convincing posterior ratio, but there has been a reasonable amount of convergence on 10 as a minimum bar (Etz & Vandekerckhove, 2016).”
BackgroundBored, distracted, and forgetful: The impact of mind wandering and boredom on memory encoding
Blondé, P., Sperduti, M., Makowski, D., & Piolino, P. (2021) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218211026301 · Cited by 59
“In line with recent statistical guidelines (Etz & Vandekerckhove, 2016; Wagenmakers et al., 2018), we used the Bayesian framework to analyse our data, using the packages rstanarm (Carpenter et al., 2017; Gabry & Goodrich, 2016) and psycho (Makowski, 2018).”
MethodOtte, W. M., Vinkers, C. H., Habets, P. C., van IJzendoorn, D. G. P., & Tijdink, J. K. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001562 · Cited by 48
Questionable research practices may have little effect on replicability
Ulrich, R., & Miller, J. O. (2020) · eLife · doi:10.7554/elife.58237 · Cited by 53
“…reducing a (Benjamin et al., 2018), (d) increasing statistical power (Button and Munafò, 2017), (e) improving statistical training (Asendorpf et al., 2013), (f) adopting Bayesian approaches (Etz and Vandekerckhove, 2016), and even (g) overhauling standard scientific methodology (Barrett, 2020).”
Backgroundvan Ravenzwaaij, D., & Wagenmakers, E. (2022) · Psychological Methods · doi:10.1037/met0000415 · Cited by 46
Conditional equivalence testing: An alternative remedy for publication bias
Campbell, H., & Gustafson, P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195145 · Cited by 45
“Bayesian methods may also provide for a better understanding of the strength of evidence (Etz & Vandekerckhove, 2016).”
MethodInternal conceptual replications do not increase independent replication success
Kunert, R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1030-9 · Cited by 36
fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience
Yücel, M. A., Luke, R., Mesquita, R. C., von Lühmann, A., Mehler, D. M. A., Lührs, M., Gemignani, J., Abdalmalak, A., Albrecht, F., Ivo, I. D. A., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08412-1 · Cited by 57
“These reports have catalyzed both meta-research to explore the factors leading to diminished replication and reproduction rates 4 – 8 , and the launch of initiatives focused on improving these critical aspects of scienti fi c research 9 – 11 .”
Assessing heterogeneity and power in replications of psychological experiments.
Schauer, J. M., & Hedges, L. V. (2020) · Psychological Bulletin · doi:10.1037/bul0000232 · Cited by 41
The Effect of Replications on Citation Patterns: Evidence From a Large-Scale Reproducibility Project
Schafmeister, F. (2021) · Psychological Science · doi:10.1177/09567976211005767 · Cited by 45
“In line with this argument, the Bayesian analysis by Etz and Vandekerckhove (2016) suggests that many original studies and their replications did not provide strong evidence for either the null or the alternative hypothesis, implying a limited need for belief updating in many cases.”
ResultThe Impact of Complexity on Methods and Findings in Psychological Science
Sanbonmatsu, D. M., Cooley, E. H., & Butner, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580111 · Cited by 50
“This is evidenced not only by failures to replicate but by analyses showing that there are significantly more reports of studies that reject the null hypothesis than is consistent with a power or Bayesian analysis of the body of work (e.g., Renkewitz et al., 2011; Etz and Vandekerckhove, 2016).”
ResultA theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
Psychology’s Replication Crisis as Scientific Opportunity: A Précis for Policymakers
Rodgers, J. L., & Shrout, P. E. (2017) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732217749254 · Cited by 40
“Fourth, Etz and Vandekerckhove (2016) used a Bayesian method to provide support for the seriousness of the findings from the OSC study.”
MethodQureshi, A., Monk, R. L., Pennington, C. R., Wilcockson, T. D. W., & Heim, D. (2018) · Addictive Behaviors · doi:10.1016/j.addbeh.2018.11.034 · Cited by 33
Will I Regret It? Anticipated Negative Emotions Modulate Choices in Moral Dilemmas
Pletti, C., Lotto, L., Tasso, A., & Sarlo, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01918 · Cited by 40
“Following the guidelines by Etz and Vandekerckhove (2016), BF10s between one and three are interpreted as ambiguous, between 3 and 10 as moderately in favor of H1, larger than 10 as strongly in favor of H1.”
MethodFrom Discovery to Justification: Outline of an Ideal Research Program in Empirical Psychology
Witte, E. H., & Zenker, F. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01847 · Cited by 38
“Worse still, a Bayesian approach finds clear and consistent results in merely 11% of 72 reanalyzed datasets (Etz and Vandekerckhove, 2016).”
BackgroundResultInfluentialPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Dispositional mindfulness attenuates the emotional attentional blink
Makowski, D., Sperduti, M., Lavallée, S., Nicolas, S., & Piolino, P. (2019) · Consciousness and Cognition · doi:10.1016/j.concog.2018.11.004 · Cited by 28
“The analysis was performed in the Bayesian framework as it appeared more reliable, with better accuracy in noisy or small data samples, a more straightforward interpretation and less prone to type I error (Andrews & Baguley, 2013; Etz and Vandekerckhove, 2016; Kruschke, 2010; Kruschke, Aguinis, & Joo, 2012; Wagenmaker…”
MethodEnhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
New Statistical Metrics for Multisite Replication Projects
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12572 · Cited by 28
“Existing work (Andrews and Kasy, 2019; Etz and Vandekerckhove, 2016; Patil et al., 2016; Simonsohn, 2015; Verhagen and Wagenmakers, 2014) has proposed analytic approaches for a single replication of a single study or designs in which numerous original studies across a discipline or domain are each…”
BackgroundToward open science in PLS-SEM: Assessing the state of the art and future perspectives
Adler, S. J., Sharma, P. N., & Radomir, L. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114291 · Cited by 44
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0967-6 · Cited by 36
“Applying the hybrid methods to studies of RPP largely confirmed the results of only the replication study as reported by the Open Science Collaboration (2015).”
InfluentialStatistical evidence in psychological networks
Huth, K. B. S., Haslbeck, J. M. B., Keetelaar, S., van Holst, R. J., & Marsman, M. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02314-2 · Cited by 44
Figdor, C. (2017) · Frontiers in Communication · doi:10.3389/fcomm.2017.00003 · Cited by 46
“(2012); Flier (2016); Bohannon (2015); Etz and Vanderkerckhove (2016).”
The heart of cognitive control: Cardiac phase modulates processing speed and inhibition
Makowski, D., Sperduti, M., Blondé, P., Nicolas, S., & Piolino, P. (2019) · Psychophysiology · doi:10.1111/psyp.13490 · Cited by 35
“…under the Bayesian framework (which demonstrated better reliability in noisy data and better estimation for small samples; Andrews & Baguley, 2013; Etz & Vandekerckhove, 2016; Kruschke, 2010; Kruschke, T A B L E 1 CoCon stimuli list Part n stimuli Global shape Global color Local shape Local color…”
BackgroundThe Sceptical Bayes Factor for the Assessment of Replication Success
Pawel, S., & Held, L. (2022) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12491 · Cited by 34
Assessing and Improving Robustness of Psychological Research Findings in Four Steps
Nuijten, M. B. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_17 · Cited by 28
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
The illusion of science in comparative cognitionPreprint
Farrar, B. G., & Ostojić, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hduyx · Cited by 31
Saunders, B., Milyavskaya, M., Etz, A. J., Randles, D., & Inzlicht, M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bxfsu · Cited by 32
Self-citationPennington, C. R., Litchfield, D., McLatchie, N. M., & Heim, D. (2018) · European Journal of Social Psychology · doi:10.1002/ejsp.2540 · Cited by 33
“0.33 are interpreted as moderate evidence for the null hypothesis., (cf. Etz & Vandekerckhove, 2016).”
BackgroundInfluentialGlöckner, A., Fiedler, S., & Renkewitz, F. (2018) · Psychologische Rundschau · doi:10.1026/0033-3042/a000384 · Cited by 28
Metadata
- Authors
- Etz, A., & Vandekerckhove, J.
- Affiliations
- University of Amsterdam (NL)
- University of California, Irvine (US)
- Corresponding author
- Etz, Alexander
- Vandekerckhove, Joachim
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-26
- Volume
- 11
- Issue
- 2
- Pages
- e0149794
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4769355
- OpenAlex
- W2282205924
- Cited by
- 295
- References
- 37
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 8 · 2023: 15 · 2022: 25 · 2021: 28 · 2020: 50 · 2019: 34 · 2018: 48 · 2017: 45 · 2016: 17
- FWCI
- 44.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation (1230118, 48192, 1534472)
- John Templeton Foundation (1534472, 1230118, 48192)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Reproducibility Project: Psychology, replication study, Bayes factor, publication bias, small sample size, effect size overestimation, Bayesian hypothesis testing, Open Science Collaboration
- MeSH terms
- Bayes Theorem, Humans, Psychology, Research Design, Reproducibility of Results, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.