A Bayesian Perspective on the Reproducibility Project: Psychology
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We revisit the results of the recent Reproducibility Project: Psychology by the Open Science Collaboration. We compute Bayes factors-a quantity that can be used to express comparative evidence for an hypothesis but also for the null hypothesis-for a large subset (N = 72) of the original papers and their corresponding replication attempts. In our computation, we take into account the likely scenario that publication bias had distorted the originally published results. Overall, 75% of studies gave qualitatively similar results in terms of the amount of evidence provided. However, the evidence was often weak (i.e., Bayes factor < 10). The majority of the studies (64%) did not provide strong evidence for either the null or the alternative hypothesis in either the original or the replication, and no replication attempts provided strong evidence in favor of the null. In all cases where the original paper provided strong evidence but the replication did not (15%), the sample size in the replication was smaller than the original. Where the replication provided strong evidence but the original did not (10%), the replication sample size was larger. We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. We further conclude that traditional sample sizes are insufficient and that a more widespread adoption of Bayesian methods is desirable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“reanalyses of the data suggest that reproducibility in psychological science might be higher than originally claimed (23, 26, 27).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
In the citing paper
“Second, and more importantly, because how reproducibility ought to be measured is the subject of a growing methodological and philosophical debate”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1972) · Cited by 17,806
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 2×
“Studies that do not meet the criteria 57 are relegated to lab file drawers [9].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“…research designs undermine the credibility of statistically significant results in addition to increasing the probability of nonsignificant ones (see (Button et al., 2013) and the references therein for a detailed argument); furthermore, low-powered studies generally provide only small amounts of…”
The American Psychological Association
Langfeld, H. S. (1919) · Science · doi:10.1126/science.49.1254.51 · Cited by 7,475
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“These two weaknesses of our field have conspired 285 to create a literature that is rife with false alarms [27].”
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 1×
“Our prior expectation regarding the 82 effect size under HA is represented by a normal distribution centered on zero with 83 variance equal to 1 (this is a unit information prior, which carries a weight equivalent to 84 approximately one observation [17]).”
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734 · cited 1×
“306 Decades of reliance on orthodox statistical inference—which is known to overstate 307 the evidence against a null hypothesis [29–32]—have obfuscated the widespread problem 308 of small samples in psychological studies in general and in replication studies specifically.”
Research works printed but not linked: 1
- Jeffreys H. Theory of Probability. 1st ed Oxford, UK: Oxford University Press; 1939.
Cited by
Items 101–200 of 293
Changing the logic of replication: A case from infant studies
Margoni, F., & Shepperd, M. (2020) · Infant Behavior and Development · doi:10.1016/j.infbeh.2020.101483 · Cited by 22
The Replication Crisis and Open Science in Psychology: Methodological Challenges and Developments
Renkewitz, F., & Heene, M. (2019) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000389 · Cited by 30
Ellis, R. J. (2022) · eNeuro · doi:10.1523/eneuro.0017-22.2022 · Cited by 30
“…et al., 2015), changing the standard threshold of statistical significance from 0.05 to 0.005 (Benjamin et al., 2018), adopting Bayesian statistics (Etz and Vandekerckhove, 2016), justifying a levels and sample sizes (Lakens et al., 2018; Lakens, 2021a), normalizing the publication of negative…”
BackgroundMethodInfluentialDecision mechanisms underlying mood-congruent emotional classification
White, C. N., Liebman, E., & Stone, P. (2017) · Cognition & Emotion · doi:10.1080/02699931.2017.1296820 · Cited by 30
“Given the problem of publication bias against null effects, it is likely that there are many more unpublished studies with null effects that have been relegated to the “file drawer” (Etz & Vandekerckhove, 2016).”
BackgroundPower to the People: A Beginner’s Tutorial to Power Analysis using jamovi
Bartlett, J. E., & Charles, S. J. (2022) · Meta-Psychology · doi:10.15626/mp.2021.3078 · Cited by 31
Redefine or justify? Comments on the alpha debate
de Ruiter, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1523-9 · Cited by 23
Psychology’s reproducibility problem is exaggerated – say psychologists
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.19498 · Cited by 21
Have psychologists increased reporting of outliers in response to the reproducibility crisis?
Valentine, K. D., Buchanan, E. M., Cunningham, A., Hopke, T. G., Wikowsky, A., & Wilson, H. (2021) · Social and Personality Psychology Compass · doi:10.1111/spc3.12591 · Cited by 25
Ulrich, R., Miller, J. O., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 21
“An application (Etz & Vandekerckhove, 2016) of this approach suggests that the failure to replicate Zeitschrift für Psychologie (2018), 226(1), 56–80 2018 Hogrefe Publishing 60 R. Ulrich et al., Effect Size Estimation Th is d oc um en t i s c op yr ig ht ed b y th e A m er ic an P sy ch ol og ic…”
BackgroundExploring the use of phonological and semantic representations in working memory.
Cowan, N., Guitard, D., Greene, N. R., & Fiset, S. (2022) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0001077 · Cited by 18
Researcher Degrees of Freedom in the Psychology of Religion
Charles, S. J., Bartlett, J. E., Messick, K. J., Coleman, T. J., III, & Uzdavines, A. (2019) · International Journal for the Psychology of Religion · doi:10.1080/10508619.2019.1660573 · Cited by 21
Reproducibility in Psychology: Theoretical Distinction of Different Types of Replications
Baucal, A., Gillespie, A., Krstić, K., & Zittoun, T. (2019) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-019-09499-y · Cited by 22
The probability of simple versus complex causal models in causal analyses
Trafimow, D. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0731-3 · Cited by 18
“Bayesians have suggested several cutoffs for the ratio needed to strongly support a model, with 10 being a common one (e.g., Etz & Vandekerckhove, 2016).”
BackgroundMachery, É. (2019) · Review of Philosophy and Psychology · doi:10.1007/s13164-019-00440-1 · Cited by 22
A critique of using the labels confirmatory and exploratory in modern psychological research
Jacobucci, R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1020770 · Cited by 28
Wu, C., Beattie, Z., Mattek, N., Sharma, N., Kaye, J., & Dodge, H. H. (2021) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1002/trc2.12220 · Cited by 18
“13,14 Thus, reproducing and replicating findings from previous studies has become important to increase the rigor of science and reduce study biases with a priori analytical plans.”
BackgroundLakens, D., & Etz, A. J. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/nnkg9 · Cited by 21
“Since these binomial probabilities ignore information (i.e., they treat all p .05 the same, regardless of whether p ¼ .049 or p ¼ .00001), a formal Bayesian analysis is the best approach when interested in quantifying one’s posterior belief in a hypothesis (e.g., see Etz & Vandekerckhove, 2016).”
BackgroundSelf-citationStatistical methods for replicability assessment
Hung, K. E., & Fithian, W. (2020) · The Annals of Applied Statistics · doi:10.1214/20-aoas1336 · Cited by 26
“Specifically with a truncated Gaussian model, Etz and Vandekerckhove (2016) have also analyzed the RP”
BackgroundMethodA Bayes Factor for Replications of ANOVA Results
Harms, C. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518787 · Cited by 24
“ues. Calculations and tools for common scenarios in psychological research exist (e.g. Rouder et al., 2012; Rouder and Morey, 2012; Rouder et al., 2009; Dienes, 2016). In the context of replications, Etz and Vandekerckhove (2016) have used Bayes factors to show that the original studies in the "Reproducibility Pr…”
MethodInfluentialHow (not) to measure replication
Fletcher, S. C. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00377-2 · Cited by 19
Reasoning supports forgiving accidental harms
Patil, I., & Trémolière, B. (2021) · Scientific Reports · doi:10.1038/s41598-021-93908-z · Cited by 15
Optimizing Research Output: How Can Psychological Research Methods Be Improved?
Miller, J. O., & Ulrich, R. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-094927 · Cited by 21
Beyond p values: utilizing multiple methods to evaluate evidence
Valentine, K. D., Buchanan, E. M., Scofield, J. E., & Beauchamp, M. T. (2019) · Behaviormetrika · doi:10.1007/s41237-019-00078-4 · Cited by 19
Insights into Criteria for Statistical Significance from Signal Detection Analysis
Witt, J. K. (2019) · Meta-Psychology · doi:10.15626/mp.2018.871 · Cited by 18
“Many studies are underpowered (e.g., Etz & Vandekerckhove, 2016; Fraley & Vazire, 2014; Ioannidis, 2005; Sedlmeier & Gigerenzer, 1989).”
Context Dependency as a Predictor of Replicability
Gollwitzer, M., & Schwabe, J. (2021) · Review of General Psychology · doi:10.1177/10892680211015635 · Cited by 18
“…to replicate an original effect is that it did exist, but the replication attempt was unlikely to find it either because it was underpowered (Etz & Vandekerckhove, 2016; Miller, 2009; Miller & Ulrich, 2016), the “replication success” criterion was underspecified, or the chance for…”
BackgroundKelter, R. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01613-6 · Cited by 17
“…to use in which setting for scientific hypothesis testing, and while some authors argue in favour of the Bayes factor (Wagen-makers et al., 2016; Etz and Vandekerckhove, 2016; Kelter, 2020b), there is also criticism about the focus on the Bayes factor in the cognitive sciences (Ten-deiro and…”
BackgroundMethodHow to become a Bayesian in eight easy steps: An annotated reading listPreprint
Etz, A. J., Gronau, Q. F., Dablander, F., Edelsbrunner, P. A., & Baribault, B. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ph6sw · Cited by 18
Self-citationOpen Access, Scholarly Communication, and Open Science in Psychology: An Overview for Researchers
Mullen, L. B. (2024) · SAGE Open · doi:10.1177/21582440231205390 · Cited by 23
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 16
A cognitive process modeling framework for the ABCD study stop-signal task
Weigard, A. S., Matzke, D., Tanis, C. C., & Heathcote, A. (2022) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2022.101191 · Cited by 19
“As several lines of work now underscore the importance of large, population-based samples for bolstering the reproducibility of behavioral and neuroscientific research (Button et al., 2013; Etz and Vandekerckhove, 2016; Falk et al., 2013; Loken and Gelman, 2017; Marek et al., 2022; Poldrack and Gorgolewski, 2014), the…”
BackgroundKoscelny, S. N., Sadralashrafi, S., & Neyens, D. M. (2025) · Applied Ergonomics · doi:10.1016/j.apergo.2025.104515 · Cited by 16
Mathur, M. B., & VanderWeele, T. J. (2019) · Psychological Methods · doi:10.1037/met0000223 · Cited by 13
Świątkowski, W., & Carrier, A. (2020) · Basic and Applied Social Psychology · doi:10.1080/01973533.2020.1792297 · Cited by 15
Oluoch, W. A., Whitney, C., Termote, C., Borgemeister, C., & Schmitt, C. B. (2022) · Human Ecology · doi:10.1007/s10745-022-00370-0 · Cited by 13
“We used Bayesian regression methods because of their reliability (Etz & Vandekerckhove, 2016), accuracy in small and noisy samples (Kruschke et al., 2012), possibility of introducing prior knowledge into the model (Andrews & Baguley, 2013; Kruschke et al., 2012), and intuitive nature of their…”
MethodBoyce, V., Mathur, M. B., & Frank, M. C. (2023) · Royal Society Open Science · doi:10.1098/rsos.231240 · Cited by 22
Reconstructing Recent Work on Macrosocial Stress as a Research Program
Witte, E. H., & Zenker, F. (2016) · Basic and Applied Social Psychology · doi:10.1080/01973533.2016.1207077 · Cited by 13
“…program strategy is free of various deficits that beset dominant research strategies, which having triggered strong doubt about whether the majority of published empirical research results are trustworthy (i.e., the “confidence crisis”; Etz & Vandekerckhove, 2016; Open Science Collaboration, 2015).”
BackgroundLow‐Choy, S., Riley, T., & Alston‐Knox, C. (2017) · Educational Media International · doi:10.1080/09523987.2017.1397404 · Cited by 13
“In this way, Bayesian updating helps address the new emphasis on reproducible research (e.g., Etz & Vandekerckhove, 2016), since it supports interpretation of statistical enquiry as a process that extends across multiple studies.”
BackgroundMakowski, D., Sperduti, M., Pelletier, J., Blondé, P., Corte, V. L., Arcangeli, M., Zalla, T., Lemaire, S., Dokic, J., Nicolas, S., et al. (2019) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-018-00681-0 · Cited by 14
“There is a general agreement that the generalization of the Bayesian approach is a way of overcoming these issues (Etz & Vandekerckhove, 2016).”
BackgroundHow Do We Choose Our Giants? Perceptions of Replicability in Psychological Science
Alister, M., Vickers-Jones, R., Sewell, D. K., & Ballard, T. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211018199 · Cited by 16
Tso, I. F., Taylor, S. F., & Johnson, T. D. (2021) · Journal of Abnormal Psychology · doi:10.1037/abn0000707 · Cited by 13
Sakaluk, J. K., & Graham, C. A. (2017) · The Journal of Sex Research · doi:10.1080/00224499.2017.1395387 · Cited by 13
“…heatedly—the extent of the replicability problem (e.g., Open Science Collaboration, 2015; versus Gilbert, King, Pettigrew, & Wilson, 2016; versus Etz & Vandekerckhove, 2016) and the merits of each solution du jour (e.g., Finkel, Eastwick, & Reis, 2017; versus LeBel, Berger, Campbell, &…”
BackgroundA Systems Approach to Understanding and Improving Research Integrity
Gorman, D. M., Elkins, A. D., & Lawley, M. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-017-9986-z · Cited by 14
Osimani, B. (2020) · Boston studies in the philosophy of science · doi:10.1007/978-3-030-29179-2_15 · Cited by 13
“While some analysts have provided formal confirmation for the plausibility of such explanations (Etz and Vandekerckhove 2016; Marsman et al. 2017), and some downplay the whole issue (Senn 2002), others have further insisted on the problem of noisy data and suggested that “to resolve the replication…”
BackgroundThe Design of Replication Studies
Hedges, L. V., & Schauer, J. M. (2021) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12688 · Cited by 17
“Moreover, while several scholars have commented on how to analyse replication studies, less attention has been paid to how to design them (see, e.g. Etz & Vandekerckhove, 2016; Gilbert et al., 2016; Hartgerink, Wicherts, & van Assen, 2017).”
BackgroundBayesian method in psychology: A bibliometric analysis
Jevremov, T., & Pajić, D. (2023) · Current Psychology · doi:10.1007/s12144-023-05003-3 · Cited by 10
“…by this research has shown that, although an increase in publishing Bayesian-related articles in psychology started in 2000s, more visible growth is noticeable during 2010s, which is the period that coincides with the beginning of replicability crisis in science (Etz & Vandekerckhove, 2016, 2018).”
How can we make sound replication decisions?
Davis‐Stober, C. P., Sarafoglou, A., Aczél, B., Chandramouli, S., Errington, T. M., Field, S. M., Fishbach, A., Freire, J., Ioannidis, J. P. A., Oberauer, K., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401236121 · Cited by 15
Self-citationWilliams, M. J., Mullane, K., & Curtis, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00005-7 · Cited by 13
A proposal for the assessment of replication of effects in single‐case experimental designs
Manolov, R., Tanious, R., & Fernández‐Castilla, B. (2022) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.923 · Cited by 15
“Finally, it does not require using more sophisticated options such as Bayesian analysis (Etz & Vandekerckhove, 2016), which can be harder to learn (Natesan, 2019) and are not likely to be included in courses for applied researchers (Wolfe &McCammon, 2022).”
MethodPower priors for replication studies
Pawel, S., Aust, F., Held, L., & Wagenmakers, E. (2023) · Test · doi:10.1007/s11749-023-00888-5 · Cited by 12
Blondé, P., Makowski, D., Sperduti, M., & Piolino, P. (2020) · Psychological Research · doi:10.1007/s00426-020-01358-5 · Cited by 16
“…as this approach is deemed to be more reliable, accurate, more straightforward to interpret and generally recommended by the recent guidelines of statistics in psychology (Andrews & Baguley, 2013; Etz & Vandekerckhove, 2016; Kruschke, 2010; Kruschke, Aguinis, & Joo, 2012; Wagenmakers et al., 2018).”
Love and Power: Grau and Pury (2014) as a Case Study in the Challenges of X-Phi Replication
Machery, É., Grau, C., & Pury, C. L. S. (2020) · Review of Philosophy and Psychology · doi:10.1007/s13164-020-00465-x · Cited by 11
“If replications are insufficiently powered—a not uncommon situation if the re-analysis of OSC 2015 by Etz and Vandekerckhove (2016) is on the right track—determining replication success and assessing replicability by means of statistical significance may be misleading, and a meta-analytic approach…”
BackgroundMethodInfluentialMoral Cognition About Harm in Anxiety Disorders: The Importance of Experienced Emotion
Patil, I., Larsen, E. M., Kichic, R., & Gleichgerrcht, E. L. (2020) · Psychological Reports · doi:10.1177/0033294120964134 · Cited by 11
“Based on prior guidelines (Etz & Vandekerckhove, 2016), BFs between 1 and 3, between 3 and 10, and larger than 10 are interpreted as ambiguous, moderate, and strong support, respectively.”
Exploring the intra-individual reliability of tDCS: A registered report
Willmot, N., Leow, L., Filmer, H. L., & Dux, P. E. (2024) · Cortex · doi:10.1016/j.cortex.2023.12.015 · Cited by 14
“There exists many potential reasons for this lack of effect, including the increased inter-individual variability inherent to large tDCS studies such as the present (Etz & Vandekerckhove, 2016; Minarik et al., 2016).”
Weedon, B. D., Liu, F., Mahmoud, W. J., Metz, R., Beunder, K., Delextrat, A. A., Morris, M. G., Esser, P., Collett, J., Meaney, A., et al. (2018) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2017-000288 · Cited by 11
A Bayes Factor for Replications of ANOVA ResultsPreprint
Harms, C. (2018) · OSF Preprints (OSF Preprints) · Cited by 9
Sharing imagery and analysis tools in a simulated submarine control room
Michailovs, S., Howard, Z. L., Pond, S., Fitzgerald, M., Visser, T. A. W., Bell, J., Pinniger, G. J., Irons, J. L., Schmitt, M., Stoker, M., et al. (2023) · Applied Ergonomics · doi:10.1016/j.apergo.2023.104125 · Cited by 10
Publication bias is bad for science if not necessarily scientists
Heesen, R., & Bright, L. K. (2025) · Royal Society Open Science · doi:10.1098/rsos.240688 · Cited by 12
Do red objects enhance sexual attractiveness? No evidence from two large replications
Pollet, T. V., Costello, J., Groeneboom, L., Peperkoorn, L. S., & Wu, J. (2018) · Displays · doi:10.1016/j.displa.2018.10.008 · Cited by 9
Priewasser, B., Fowles, F. A., Schweller, K., & Perner, J. (2019) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2019.104756 · Cited by 13
Protzko, J., & Schooler, J. W. (2020) · PeerJ · doi:10.7717/peerj.8014 · Cited by 9
Failing to replicate predicts citation declines in psychology
Clark, C. J., Connor, P. R., & Isch, C. G. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2304862120 · Cited by 11
“Often, it is not obvious whether a replication effort did or should cast doubt on original findings (10).”
BackgroundKrefeld-Schwalb, A., Witte, E. H., & Zenker, F. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00460 · Cited by 17
“Since all this keeps from having too much trust in our published empirical results, the term “confidence-crisis” is rather apt (Baker, 2015; Etz and Vandekerckhove, 2016).”
ResultAn evaluation of statistical methods for aggregate patterns of replication failure
Schauer, J. M., Fitzgerald, K. G., Peko-Spicer, S., Whalen, M. C. R., Zejnullahi, R., & Hedges, L. V. (2021) · The Annals of Applied Statistics · doi:10.1214/20-aoas1387 · Cited by 8
Berntsen, H., Ivarsson, A., & Kristiansen, E. (2019) · Current Issues in Sport Science (CISS) · doi:10.15203/ciss_2019.010 · Cited by 14
“In line with previous recommendations, a BF above 10 was determined to be in strong support of the alternative hypothesis (i.e., there is a statistical relationship between the two variables; Etz & Vandekerckhove, 2016).”
MethodA 21st century cognitive portrait of the Himba, a remote people of Namibia
Trémolière, B., Davidoff, J. B., & Caparos, S. (2021) · British Journal of Psychology · doi:10.1111/bjop.12539 · Cited by 9
Are P-values and Bayes factors valid measures of evidential strength?
Huisman, L. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02205-x · Cited by 8
“…nature of that debate, but also considering the recent and very practical problem with replication in many research areas (Camerer et al., 2018; Etz & Vandekerckhove, 2016; OSC, 2015), it is worth stepping back for a moment from the mere comparisons between these measures and reconsider their…”
BackgroundQuantifying Evidence for—and against—Granger Causality with Bayes Factors
Oravecz, Z., & Vandekerckhove, J. (2023) · Multivariate Behavioral Research · doi:10.1080/00273171.2023.2214890 · Cited by 10
Self-citationHeino, M. T. J., Knittle, K. P., Haukkala, A., Vasankari, T. J., & Hankonen, N. E. (2018) · BMC Public Health · doi:10.1186/s12889-018-6121-2 · Cited by 5
An Evidence-Hierarchical Decision Aid for Ranking in Evidence-Based Medicine
Landes, J. (2020) · Boston studies in the philosophy of science · doi:10.1007/978-3-030-29179-2_11 · Cited by 7
“…see (Chan and Altman, 2005, p. 1160), see (Button et al., 2013) for this problem in neuro-science, (Doll and Peto, 1980) in cancer research, (Etz and Vandekerckhove, 2016, p. 10) in psychology and a philosophical discussion of this problem in (Worrall, 2007a, p. 992) and (Bertamini and…”
BackgroundChanging-state irrelevant speech disrupts visual–verbal but not visual–spatial serial recall.
Marsh, J. E., Hurlstone, M. J., Marois, A., Ball, L. J., Moore, S. B., Vachon, F., Schlittmeier, S. J., Röer, J. P., Buchner, A., Aust, F., et al. (2024) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0001360 · Cited by 12
Dougherty, M. R. P., & Horne, Z. (2019) · doi:10.31234/osf.io/9g5wk · Cited by 6
Perra, O., Alderdice, F. A., Sweet, D. G., McNulty, A., Johnston, M., Bilello, D., Papageorgiou, K. Α., & Wass, S. V. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273767 · Cited by 6
Glass, D. J., & Flier, J. S. (2017) · Cell Metabolism · doi:10.1016/j.cmet.2017.10.010 · Cited by 6
“Indeed, there have been some focused efforts to try to reproduce some very high-profile papers (Errington et al., 2014; Etz and Vandekerckhove, 2016; see also http://validation.scienceexchange.com/#/ cancer-biology).”
BackgroundEqualBITE: Gender equality in higher education
Robertson, J. (2018) · BRILL eBooks · doi:10.1163/9789463511438 · Cited by 13
“If you prefer a Bayesian perspective on this project, see Etz & Vandekerckhove (2016).”
BackgroundReliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Two replications of Raymond, Shapiro, and Arnell (1992), The Attentional Blink
Grassi, M., Crotti, C., Giofrè, D., Boedker, I., & Toffalini, E. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01457-6 · Cited by 7
“For this reason, the picture of the effect returned by the original paper may not have been entirely veridical (see also Etz & Vandekerckhove, 2016; Morey & Lakens, 2016).”
BackgroundGerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
The "Tau" of Science - How to Measure, Study, and Integrate Quantitative and Qualitative KnowledgePreprint
Fanelli, D. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/67sak · Cited by 5
Replication for Quantitative Research
Freese, J., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Bayesian Tools of the Trade for Developmental Psychologists: A Quick-Start Guide Using JASP
Coon, J., Silva, P. N., Etz, A. J., & Sarnecka, B. W. (2024) · Journal of Cognition and Development · doi:10.1080/15248372.2024.2386032 · Cited by 8
“There are situations in which model averaging would be preferable (Etz et al., 2016; Gronau et al., 2021; Guan & Vandekerckhove, 2016).”
Self-citationDavis, C. E., Martin, J. G., & Thorpe, S. J. (2020) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2020.00002 · Cited by 4
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 5
Accelerating Understanding of Human Response to Automation Failure
Loft, S. D. (2024) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434241234108 · Cited by 5
“…can be certain for or against a hypothesis (e.g. BF>3 as ‘ weak evidence ’ , BF>5 as ‘ moderate evidence ’ , BF>10 as ‘ strong evidence ’ , and BF>100 as ‘ very strong evidence ’ ; Etz & Vandekerckhove, 2016), avoiding the use of a single arbitrary cut-off ( p < .05) for hypothesis testing.”
Can shared value and entrepreneurship contribute to SME development within rural locations?
Blanchard, K., & Gray, D. P. (2019) · Strategic Change · doi:10.1002/jsc.2256 · Cited by 4
Why It Is Problematic to Calculate Probabilities of Findings Given Range Null Hypotheses
Trafimow, D. (2017) · Open Journal of Statistics · doi:10.4236/ojs.2017.73034 · Cited by 7
Bayesian evaluation of effect size after replicating an original studyPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/g3sne · Cited by 6
Allocation of Space-Based Attention is Guided by Efficient Comprehension of Spatial Direction
Barnas, A. J., Ebner, N. C., & Weisberg, S. M. (2024) · Journal of Cognition · doi:10.5334/joc.325 · Cited by 3
Augmenting Human Cognition With a Digital Submarine Periscope
Michailovs, S., Irons, J. L., Howard, Z. L., Pond, S., Schmitt, M., Stoker, M., Visser, T. A. W., Bell, J., Pinniger, G. J., Fitzgerald, M., et al. (2024) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434241235770 · Cited by 3
“The commonly reported standard of BF >3 as ‘ weak evidence ’ , BF >5 as ‘ moderate evidence ’ , BF >10 as ‘ strong evidence ’ , and BF >100 as ‘ very strong evidence ’ (e.g. Etz & Vandekerckhove, 2016) is followed, avoiding the use of a single arbitrary cut-off for our hypothesis tests.”
BackgroundResearch is an Activity and a Subject of Study: A Proposed Metaconcept and Its Practical Application
Hosier, A. (2019) · College & Research Libraries · doi:10.5860/crl.80.1.44 · Cited by 8
Replicability of studies following a dual‐criterion design
Rosenkranz, G. K. (2021) · Statistics in Medicine · doi:10.1002/sim.9014 · Cited by 4
“(X) and an independent standard normal variable T̃ from reverting Equation (1): 𝜃(Y) = ?”
BackgroundInfluentialContributions towards understanding and building sustainable sciencePreprint
Hartgerink, C. H. J. (2019) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/4wtpc · Cited by 3
What should comparative cognition expect from direct replication studies?Preprint
Farrar, B. G., & Clayton, N. S. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4urm8 · Cited by 3
Dialogues about the practice of science
Shiffrin, R. M., Trueblood, J. S., Kellen, D., & Vandekerckhove, J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2423782122 · Cited by 5
Self-citationSeverinsen, L., Stiegler, J. R., Nissen‐Lie, H. A., Shahar, B., Bertelsen, T. B., & Zahl‐Olsen, R. (2025) · Psychotherapy Research · doi:10.1080/10503307.2025.2491477 · Cited by 4
“These allow us to distinguish between genuine evidence in favor of the null hypothesis in a scenario where there is insufficient data to draw strong conclusions (Etz & Vandekerckhove, 2016).”
Transparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 3
Programas de intervención y replicabilidad: consideraciones sobre su evaluación en psicología
Romo, J. F. M. (2021) · Revista Iberoamericana de Psicología · doi:10.33881/2027-1786.hrip.14108 · Cited by 6
On the Accuracy of Replication Failure Rates
Schauer, J. M. (2022) · Multivariate Behavioral Research · doi:10.1080/00273171.2022.2066500 · Cited by 3
“To date, several possible analysis procedures have been proposed and used (e.g., Etz & Vandekerckhove, 2016; Hedges & Schauer, 2019b; Simonsohn, 2015; van Aert & van Assen, 2017).”
BackgroundMethodHenry, N., Pedersen, M., Williams, M. N., & Donkin, L. (2025) · Archives of Sexual Behavior · doi:10.1007/s10508-025-03287-z · Cited by 2
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 3
Metadata
- Authors
- Etz, A., & Vandekerckhove, J.
- Affiliations
- University of Amsterdam (NL)
- University of California, Irvine (US)
- Corresponding author
- Etz, Alexander
- Vandekerckhove, Joachim
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-26
- Volume
- 11
- Issue
- 2
- Pages
- e0149794
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4769355
- OpenAlex
- W2282205924
- Cited by
- 295
- References
- 37
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 8 · 2023: 15 · 2022: 25 · 2021: 28 · 2020: 50 · 2019: 34 · 2018: 48 · 2017: 45 · 2016: 17
- FWCI
- 44.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation (1230118, 48192, 1534472)
- John Templeton Foundation (1534472, 1230118, 48192)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Reproducibility Project: Psychology, replication study, Bayes factor, publication bias, small sample size, effect size overestimation, Bayesian hypothesis testing, Open Science Collaboration
- MeSH terms
- Bayes Theorem, Humans, Psychology, Research Design, Reproducibility of Results, Publication Bias, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.