The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Bayesian tests to quantify the result of a replication attempt.
Open accessAbstract
Replication attempts are essential to the empirical sciences. Successful replication attempts increase researchers' confidence in the presence of an effect, whereas failed replication attempts induce skepticism and doubt. However, it is often unclear to what extent a replication attempt results in success or failure. To quantify replication outcomes we propose a novel Bayesian replication test that compares the adequacy of 2 competing hypotheses. The 1st hypothesis is that of the skeptic and holds that the effect is spurious; this is the null hypothesis that postulates a zero effect size, H₀ : δ = 0. The 2nd hypothesis is that of the proponent and holds that the effect is consistent with the one found in the original study, an effect that can be quantified by a posterior distribution. Hence, the 2nd hypothesis-the replication hypothesis-is given by Hr : δ ∼ "posterior distribution from original study." The weighted-likelihood ratio between H₀ and Hr quantifies the evidence that the data provide for replication success and failure. In addition to the new test, we present several other Bayesian tests that address different but related questions concerning a replication study. These tests pertain to the independent conclusions of the separate experiments, the difference in effect size between the original experiment and the replication attempt, and the overall conclusion based on the pooled results. Together, this suite of Bayesian tests allows a relatively complete formalization of the way in which the result of a replication attempt alters our knowledge of the phenomenon at hand. The use of all Bayesian replication tests is illustrated with 3 examples from the literature. For experiments analyzed using the t test, computation of the new replication test only requires the t values and the numbers of participants from the original study and the replication study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Kendall's Advanced Theory of Statistics.
AA, Stuart, A., & Ord, K. (1995) · Journal of the American Statistical Association · doi:10.2307/2291187 · Cited by 2,619
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Computing Bayes Factors by Combining Simulation and Asymptotic Approximations
Diciccio, T. J., Kass, R. E., Raftery, A., & Wasserman, L. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474045 · Cited by 1,943
Default Bayes factors for ANOVA designs
Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.08.001 · Cited by 1,800
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407
Comparison processes in social judgment: Mechanisms and consequences.
Mussweiler, T. (2003) · Psychological Review · doi:10.1037/0033-295x.110.3.472 · Cited by 1,350
Kendall's Advanced Theory of Statistics.
Kemp, A. W., Stuart, A., & Ord, J. K. (1994) · The Statistician · doi:10.2307/2348968 · Cited by 1,349
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
An Introduction to the Bootstrap.
Kimber, A., Efron, B., & Tibshirani, R. J. (1994) · The Statistician · doi:10.2307/2348146 · Cited by 1,245
Introduction to Probability and Statistics
(2015) · Environmental Risk Analysis · doi:10.1201/b19097-5 · Cited by 1,153
Smith, J. Q., Bernardo, J. M., Berger, J. O., Dawid, A. P., & Smith, A. F. M. (1993) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982740 · Cited by 1,097
Mixtures of g Priors for Bayesian Variable Selection
Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008) · Journal of the American Statistical Association · doi:10.1198/016214507000001337 · Cited by 1,089
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Self-citationThe Intrinsic Bayes Factor for Model Selection and Prediction
Berger, J. O., & Pericchi, L. R. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476668 · Cited by 1,007
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0022790 · Cited by 977
Self-citationStatistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Self-citationBayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938
On assessing prior distributions and Bayesian regression analysis with g-prior distributions
Zellner, A. (1986) · Medical Entomology and Zoology · Cited by 886
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864
Self-citationCalibration of ρ Values for Testing Precise Null Hypotheses
Sellke, T., Bayarri, M. J., & Berger, J. O. (2001) · The American Statistician · doi:10.1198/000313001300339950 · Cited by 854
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
Statistical Evidence: A Likelihood Paradigm
Reeves, J. H., & Royall, R. M. (1998) · Journal of the American Statistical Association · doi:10.2307/2669869 · Cited by 814
Introduction to Probability and Statistics from a Bayesian Viewpoint
Lindley, D. V. (1965) · Cambridge University Press eBooks · doi:10.1017/cbo9780511662973 · Cited by 760
Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.12.001 · Cited by 740
Self-citationThe Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
The relation between perception and behavior, or how to win a game of Trivial Pursuit.
Dijksterhuis, A., & van Knippenberg, A. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.4.865 · Cited by 582
Bayes factor approaches for testing interval null hypotheses.
Morey, R. D., & Rouder, J. N. (2011) · Psychological Methods · doi:10.1037/a0024377 · Cited by 582
An EZ-diffusion model for response time and accuracy
Wagenmakers, E.-J., Van Der Maas, H. L. J., & Grasman, R. P. P. P. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194023 · Cited by 555
Self-citationThings I have learned (so far).
Cohen, J. (1992) · Methodological issues & strategies in clinical research. · doi:10.1037/10109-028 · Cited by 553
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528
A default Bayesian hypothesis test for correlations and partial correlations
Wetzels, R., & Wagenmakers, E.-J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0295-x · Cited by 510
Self-citationOptional stopping: No problem for Bayesians
Rouder, J. N. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0595-4 · Cited by 492
Default Bayes Factors for Model Selection in Regression
Rouder, J. N., & Morey, R. D. (2012) · Multivariate Behavioral Research · doi:10.1080/00273171.2012.734737 · Cited by 477
Posterior odds ratios for selected regression hypotheses
Zellner, A., & Siow, A. (1980) · Trabajos de Estadistica Y de Investigacion Operativa · doi:10.1007/bf02888369 · Cited by 463
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-033 · Cited by 395
The importance of proving the null
Gallistel, C. R. (2009) · Psychological Review · doi:10.1037/a0015251 · Cited by 392
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar-Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 368
Seeing one thing and doing another: Contrast effects in automatic behavior.
Dijksterhuis, A., Spears, R., Postmes, T., Stapel, D., Koomen, W., Knippenberg, A. V., & Scheepers, D. (1998) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.75.4.862 · Cited by 344
Selective attention: A reevaluation of the implications of negative priming.
Milliken, B., Joordens, S., Merikle, P. M., & Seiffert, A. E. (1998) · Psychological Review · doi:10.1037//0033-295x.105.2.203 · Cited by 325
Criteria for Bayesian model choice with application to variable selection
Bayarri, M. J., Berger, J. O., Forte, A., & García-Donato, G. (2012) · The Annals of Statistics · doi:10.1214/12-aos1013 · Cited by 293
The Weighted Likelihood Ratio, Sharp Hypotheses about Chances, the Order of a Markov Chain
Dickey, J. M., & Lientz, B. P. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697203 · Cited by 251
Priming Intelligent Behavior: An Elusive Phenomenon
Shanks, D. R., Newell, B. R., Lee, E. H., Balakrishnan, D., Ekelund, L., Cenac, Z., Kavvadia, F., & Moore, C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0056515 · Cited by 241
Replication in behavioral research.
Rosenthal, R. (1990) · Journal of social behavior and personality · Cited by 214
Bayesian Versus Frequentist Inference
Wagenmakers, E.-J., Lee, M., Lodewyckx, T., & Iverson, G. J. (2008) · Bayesian Evaluation of Informative Hypotheses · doi:10.1007/978-0-387-09612-4_9 · Cited by 205
Self-citationA Bayes factor meta-analysis of Bem’s ESP claim
Rouder, J. N., & Morey, R. D. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0088-7 · Cited by 194
de Groot, A. D. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.02.001 · Cited by 188
Red, rank, and romance in women viewing men.
Elliot, A. J., Niesta Kayser, D., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier, M. A., & Liu, H. (2010) · Journal of Experimental Psychology: General · doi:10.1037/a0019689 · Cited by 185
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability
Nosek, B. A., Spies, J. R., & Motyl, M. S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.4251 · Cited by 180
John, L., Loewenstein, G., & Prelec, D. (2012) · PsycEXTRA Dataset · doi:10.1037/e632032012-001 · Cited by 177
Prior sensitivity in theory testing: An apologia for the Bayes factor
Vanpaemel, W. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.07.003 · Cited by 164
Bayes factors: Prior sensitivity and model generalizability
Liu, C. C., & Aitkin, M. (2008) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2008.03.002 · Cited by 163
LeBel, E. P., & Peters, K. R. (2011) · Review of General Psychology · doi:10.1037/a0025172 · Cited by 162
The frequency of excess success for articles in Psychological Science
Francis, G. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0601-x · Cited by 130
A Default Bayesian Hypothesis Test for ANOVA Designs
Wetzels, R., Grasman, R. P. P. P., & Wagenmakers, E. (2012) · The American Statistician · doi:10.1080/00031305.2012.695956 · Cited by 128
Self-citationReplication, statistical consistency, and publication bias
Francis, G. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.02.003 · Cited by 126
Must psychologists change the way they analyze their data?
Bem, D. J., Utts, J., & Johnson, W. O. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0024777 · Cited by 105
Posterior Probabilities of Alternative Linear Models.
Gaver, K. M., & Lempers, F. B. (1973) · Journal of the American Statistical Association · doi:10.2307/2284117 · Cited by 104
Kendall's advanced theory of statistics. Vol.2B: Bayesian inference
O’Hagan, A., & Forster, J. J. (2004) · Cited by 99
Nobel laureate challenges psychologists to clean up their act
Yong, E. (2012) · Nature · doi:10.1038/nature.2012.11535 · Cited by 98
Kendall's Advanced Theory of Statistics, Vol. 2B: Bayesian Inference
Ankerst, D. P. (2005) · Journal of the American Statistical Association · doi:10.1198/jasa.2005.s62 · Cited by 92
Simple relation between Bayesian order-restricted and point-null hypothesis tests
Morey, R. D., & Wagenmakers, E. (2014) · Statistics & Probability Letters · doi:10.1016/j.spl.2014.05.010 · Cited by 80
Self-citationExtending conventional priors for testing general hypotheses in linear models
Bayarri, M. J., & García‐Donato, G. (2007) · Biometrika · doi:10.1093/biomet/asm014 · Cited by 77
Bayesian methods for assessing importance of effects.
Rouanet, H. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.1.149 · Cited by 75
Why Hypothesis Tests Are Essential for Psychological Science: A Comment on Cumming (2014)
Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E. (2014) · Psychological Science · doi:10.1177/0956797614525969 · Cited by 72
Self-citationThe Nine Circles of Scientific Hell
Neuroskeptic (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459519 · Cited by 64
Doing Bayesian Data Analysis : A Tutorial Introduction with R and BUGS
Kruschke, J. K. (2010) · Elsevier eBooks · Cited by 63
Response conflict reverses priming: A replication
Neill, W. T., & Kahan, T. A. (1999) · Psychonomic Bulletin & Review · doi:10.3758/bf03212334 · Cited by 52
Bayesian Statistical Inference for Psychological Research
Edwards, W., Lindman, H. R., & Savage, L. J. (1992) · Springer series in statistics · doi:10.1007/978-1-4612-0919-5_34 · Cited by 52
Bones, A. K. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612441216 · Cited by 51
LeBoeuf, R. A., & Estes, Z. (2004) · Social Cognition · doi:10.1521/soco.22.6.607.54817 · Cited by 43
Evaluating Replication ResultsPreprint
Simonsohn, U. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2259879 · Cited by 38
Bayesian Design of “Successful” Replications
Bayarri, M. J., & Mayoral, M. A. M. (2002) · The American Statistician · doi:10.1198/000313002155 · Cited by 32
Special Issue of Social Psychology on “Replications of Important Results in Social Psychology”
Nosek, B. A., & Lakens, D. (2013) · Social Psychology · doi:10.1027/1864-9335/a000143 · Cited by 26
More is not less: Greater information quantity does not diminish liking.
Ullrich, J., Krueger, J. I., Brod, A., & Groschupf, F. (2013) · Journal of Personality and Social Psychology · doi:10.1037/a0033183 · Cited by 21
Two useful distributions for Bayesian predictive procedures under normal models
Lecoutre, B. (1999) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(98)00231-6 · Cited by 21
Bayesian analysis and design for comparison of effect-sizes
Bayarri, M. J., & Mayoral, M. A. M. (2002) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(01)00223-3 · Cited by 15
Another Look at the Confidence Intervals for the Noncentral T Distribution
Lecoutre, B. (2007) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1177992600 · Cited by 14
di Pellegrino, G. (2011) · Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) · Cited by 13
Replications of Important Results in Social Psychology: Special Issue of Social PsychologyPreprint
Nosek, B. A., & Lakens, D. (2013) · Cited by 8
Cited by
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Self-citationCamerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Self-citationEvaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742
Metadata
- Authors
- Verhagen, J., & Wagenmakers, E.-J.
- Venue
- Journal of Experimental Psychology: General
- Published
- 2014-08
- Volume
- 143
- Issue
- 4
- Pages
- 1457-1475
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2155930071
- Cited by
- 291
- References
- 89
- FWCI
- 21.31
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.