The DOI matches its Crossref record; 80 of the 81 references it cites were found in a registry or an index.
Bayesian Meta-Analysis with Weakly Informative Prior Distributions
Open accessAbstract
Developing meta-analytic methods is an important goal for psychological science. When there are few studies in particular, commonly used methods have several limitations, most notably of which is underestimating between-study variability. Although Bayesian methods are often recommended for small sample situations, their performance has not been thoroughly examined in the context of meta-analysis. Here, we characterize and apply weakly informativepriors for estimating meta-analytic models and demonstrate with extensive simulations that fully Bayesian methods overcome boundary estimates of exactly zero between-study variance, better maintain error rates, and have lower frequentist risk according toKullback-Leibler divergence. While our results show that combining evidence with few studiesis non-trivial, we argue that this is an important goal that deserves further considerationin psychology. Further, we suggest that frequentist properties can provide importantinformation for Bayesian modeling. We conclude with meta-analytic guidelines for appliedresearchers that can be implemented with the provided computer code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“12 Results from a complementary Bayesian multilevel meta-analysis model (Harrer etal., 2019; Williams etal., 2018) with weakly informative priors for true pooled effect size (µ ~ N (0, 1)) and between-study heterogeneity ( τ ~ HC (0, 0.5)) aligned with these pooled effect size estimates: Hedge's g = 0.36, 95% CI [0.19, 0.52].”
From When Problem Solving Followed by Instruction Works: Evidence for Productive Failure (2021) · cites this paper as Williams et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476
Self-citationStatistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte CarloPreprint
Homan, M. D., & Gelman, A. (2014) · arXiv (Cornell University) · Cited by 3,078
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Park, T., & Casella, G. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000337 · Cited by 2,941
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings.
Raudenbush, S. W., Hunter, J. E., & Schmidt, F. L. (1991) · Journal of the American Statistical Association · doi:10.2307/2289738 · Cited by 2,368
A weakly informative default prior distribution for logistic and other regression models
Gelman, A., Jakulin, A., Pittau, M. G., & Su, Y.-S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas191 · Cited by 2,115
A Conceptual Introduction to Hamiltonian Monte CarloPreprint
Betancourt, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.02434 · Cited by 1,766
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician
Rubin, D. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346785 · Cited by 1,503
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343
Why We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Integrative data analysis: The simultaneous analysis of multiple data sets
Curran, P. J., & Hussong, A. M. (2009) · Psychological Methods · doi:10.1037/a0015914 · Cited by 867
Turner, R. M., Davey, J., Clarke, M. J., Thompson, S. G., & Higgins, J. P. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dys041 · Cited by 837
Hunter, J. E., & Schmidt, F. L. (2000) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00156 · Cited by 833 · cited 1×
“Namely, too small variance estimates typically result in liberal estimates of µ, as the confidence intervals are too narrow (Hunter & Schmidt, 2000).”
BackgroundBayesian methods in meta-analysis and evidence synthesis
Sutton, A. J., & Abrams, K. R. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000404 · Cited by 734
Schmidt, F. L., Oh, I., & Hayes, T. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711007x255327 · Cited by 729
On Using Bayesian Methods to Address Small Sample Problems
McNeish, D. (2016) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2016.1186549 · Cited by 612
A comparison of Bayesian and likelihood-based methods for fitting multilevel models
Browne, W. J., & Draper, D. (2006) · Bayesian Analysis · doi:10.1214/06-ba117 · Cited by 603
A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models
Chung, Y., Rabe‐Hesketh, S., Dorie, V., Gelman, A., & Liu, J. (2013) · Psychometrika · doi:10.1007/s11336-013-9328-2 · Cited by 598
A comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 1×
“An important goal for psychological science is developing quantitative methods that can combine evidence from multiple studies (Carlsson, Schimmack, Williams, & Bürkner, 2017; Scheibehenne, Jamil, & Wagenmakers, 2016; Schmidt, 1992).”
BackgroundLambert, P. C., Sutton, A. J., Burton, P. R., Abrams, K. R., & Jones, D. R. (2005) · Statistics in Medicine · doi:10.1002/sim.2112 · Cited by 511
The Prior Can Often Only Be Understood in the Context of the Likelihood
Gelman, A., Simpson, D., & Betancourt, M. (2017) · Entropy · doi:10.3390/e19100555 · Cited by 470
On the Half-Cauchy Prior for a Global Scale Parameter
Polson, N. G., & Scott, J. G. (2012) · Bayesian Analysis · doi:10.1214/12-ba730 · Cited by 456
Abandon Statistical SignificancePreprint
McShane, B. B., Gal, D., Gelman, A., Robert, C. P., & Tackett, J. L. (2019) · HAL · Cited by 449
Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist.
Depaoli, S., & van de Schoot, R. (2017) · Psychological Methods · doi:10.1037/met0000065 · Cited by 448
Analyzing effect sizes: Random-effects models.
Raudenbush, S. W. (2009) · Cited by 418
Confidence intervals for the amount of heterogeneity in meta‐analysis
Viechtbauer, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2514 · Cited by 399
Rhodes, K. M., Turner, R., & Higgins, J. P. T. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.08.012 · Cited by 398
Type S error rates for classical and Bayesian single and multiple comparison procedures
Gelman, A., & Tuerlinckx, F. (2000) · Computational Statistics · doi:10.1007/s001800000040 · Cited by 380 · cited 1×
“Further, there are other concepts such as type S (sign) and M (magnitude) errors (Gelman & Carlin, 2014; Gelman & Tuerlinckx, 2000), which are arguably more appropriate to describe errors in statistical inference than type I and II errors (Chen et al., 2017; Gelman, Hill, & Yajima, 2012).”
BackgroundA comparison of heterogeneity variance estimators in combining results of studies
Sidik, K., & Jonkman, J. N. (2007) · Statistics in Medicine · doi:10.1002/sim.2688 · Cited by 380
Barnes, T. R. E. (2011) · Journal of Psychopharmacology · doi:10.1177/0269881110391123 · Cited by 369
Overton, R. C. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.3.354 · Cited by 365
Could Fisher, Jeffreys and Neyman Have Agreed on Testing?
Berger, J. O. (2003) · Statistical Science · doi:10.1214/ss/1056397485 · Cited by 348
Validation of Software for Bayesian Models Using Posterior Quantiles
Cook, S. R., Gelman, A., & Rubin, D. B. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x136976 · Cited by 345
Testing Negative Error Variances
Kolenikov, S., & Bollen, K. A. (2012) · Sociological Methods & Research · doi:10.1177/0049124112442138 · Cited by 319
Faster estimation of Bayesian models in ecology using Hamiltonian Monte Carlo
Monnahan, C. C., Thorson, J. T., & Branch, T. A. (2016) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12681 · Cited by 287
A re‐evaluation of fixed effect(s) meta‐analysis
Rice, K., Higgins, J. P. T., & Lumley, T. (2018) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12275 · Cited by 287
Simple Heterogeneity Variance Estimation for Meta-Analysis
Sidik, K., & Jonkman, J. N. (2005) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2005.00489.x · Cited by 254
Maximum Likelihood, Profile Likelihood, and Penalized Likelihood: A Primer
Cole, S. R., Chu, H., & Greenland, S. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kwt245 · Cited by 220
Invited Commentary: The Need for Cognitive Science in Methodology
Greenland, S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx259 · Cited by 212
Random-effects meta-analysis: The number of studies matters
Guolo, A., & Varin, C. (2017) · Statistical Methods in Medical Research · doi:10.1177/0962280215583568 · Cited by 203
Meta‐analysis of few small studies in orphan diseases
Friede, T., Röver, C., Wandel, S., & Neuenschwander, B. E. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1217 · Cited by 187
Remove, rather than redefine, statistical significance
Amrhein, V., & Greenland, S. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0224-0 · Cited by 182
Kenny, D. A., & Judd, C. M. (2019) · Psychological Methods · doi:10.1037/met0000209 · Cited by 167
Mallinckrodt, C. H., Clark, W. S., & David, S. R. (2001) · Drug Information Journal · doi:10.1177/009286150103500418 · Cited by 160
Gronau, Q. F., van Erp, S., Heck, D. W., Cesario, J., Jonas, K. J., & Wagenmakers, E. (2017) · Comprehensive Results in Social Psychology · doi:10.1080/23743603.2017.1326760 · Cited by 153
Higgins, J. P. T., & Spiegelhalter, D. J. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.96 · Cited by 123
Underreporting in Psychology Experiments
Franco, A., Malhotra, N. A., & Simonovits, G. (2015) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 119
Schmidt, F. M. (2007) · Organizational Research Methods · doi:10.1177/1094428107303161 · Cited by 117
Greenland, S., & Poole, C. (2012) · Epidemiology · doi:10.1097/ede.0b013e3182785741 · Cited by 115
Avoiding zero between‐study variance estimates in random‐effects meta‐analysis
Chung, Y., Rabe‐Hesketh, S., & Choi, I. (2013) · Statistics in Medicine · doi:10.1002/sim.5821 · Cited by 109
HDInterval: Highest (Posterior) Density Intervals
Meredith, M., & Kruschke, J. K. (2016) · doi:10.32614/cran.package.hdinterval · Cited by 98
Bayesian Evidence Synthesis Can Reconcile Seemingly Inconsistent Results
Scheibehenne, B., Jamil, T., & Wagenmakers, E. (2016) · Psychological Science · doi:10.1177/0956797616644081 · Cited by 82
Bürkner, P., Williams, D. R., Simmons, T. C., & Woolley, J. D. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx053 · Cited by 82
Self-citationClarke, K. A. (2009) · Conflict Management and Peace Science · doi:10.1177/0738894208097666 · Cited by 79
Notes on Kullback-Leibler Divergence and LikelihoodPreprint
Shlens, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2000 · Cited by 77
USING FULL PROBABILITY MODELS TO COMPUTE PROBABILITIES OF ACTUAL INTEREST TO DECISION MAKERS
Harrell, F. E., & Shih, Y. T. (2001) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462301104034 · Cited by 72
An informed reference prior for between‐study heterogeneity in meta‐analyses of binary outcomes
Pullenayegum, E. (2011) · Statistics in Medicine · doi:10.1002/sim.4326 · Cited by 64
Effects of intranasal oxytocin on symptoms of schizophrenia: A multivariate Bayesian meta-analysis
Williams, D. R., & Bürkner, P. (2016) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2016.10.013 · Cited by 58
Self-citationBayesian evaluation of effect size after replicating an original study
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175302 · Cited by 57
Rhodes, K. M., Turner, R., White, I. R., Jackson, D., Spiegelhalter, D. J., & Higgins, J. P. T. (2016) · Statistics in Medicine · doi:10.1002/sim.7090 · Cited by 49
Applying Bayesian ideas in simulation
Andradóttir, S., & Bier, V. M. (2000) · Simulation Practice and Theory · doi:10.1016/s0928-4869(00)00025-2 · Cited by 41
metaBMA: Bayesian Model Averaging for Random and Fixed Effects Meta-Analysis
Heck, D. W. (2017) · doi:10.32614/cran.package.metabma · Cited by 36
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386.fmatter · Cited by 28
CALIBRATING BAYES FACTOR UNDER PRIOR PREDICTIVE DISTRIBUTIONS
Garc, G., & Chen, M. (2005) · Cited by 23
A Review of Statistical Rethinking: A Bayesian Course With Examples in R and Stan
Sweet, T. M. (2016) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998616659752 · Cited by 20
Carlsson, R., Schimmack, U., Williams, D. R., & Bürkner, P. (2017) · Psychological Science · doi:10.1177/0956797616684682 · Cited by 19
Self-citationBijlani, R. (2008) · Jaypee Brothers Medical Publishers (P) Ltd. eBooks · doi:10.5005/jp/books/10519 · Cited by 15
Hoijtink, H., van Kooten, P., & Hulsker, K. (2016) · Multivariate Behavioral Research · doi:10.1080/00273171.2015.1071705 · Cited by 11
Gelman, A., van Dyk, D. A., Huang, Z., & Boscardin, W. J. (2007) · Cited by 10
Semi-Separable Hamiltonian Monte Carlo for Inference in Bayesian Hierarchical ModelsPreprint
Zhang, Y., & Sutton, C. A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3843 · Cited by 9
Handling Multiplicity in Neuroimaging through Bayesian Lenses with Multilevel ModelingPreprint
Chen, G., Xiao, Y., Taylor, P. A., Rajendra, J. K., Riggins, T., Geng, F., Redcay, E., & Cox, R. W. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/238998 · Cited by 8
Cited by
Sorted by citations
A global meta-analysis of soil organic carbon in the Anthropocene
Beillouin, D., Corbeels, M., Demenois, J., Berre, D., Boyer, A., Fallot, A., Feder, F., & Cardinael, R. (2023) · Nature Communications · doi:10.1038/s41467-023-39338-z · Cited by 524
Human disturbance causes widespread disruption of animal movement
Doherty, T. S., Hays, G. C., & Driscoll, D. A. (2021) · Nature Ecology & Evolution · doi:10.1038/s41559-020-01380-1 · Cited by 299
When Problem Solving Followed by Instruction Works: Evidence for Productive Failure
Sinha, T., & Kapur, M. (2021) · Review of Educational Research · doi:10.3102/00346543211019105 · Cited by 212
“12 Results from a complementary Bayesian multilevel meta-analysis model (Harrer et al., 2019; Williams et al., 2018) with weakly informative priors for true pooled effect size (µ ~N(0, 1)) and between-study heterogeneity (τ ~ HC(0, 0.5)) aligned with these pooled effect size estimates: Hedge’s g =…”
InfluentialRöver, C., Bender, R., Dias, S., Schmid, C. H., Schmidli, H., Sturtz, S., Weber, S., & Friede, T. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1475 · Cited by 209
Reptile responses to anthropogenic habitat modification: A global meta‐analysis
Doherty, T. S., Balouch, S., Bell, K., Burns, T. J., Feldman, A., Fist, C. N., Garvey, T. F., Jessop, T. S., Meiri, S., & Driscoll, D. A. (2020) · Global Ecology and Biogeography · doi:10.1111/geb.13091 · Cited by 182
Metadata
- Authors
- Williams, D. R., Rast, P., & Bürkner, P. C.
- Affiliations
- University of California, Davis (US)
- University of Münster (DE)
- Publisher
- Center for Open Science
- Published
- 2018-01-10
- OpenAlex
- W2785297454
- Cited by
- 124
- References
- 81
- Citations by year
- 2026: 18 · 2025: 18 · 2024: 27 · 2023: 18 · 2022: 14 · 2021: 15 · 2020: 10 · 2019: 4
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- Bayesian meta-analysis, weakly informative priors, study heterogeneity, Kullback-Leibler divergence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
