Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models
Open accessPublished version of a preprint
The DOI matches its Crossref record; 26 of the 28 references it cites were found in a registry or an index.
Abstract
In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the exact form of the dependence is unknown. RVE uses a working model of the dependence structure, but the two currently available working models are limited to each describing a single type of dependence. Drawing on flexible tools from multilevel and multivariate meta-analysis, this paper describes an expanded range of working models, along with accompanying estimation methods, which offer potential benefits in terms of better capturing the types of data structures that occur in practice and, under some circumstances, improving the efficiency of meta-regression estimates. We describe how the methods can be implemented using existing software (the “metafor” and “clubSandwich” packages for R), illustrate the proposed approach in a meta-analysis of randomized trials on the effects of brief alcohol interventions for adolescents and young adults, and report findings from a simulation study evaluating the performance of the new methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Because effect sizes within studies are statistically dependent, a three-level Correlated and Hierarchical Effects (CHE) model with Robust Variance Estimation (RVE) was used.This approach estimates random effects at both the study and within-study levels and provides small-sample-adjusted standard errors and confidence intervals (Harrer et al., 2021;Pustejovsky & Tipton, 2022).”
From Gamification on Mathematics Engagement and Motivation in Secondary School and Higher Education: A Systematic Review and Meta-Analysis (2026) · cites this paper as Pustejovsky & Tipton, 2022
In the citing paper
“This working model takes account of both within-study and between-study heterogeneity in true effect size, as in the hierarchical working model, while also handling correlation between the effect sizes within the study while assuming the constant correlation, as in the correlated working model.”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Pustejovsky & Tipton, 2022
In the citing paper
“Importantly, in RVE, the dependence structure does not have to be completely known or accurately specified for the estimates to be unbiased.”
From Early childhood executive function predicts concurrent and later social and behavioral outcomes: A review and meta-analysis. (2024) · cites this paper as Pustejovsky & Tipton, 2022
In the citing paper
“the standardised mean reduction was small but statistically significant SMD = 0.106; (95% CI: 0.039 to 0.173; p < 0.001)”
From School‐Based Interventions for Reducing Disciplinary School Exclusion. An Updated Systematic Review (2025) · cites this paper as Pustejovsky and Tipton 2022
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · cited 1×
“After a working model is selected, it can be implemented in R using a combination of the clubSandwich (Pustejovsky, 2020) and metafor packages (Viechtbauer, 2010).”
MethodStatistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 3×
“Furthermore, if we knew the true dependence structure, we could exactly calculate standard errors, test statistics, and confidence intervals for the WLS estimator using standard formulas (Hedges et al., 2010).”
MethodInfluentialSelf-citationMethods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685 · cited 1×
“Furthermore, for univariate models with only a single variance component, REML estimation has been generally recommended over moment estimation methods (Veroniki et al., 2016).”
Small sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 3×
“Specifically, using a working model that more accurately captures the dependence structure will lead to meta-analysis and meta-regression coefficient estimates that are more precise and accurate (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588
Self-citationStochastically dependent effect sizes.
Gleser, L. J., & Olkin, I. (1994) · Cited by 485
Brief alcohol interventions for adolescents and young adults: A systematic review and meta-analysis
Tanner-Smith, E. E., & Lipsey, M. W. (2015) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2014.09.001 · Cited by 468
Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R
Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-016-0026-5 · Cited by 390 · cited 1×
“Existing guidance on use of RVE encourages analysts to select a hierarchical effects or correlated effects model based upon the structure that is most common in their analyses, noting that for the purposes of weighting and inference about the metaregression coefficients, using the exactly correct dependence structure…”
MethodSelf-citationTipton, E., & Pustejovsky, J. E. (2015) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998615606099 · Cited by 362 · cited 3×
“Specifically, using a working model that more accurately captures the dependence structure will lead to meta-analysis and meta-regression coefficient estimates that are more precise and accurate (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citation
Cited by
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“To estimate mean effects, we conducted Correlated and Hierarchical Effects (CHE) models (Pustejovsky & Tipton, 2022) with Robust Variance Estimation (RVE).”
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Cuijpers, P., Miguel, C., Harrer, M., Plessen, C. Y., Ciharova, M., Ebert, D., & Karyotaki, E. (2023) · World Psychiatry · doi:10.1002/wps.21069 · Cited by 354
Nakagawa, S., Yang, Y., Macartney, E. L., Spake, R., & Lagisz, M. (2023) · Environmental Evidence · doi:10.1186/s13750-023-00301-6 · Cited by 210
“Meta-analysis using RVE is relatively new, and this method has been applied to multilevel metaanalytic models only recently [58].”
BackgroundMethodPriyadarshana, T. S., Martin, E. A., Sirami, C., Woodcock, B. A., Goodale, E., Martínez‐Núñez, C., Lee, M., Pagani‐Núñez, E., Raderschall, C. A., Brotons, L., et al. (2024) · Ecology Letters · doi:10.1111/ele.14412 · Cited by 166
Metadata
- Authors
- Pustejovsky, J. E., & Tipton, E.
- Affiliations
- University of Wisconsin System (US)
- Northwestern University (US)
- Corresponding author
- Pustejovsky, James E
- Venue
- Prevention Science
- Published
- 2021-05-07
- Volume
- 23
- Issue
- 3
- Pages
- 425-438
- ISSN
- 1389-4986 (ISSN-L)
- 1573-6695
- OpenAlex
- W3087936928
- Cited by
- 737
- References
- 28
- Citations by year
- 2026: 207 · 2025: 157 · 2024: 172 · 2023: 115 · 2022: 65 · 2021: 19
- FWCI
- 54.07
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Prevention Science (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- MetArXiv (OSF Preprints) (submittedVersion)
- MetArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- National Science Foundation (1937633)
- Institute of Education Sciences (R305B170019)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- robust variance estimation, meta-regression, working models, dependence structure, multilevel meta-analysis, multivariate meta-analysis, metafor, brief alcohol intervention, adolescents, young adults, simulation study
- MeSH terms
- Adolescent, Computer Simulation, Data Collection, Humans, Multivariate Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models. https://app.cochat.ai/papers/10.1007/s11121-021-01246-3
- The paper
- Pustejovsky, J. E., & Tipton, E. (2022). Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models. Prevention Science. https://doi.org/10.1007/s11121-021-01246-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.