Small sample adjustments for robust variance estimation with meta-regression.
The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index.
Abstract
Although primary studies often report multiple outcomes, the covariances between these outcomes are rarely reported. This leads to difficulties when combining studies in a meta-analysis. This problem was recently addressed with the introduction of robust variance estimation. This new method enables the estimation of meta-regression models with dependent effect sizes, even when the dependence structure is unknown. Although robust variance estimation has been shown to perform well when the number of studies in the meta-analysis is large, previous simulation studies suggest that the associated tests often have Type I error rates that are much larger than nominal. In this article, I introduce 6 estimators with better small sample properties and study the effectiveness of these estimators via 2 simulation studies. The results of these simulations suggest that the best estimator involves correcting both the residuals and degrees of freedom used in the robust variance estimator. These studies also suggest that the degrees of freedom depend on not only the number of studies but also the type of covariates in the meta-regression. The fact that the degrees of freedom can be small, even when the number of studies is large, suggests that these small-sample corrections should be used more generally. I conclude with an example comparing the results of a meta-regression with robust variance estimation with the results from the corrected estimator.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As suggested in Tipton (2015), we investigated this assumption with a sensitivity analysis using a range of values of ρ.”
From Montessori education's impact on academic and nonacademic outcomes: A systematic review (2023) · cites this paper as Tipton, 2015
In the citing paper
“For studies that reported multiple effect sizes for a single outcome, we used a cluster‐robust method for synthesizing effect sizes as described in Tanner‐Smith (2014), Tanner‐Smith (2016), and Tipton (2015) using the robumeta package (Fisher, 2017) in R.”
From Montessori education's impact on academic and nonacademic outcomes: A systematic review (2023) · cites this paper as Tipton, 2015
In the citing paper
“We used Tipton (2015)'s small‐sample correction and assumed the within‐study effect size correlation (ρ) to be 0.80.”
From Montessori education's impact on academic and nonacademic outcomes: A systematic review (2023) · cites this paper as Tipton, 2015
In the citing paper
“We used a cluster–robust variance estimation procedure that accounts for dependencies in the data and makes accurate point estimates; however, that procedure is less powerful than other synthesis methods and, therefore, will yield variance estimates that are less precise (Tanner‐Smith, 2014; Tanner‐Smith, 2016; Tipton, 2015).”
From Montessori education's impact on academic and nonacademic outcomes: A systematic review (2023) · cites this paper as Tipton, 2015
In the citing paper
“As a secondary robustness check for effect-size dependence, robust variance estimation (RVE) with small-sample correction was also applied (Tipton, 2015).”
From Mindfulness- and acceptance-based interventions for music performance anxiety: a three-level meta-analysis of therapeutic effects, mechanisms, and boundary conditions (2026) · cites this paper as Tipton, 2015
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology · doi:10.2307/2075823 · Cited by 18,724
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
(1967) · Project Euclid (Cornell University) · Cited by 4,720
An Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086
Self-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899
Self-citation
Cited by
Eagly, A. H., Nater, C., Miller, D. I., Kaufmann, M., & Sczesny, S. (2020) · American Psychologist · doi:10.1037/amp0000494 · Cited by 1,136
Schoenfeld, B. J., Grgic, J., Ogborn, D., & Krieger, J. W. (2017) · Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000002200 · Cited by 851
Childhood maltreatment and characteristics of adult depression: Meta-analysis
Nelson, J., Klumparendt, A., Doebler, P., & Ehring, T. (2017) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.180752 · Cited by 822
“Note that the difference between these means is not identical to the pooled raw mean difference of 4.39, since the quantities were computed in three separate RVE meta-analyses.”
InfluentialShields, G. S., Sazma, M. A., & Yonelinas, A. P. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.038 · Cited by 818
“4It should be noted, however, that because df are less than four in the prior analysis, the risk of making a Type I error increases approximately twofold (Tipton, 2014).”
InfluentialMeta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
“Toaccount for the hierarchical structure of our data set, robust variance estimation (Hedges, Tipton, & Johnson, 2010) with hierarchical weights and small sample corrections (Tipton, 2015; Tip-ton & Pustejovsky, 2015) were used throughout.”
MethodSelf-citation
Metadata
- Authors
- Tipton, E.
- Affiliations
- Columbia University (US)
- Corresponding author
- Tipton, Elizabeth
- ORCID
- Tipton, Elizabeth
- Venue
- Psychological Methods
- Published
- 2015
- Volume
- 20
- Issue
- 3
- Pages
- 375-393
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2317922289
- Cited by
- 810
- References
- 31
- Citations by year
- 2026: 152 · 2025: 77 · 2024: 91 · 2023: 83 · 2022: 89 · 2021: 87 · 2020: 69 · 2019: 55 · 2018: 38 · 2017: 34 · 2016: 18 · 2015: 14
- FWCI
- 22.11
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- robust variance estimation, meta-regression, small sample correction, type I error rate, dependent effect sizes, multiple outcomes, residual correction, meta-analysis
- MeSH terms
- Data Interpretation, Statistical, Humans, Regression Analysis, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Small sample adjustments for robust variance estimation with meta-regression. https://app.cochat.ai/papers/10.1037/met0000011
- The paper
- Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods. https://doi.org/10.1037/met0000011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.