Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression
The DOI matches its Crossref record; 29 of the 31 references it cites were found in a registry or an index.
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since there is uncertainty around the correlation value used for imputation, robust inference methods with an adjustment for small samples was used, so that the interpretation of fi xed effects were unbiased [51].”
From The effect of caffeine on subsequent sleep: A systematic review and meta-analysis (2023) · cites this paper as [51]
In the citing paper
“Given the limited number of studies included, we employed small sample correction to adjust for inflated Type I errors in test statistics based on RVE using t tests (Tipton, 2015;Tipton & Pustejovsky, 2015).”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Tipton & Pustejovsky, 2015
In the citing paper
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky (2015)
In the citing paper
“Like the bias of the CRVE estimator itself, the accuracy of the asymptotic approximations depends on design features such as the degree of imbalance across clusters, skewness or leverage of the covariates, and the similarity of cluster sizes (Tipton and Pustejovsky 2015; McCaffrey, Bell, and Botts 2001; MacKinnon and Webb 2016; Carter, Schnepel, and Steigerwald 2013).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
In the citing paper
“However, simulation studies reported here and elsewhere (Tipton 2015; Tipton and Pustejovsky 2015) have demonstrated that the approach is actually robust to a high degree of misspecification in the working model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“The multivariate meta-analysis model requires good estimates of the correlation between dependent effect sizes (Gleser & Olkin, 2009), which typically are not available to the meta-analyst.”
MethodAspects of multivariate statistical theory
Muirhead, R. J. 1. (1982) · Cited by 3,032
Aspects of Multivariate Statistical Theory
Muirhead, R. J. (1982) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316559 · Cited by 2,260 · cited 2×
“For a random q q matrix D such that ηD follows a Wishart distribution with η degrees of freedom and scale matrix Iq, the covariances among the entries of D satisfy: ηCovðdst; duvÞ ¼ Iðs ¼ uÞIðt ¼ vÞ þ Iðs ¼ vÞIðt ¼ uÞ; ð11Þ for s,t,u,v¼ 1, . . . , q (Muirhead, 1982, section 3.2.2).”
BackgroundInfluentialRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 20×
“However, evidence from several simulation studies indicates that, even with these corrections, the test has inflated Type I error when there are fewer than 40 independent studies (Hedges et al., 2010; Tipton, 2013, 2015; Williams, 2012). Tipton (2015) proposed alternative methods for testing single meta-regression coe…”
BackgroundMethodSelf-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899 · cited 1×
“…(e.g., Wilson, Lipsey, Tanner-Smith, Huang, & Steinka-Fry, 2011), psychology (e.g., Samson, Tipton and Pustejovsky 605 Ojanen, & Hollo, 2012; Uttal et al., 2013), and intervention science (e.g., De Vibe, Bjørndal, Tipton, Hammerstrøm, & Kowalski, 2012; Tanner-Smith, Wilson, & Lipsey, 2013).”
Self-citationSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 28×
“Another approach, introduced by Bell and McCaffrey (2002) and followed by Tipton (2015), is to instead use the same working model that was used to develop the weighting matrices Wj and adjustment matrices Aj.”
BackgroundMethodResultSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588 · cited 4×
“The per-group sample size per study (nj) ranged from 32 to 130 and included the same values as in the second simulation from Tipton (2015). We chose to use a variable number of effect sizes per study and variable sample sizes in order to emulate conditions likely to be observed in practice as well as to induce a great…”
MethodResultSelf-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.05.006 · Cited by 331 · cited 2×
“As with the t-test developed by Tipton (2015) for single-parameter hypothesis tests, we seek a test that performs well in small samples, thus enabling broad application in meta-analysis.”
MethodHrong-Tai Fai, A., & Cornelius, P. L. (1996) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659608811740 · Cited by 260
(2012) · Campbell Systematic Reviews · doi:10.4073/csr · Cited by 238
A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses
Ahn, S., Ames, A. J., & Myers, N. D. (2012) · Review of Educational Research · doi:10.3102/0034654312458162 · Cited by 179 · cited 2×
“Additionally, we repeated the analysis using a subset of 32 studies, which Tipton and Pustejovsky 625 is a typical sample size for meta-analyses in education and the social sciences (Ahn et al., 2012; Polanin & Pigott, 2015).”
BackgroundMethodPan, W., & Wall, M. M. (2002) · Statistics in Medicine · doi:10.1002/sim.1142 · Cited by 150 · cited 5×
“Versions of this approach are found in the literature on heteroskedastic ANOVA (Zhang, 2013), MANOVA with unequal variance–covariance structures (Krishnamoorthy & Yu, 2004; Nel & van der Merwe, 1986; Zhang, 2012), and GEE models (Pan & Wall, 2002).”
BackgroundMethodInfluentialA New and Simpler Approximation for ANOVA under Variance Heterogeneity
Alexander, R. A., & Govern, D. M. (1994) · Journal of Educational Statistics · doi:10.2307/1165140 · Cited by 135 · cited 5×
“Versions of this approach are found in the literature on heteroskedastic ANOVA (Zhang, 2013), MANOVA with unequal variance–covariance structures (Krishnamoorthy & Yu, 2004; Nel & van der Merwe, 1986; Zhang, 2012), and GEE models (Pan & Wall, 2002).”
MethodInfluentialThirty Years of Heteroskedasticity-Robust Inference
MacKinnon, J. G. (2013) · Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis · doi:10.1007/978-1-4614-1653-1_17 · Cited by 114 · cited 5×
“Thus, we have not considered iterative procedures such as bootstrapping, although some recent work in econometrics has proposed promising techniques for bootstrapping in settings with cluster-dependent observations (Webb & MacKinnon, 2013). Neither have we considered corrections such as saddlepoint approximations. McC…”
BackgroundMethodInfluentialA solution to the multivariate behrens-fisher problem
Nel, D. G., & Van Der Merwe, C. A. (1986) · Communications in Statistics - Theory and Methods · doi:10.1080/03610928608829342 · Cited by 109
Robust variance estimation in meta‐regression with binary dependent effects
Tipton, E. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1070 · Cited by 102 · cited 12×
“Just as Tipton (2015) concluded, this means that the best indicator of the need for a small-sample correction would appear to be the degrees of freedom of the small-sample correction itself.”
BackgroundMethodResultSelf-citationModified Nel and Van der Merwe test for the multivariate Behrens–Fisher problem
Krishnamoorthy, K., & Yu, J. (2003) · Statistics & Probability Letters · doi:10.1016/j.spl.2003.10.012 · Cited by 95 · cited 1×
“Versions of this approach are found in the literature on heteroskedastic ANOVA (Zhang, 2013), MANOVA with unequal variance–covariance structures (Krishnamoorthy & Yu, 2004; Nel & van der Merwe, 1986; Zhang, 2012), and GEE models (Pan & Wall, 2002).”
MethodThe use of meta-analytic statistical significance testing
Polanin, J. R., & Pigott, T. D. (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1124 · Cited by 78 · cited 2×
“Additionally, we repeated the analysis using a subset of 32 studies, which Tipton and Pustejovsky 625 is a typical sample size for meta-analyses in education and the social sciences (Ahn et al., 2012; Polanin & Pigott, 2015).”
BackgroundMethodA New Test of Linear Hypotheses in OLS Regression Under Heteroscedasticity of Unknown Form
Cai, L., & Hayes, A. F. (2007) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998607302628 · Cited by 53 · cited 4×
“Following Cai and Hayes (2008), we use a normalizing transformation proposed by Hill (1970).”
MethodInfluentialWild Bootstrap Inference for Wildly Different Cluster Sizes
MacKinnon, J. G., & Webb, M. D. (2013) · doi:10.1920/ps.ifs.2024.0794 · Cited by 37
Social Goals and Youth Aggression: Meta‐analysis of Prosocial and Antisocial Goals
Samson, J. E., Ojanen, T., & Hollo, A. (2012) · Social Development · doi:10.1111/j.1467-9507.2012.00658.x · Cited by 32
Hedberg, E. C. (2011) · RePEc: Research Papers in Economics · Cited by 31
McCaffrey, D. F., & Bell, R. M. (2006) · Statistics in Medicine · doi:10.1002/sim.2502 · Cited by 29 · cited 1×
“McCaffrey and Bell (2006) present evidence that, for tests of single regression coefficients, a saddlepoint approximation may provide even more accurate error control than the Satterthwaite approximation studied in Tipton (2015).”
BackgroundAlgorithm 395: Students t -distribution
Hill, G. W. (1970) · Communications of the ACM · doi:10.1145/355598.362775 · Cited by 27 · cited 1×
“Following Cai and Hayes (2008), we use a normalizing transformation proposed by Hill (1970).”
MethodUsing robust standard errors to combine multiple regression estimates with meta-analysis
Williams, R. T. (2012) · Loyola eCommons (Loyola University of Chicago) · Cited by 23
An Approximate Hotelling T2-Test for Heteroscedastic One-Way MANOVA
Zhang, J. (2012) · Open Journal of Statistics · doi:10.4236/ojs.2012.21001 · Cited by 23
Tests of Linear Hypotheses in the ANOVA under Heteroscedasticity
Zhang, J. (2013) · International Journal of Advanced Statistics and Probability · doi:10.14419/ijasp.v1i2.908 · Cited by 9
Research works printed but not linked: 6
- 10.3102/10769986019002091 · doi:10.3102/10769986019002091 · Paper page
- Bell R. M., McCaffrey D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.
- Cooper H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.
- 10.4073/csr.2012.3 · DOI · Paper page
- Fisher Z. Tipton E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Retrieved from arXiv preprint arXiv:1503.02220 · arXiv
- McCaffrey D. F. Bell R. M. Botts C. H. (2001 August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.
Cited by
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
Childhood maltreatment and characteristics of adult depression: Meta-analysis
Nelson, J., Klumparendt, A., Doebler, P., & Ehring, T. (2017) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.180752 · Cited by 822
“We used an extension of the RVE method by Tipton & Pustejovsky 32 to compare the effects of different types of maltreatment with the grand mean effect (averaged over all maltreatment types) for research questions 1–3.”
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
Self-citationMeta-analysis with robust variance estimation: Expanding the range of working models
Pustejovsky, J. E., & Tipton, E. (2022) · Prevention Science · doi:10.1007/s11121-021-01246-3 · Cited by 737
“Specifically, using a working model that more accurately captures the dependence structure will lead to meta-analysis and meta-regression coefficient estimates that are more precise and accurate (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citationCipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“We then applied a sandwich estimator using the “clubSandwich” package (Pustejovsky, 2022) to employ RVE for obtaining robust confidence intervals and p - values (Hedges et al., 2010; Tipton, 2015; Tipton & Pustejovsky, 2015).”
Metadata
- Authors
- Tipton, E., & Pustejovsky, J. E.
- Affiliations
- Columbia University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2015-12
- Volume
- 40
- Issue
- 6
- Pages
- 604-634
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2276186231
- Cited by
- 362
- References
- 31
- Citations by year
- 2026: 66 · 2025: 49 · 2024: 46 · 2023: 47 · 2022: 49 · 2021: 30 · 2020: 31 · 2019: 21 · 2018: 14 · 2017: 6 · 2016: 3
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- robust variance estimation, meta-regression, type I error inflation, F-statistic, moderators, dependent effect sizes, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.