Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression
The DOI matches its Crossref record; 29 of the 31 references it cites were found in a registry or an index.
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since there is uncertainty around the correlation value used for imputation, robust inference methods with an adjustment for small samples was used, so that the interpretation of fi xed effects were unbiased [51].”
From The effect of caffeine on subsequent sleep: A systematic review and meta-analysis (2023) · cites this paper as [51]
In the citing paper
“Given the limited number of studies included, we employed small sample correction to adjust for inflated Type I errors in test statistics based on RVE using t tests (Tipton, 2015;Tipton & Pustejovsky, 2015).”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Tipton & Pustejovsky, 2015
In the citing paper
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky (2015)
In the citing paper
“Like the bias of the CRVE estimator itself, the accuracy of the asymptotic approximations depends on design features such as the degree of imbalance across clusters, skewness or leverage of the covariates, and the similarity of cluster sizes (Tipton and Pustejovsky 2015; McCaffrey, Bell, and Botts 2001; MacKinnon and Webb 2016; Carter, Schnepel, and Steigerwald 2013).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
In the citing paper
“However, simulation studies reported here and elsewhere (Tipton 2015; Tipton and Pustejovsky 2015) have demonstrated that the approach is actually robust to a high degree of misspecification in the working model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“The multivariate meta-analysis model requires good estimates of the correlation between dependent effect sizes (Gleser & Olkin, 2009), which typically are not available to the meta-analyst.”
MethodAspects of multivariate statistical theory
Muirhead, R. J. 1. (1982) · Cited by 3,032
Aspects of Multivariate Statistical Theory
Muirhead, R. J. (1982) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316559 · Cited by 2,260 · cited 2×
“For a random q q matrix D such that ηD follows a Wishart distribution with η degrees of freedom and scale matrix Iq, the covariances among the entries of D satisfy: ηCovðdst; duvÞ ¼ Iðs ¼ uÞIðt ¼ vÞ þ Iðs ¼ vÞIðt ¼ uÞ; ð11Þ for s,t,u,v¼ 1, . . . , q (Muirhead, 1982, section 3.2.2).”
BackgroundInfluentialRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 20×
“However, evidence from several simulation studies indicates that, even with these corrections, the test has inflated Type I error when there are fewer than 40 independent studies (Hedges et al., 2010; Tipton, 2013, 2015; Williams, 2012). Tipton (2015) proposed alternative methods for testing single meta-regression coe…”
BackgroundMethodSelf-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899 · cited 1×
“…(e.g., Wilson, Lipsey, Tanner-Smith, Huang, & Steinka-Fry, 2011), psychology (e.g., Samson, Tipton and Pustejovsky 605 Ojanen, & Hollo, 2012; Uttal et al., 2013), and intervention science (e.g., De Vibe, Bjørndal, Tipton, Hammerstrøm, & Kowalski, 2012; Tanner-Smith, Wilson, & Lipsey, 2013).”
Self-citationSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 28×
“Another approach, introduced by Bell and McCaffrey (2002) and followed by Tipton (2015), is to instead use the same working model that was used to develop the weighting matrices Wj and adjustment matrices Aj.”
BackgroundMethodResultSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588 · cited 4×
“The per-group sample size per study (nj) ranged from 32 to 130 and included the same values as in the second simulation from Tipton (2015). We chose to use a variable number of effect sizes per study and variable sample sizes in order to emulate conditions likely to be observed in practice as well as to induce a great…”
MethodResultSelf-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.05.006 · Cited by 331 · cited 2×
“As with the t-test developed by Tipton (2015) for single-parameter hypothesis tests, we seek a test that performs well in small samples, thus enabling broad application in meta-analysis.”
Method
Research works printed but not linked: 6
- 10.3102/10769986019002091 · doi:10.3102/10769986019002091 · Paper page
- Bell R. M., McCaffrey D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.
- Cooper H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.
- 10.4073/csr.2012.3 · DOI · Paper page
- Fisher Z. Tipton E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Retrieved from arXiv preprint arXiv:1503.02220 · arXiv
- McCaffrey D. F. Bell R. M. Botts C. H. (2001 August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.
Cited by
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
Childhood maltreatment and characteristics of adult depression: Meta-analysis
Nelson, J., Klumparendt, A., Doebler, P., & Ehring, T. (2017) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.180752 · Cited by 822
“We used an extension of the RVE method by Tipton & Pustejovsky 32 to compare the effects of different types of maltreatment with the grand mean effect (averaged over all maltreatment types) for research questions 1–3.”
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
Self-citationMeta-analysis with robust variance estimation: Expanding the range of working models
Pustejovsky, J. E., & Tipton, E. (2022) · Prevention Science · doi:10.1007/s11121-021-01246-3 · Cited by 737
“Specifically, using a working model that more accurately captures the dependence structure will lead to meta-analysis and meta-regression coefficient estimates that are more precise and accurate (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citationCipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“We then applied a sandwich estimator using the “clubSandwich” package (Pustejovsky, 2022) to employ RVE for obtaining robust confidence intervals and p - values (Hedges et al., 2010; Tipton, 2015; Tipton & Pustejovsky, 2015).”
Metadata
- Authors
- Tipton, E., & Pustejovsky, J. E.
- Affiliations
- Columbia University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2015-12
- Volume
- 40
- Issue
- 6
- Pages
- 604-634
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2276186231
- Cited by
- 362
- References
- 31
- Citations by year
- 2026: 66 · 2025: 49 · 2024: 46 · 2023: 47 · 2022: 49 · 2021: 30 · 2020: 31 · 2019: 21 · 2018: 14 · 2017: 6 · 2016: 3
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- robust variance estimation, meta-regression, type I error inflation, F-statistic, moderators, dependent effect sizes, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression. https://app.cochat.ai/papers/10.3102/1076998615606099
- The paper
- Tipton, E., & Pustejovsky, J. E. (2015). Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998615606099
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.