Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression
The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index.
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since there is uncertainty around the correlation value used for imputation, robust inference methods with an adjustment for small samples was used, so that the interpretation of fi xed effects were unbiased [51].”
From The effect of caffeine on subsequent sleep: A systematic review and meta-analysis (2023) · cites this paper as [51]
In the citing paper
“Given the limited number of studies included, we employed small sample correction to adjust for inflated Type I errors in test statistics based on RVE using t tests (Tipton, 2015;Tipton & Pustejovsky, 2015).”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Tipton & Pustejovsky, 2015
In the citing paper
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky (2015)
In the citing paper
“Like the bias of the CRVE estimator itself, the accuracy of the asymptotic approximations depends on design features such as the degree of imbalance across clusters, skewness or leverage of the covariates, and the similarity of cluster sizes (Tipton and Pustejovsky 2015; McCaffrey, Bell, and Botts 2001; MacKinnon and Webb 2016; Carter, Schnepel, and Steigerwald 2013).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
In the citing paper
“However, simulation studies reported here and elsewhere (Tipton 2015; Tipton and Pustejovsky 2015) have demonstrated that the approach is actually robust to a high degree of misspecification in the working model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“The multivariate meta-analysis model requires good estimates of the correlation between dependent effect sizes (Gleser & Olkin, 2009), which typically are not available to the meta-analyst.”
MethodAspects of multivariate statistical theory
Muirhead, R. J. 1. (1982) · Cited by 3,032
Aspects of Multivariate Statistical Theory
Muirhead, R. J. (1982) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316559 · Cited by 2,260 · cited 2×
“For a random q q matrix D such that ηD follows a Wishart distribution with η degrees of freedom and scale matrix Iq, the covariances among the entries of D satisfy: ηCovðdst; duvÞ ¼ Iðs ¼ uÞIðt ¼ vÞ þ Iðs ¼ vÞIðt ¼ uÞ; ð11Þ for s,t,u,v¼ 1, . . . , q (Muirhead, 1982, section 3.2.2).”
BackgroundInfluentialRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 20×
“However, evidence from several simulation studies indicates that, even with these corrections, the test has inflated Type I error when there are fewer than 40 independent studies (Hedges et al., 2010; Tipton, 2013, 2015; Williams, 2012). Tipton (2015) proposed alternative methods for testing single meta-regression coe…”
BackgroundMethodSelf-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899 · cited 1×
“…(e.g., Wilson, Lipsey, Tanner-Smith, Huang, & Steinka-Fry, 2011), psychology (e.g., Samson, Tipton and Pustejovsky 605 Ojanen, & Hollo, 2012; Uttal et al., 2013), and intervention science (e.g., De Vibe, Bjørndal, Tipton, Hammerstrøm, & Kowalski, 2012; Tanner-Smith, Wilson, & Lipsey, 2013).”
Self-citationSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 28×
“Another approach, introduced by Bell and McCaffrey (2002) and followed by Tipton (2015), is to instead use the same working model that was used to develop the weighting matrices Wj and adjustment matrices Aj.”
BackgroundMethodResultSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588 · cited 4×
“The per-group sample size per study (nj) ranged from 32 to 130 and included the same values as in the second simulation from Tipton (2015). We chose to use a variable number of effect sizes per study and variable sample sizes in order to emulate conditions likely to be observed in practice as well as to induce a great…”
MethodResultSelf-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.05.006 · Cited by 331 · cited 2×
“As with the t-test developed by Tipton (2015) for single-parameter hypothesis tests, we seek a test that performs well in small samples, thus enabling broad application in meta-analysis.”
Method
Research works printed but not linked: 5
- 10.3102/10769986019002091 · doi:10.3102/10769986019002091 · Paper page
- Bell R. M., McCaffrey D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.
- Cooper H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.
- 10.4073/csr.2012.3 · DOI · Paper page
- McCaffrey D. F. Bell R. M. Botts C. H. (2001 August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.
Cited by
Items 201–300 of 367
Napping and memory consolidation in early childhood: A systematic review and meta-analysis
Souabni, M., Souabni, M. J., Salem, A., Hammouda, O., Ammar, A., Saidi, O., Żmijewski, P., Jahrami, H., de Marco, G., Trabelsi, K., et al. (2025) · Sleep Medicine · doi:10.1016/j.sleep.2025.106649 · Cited by 14
Ruhl, J., Perels, F., & Dörrenbächer-Ulrich, L. (2025) · Learning and Individual Differences · doi:10.1016/j.lindif.2025.102760 · Cited by 15
Electroencephalographic Biomarkers of Relaxation: A Systematic Review and Meta-analysis
Sugimoto, K., Kurashiki, H., Xu, Y., Takemi, M., & Amano, K. (2026) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.a.2392 · Cited by 10
Petrowsky, H. M., Boecker, L., Escher, Y. A., Frech, M., Friese, M., Galinsky, A. D., Gunia, B. C., Lee, A. J., Schaerer, M., Schweinsberg, M., et al. (2025) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2025.104448 · Cited by 13
“We fitted all RVE models with the robumeta package (version 2.0) for R with small-sample adjustment of covariance structures and degrees of freedom (Fisher & Tipton, 2015).”
InfluentialDeneault, A., Laquerre, D., Beaulieu, L., Myre, G., Racine, N. M., & Madigan, S. L. (2025) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2025.107651 · Cited by 12
Robust variance estimation in small meta‐analysis with the standardized mean difference
Zejnullahi, R., & Hedges, L. V. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1668 · Cited by 14
“These estimators are also special cases of the cluster robust variance estimators (see, e.g., References 23, 24).”
BackgroundKeenan, C., Miller, S. J., Hanratty, J., Pigott, T., Mackie, P., Cowman, J., Coughlan, C., Hamilton, J., & Fitzpatrick, S. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1103 · Cited by 8
“The correction for small samples (Tipton & Pustejovsky, 2015) will be implemented when necessary.”
Metcalfe, K. B., McFeaters, C. D., & Voyer, D. (2023) · Developmental Neuropsychology · doi:10.1080/87565641.2023.2293712 · Cited by 14
“These robust variance estimates were implemented by means of the robust procedure in metafor , and we applied the CR2 small samples correction derived from the clubSandwich package, as recommended by Tipton and Pustejovsky (2015).”
MethodZhang, F., Wang, F., & Wang, Q. (2022) · Annals of Public and Cooperative Economics · doi:10.1111/apce.12401 · Cited by 8
Liu, Y., Masina, F., Ridderinkhof, K. R., & Pezzetta, R. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105127 · Cited by 12
Smith, T. E., LaBrot, Z. C., Maxime, E., & Lawson, A. (2023) · Perspectives on early childhood psychology and education /Perspectives on early childhood psychology and education · doi:10.58948/2834-8257.1023 · Cited by 8
Smith, T. E., Thompson, A. M., Maynard, B. R., & Kim, A. M. (2024) · Research on Social Work Practice · doi:10.1177/10497315231200351 · Cited by 11
“Speci fi cally, we used cluster-robust variance estimation methods with small sample adjustments (Tipton & Pustejovsky, 2015) to account for the potential inaccuracy of standard errors for individual LRRi estimates, which may occur if autocorrelation or trend is present in the data.”
School-based consultation and coaching for promoting teachers’ generalized outcomes: A meta-analysis
LaBrot, Z. C., Smith, T. E., Maxime, E., & Lawson, A. (2024) · Journal of School Psychology · doi:10.1016/j.jsp.2024.101379 · Cited by 11
Tan, Z., Castagna, C., Krustrup, P., Wong, D. P., Póvoas, S. C. A., Boullosa, D. A., Xu, K., & Čuk, I. (2025) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.70054 · Cited by 10
“We employed cluster-robust variance estimation methods with small-sample adjustments to account for the within-study correlations between effect sizes [47, 48].”
Integrated Primary Care and Mental Health Service Utilization: A Meta-Analysis
Hostutler, C. A. A., Shahidullah, J. D., Rybak, T. M., Stephenson, K. G., Freeman, K. A., LaLonde, L., Danzo, S., Koval, E. T., Hughes, H., & Riley, A. R. (2025) · PEDIATRICS · doi:10.1542/peds.2025-071275 · Cited by 10
Hanratty, J., Miller, S. J., Keenan, C., Cowman, J., Hamilton, J., & Mackie, P. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1109 · Cited by 7
Park, S., Lee, Y. R., Nelson, G., & Tipton, E. (2023) · Learning Disability Quarterly · doi:10.1177/07319487231185133 · Cited by 7
“In practice, RVE can be used with as few as four studies, so long as small sample corrections are included (Tipton, 2015).”
Self-citationPustejovsky, J. E., & Chen, M. (2024) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986241232524 · Cited by 9
“However, subsequent work (Tipton, 2015; Tipton & Pus-tejovsky, 2015) has provided small-sample corrections that reduce the bias of the variance estimators and improve the calibration of hypothesis tests and confidence intervals when the data have a limited number of clusters.”
Self-citationGaffney, H., Jolliffe, D., Eggins, E., Ferreira, J., Skinner, G. C. M., Ariel, B., & Strang, H. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1403 · Cited by 21
“If the number of studies to be meta ‐ analysed is small, we will utilise the ‘ CR2 ’ method for small sample adjustments (Tipton & Pustejovsky, 2015).”
MethodAre Bilinguals More Creative Than Monolinguals? A Meta‐Analysis
Acar, S., Balayar, B., Ozcelik, H. N., & Kharkhurin, A. V. (2024) · The Journal of Creative Behavior · doi:10.1002/jocb.1521 · Cited by 9
Cluster-robust estimators for multivariate mixed-effects meta-regression
Welz, T., Viechtbauer, W., & Pauly, M. (2022) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2022.107631 · Cited by 9
“However, the F -test has been criticized for only performing well in certain scenarios (Tipton, 2015).”
BackgroundTo, M., Prakash, S., Poonnoose, S. I., & Bihari, S. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2018.01.184 · Cited by 7
“variance weighting and robust variance estimation[31] with small sample adjustment[32] was performed to assess the effect of statin dose on transformed AR in a random effects model.”
MethodWang, T., Zhang, Y., Li, F., & Cao, N. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110205 · Cited by 6
Le, V. (2020) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146812012200304 · Cited by 8
“In addition, the analyses used the small-sample corrections recommended by Tipton (2015) and Tipton and Pustejovsky (2015), which makes adjustments to both the residuals and the degrees of freedom of the treatment estimates.”
MethodHanratty, J., Keenan, C., O'Connor, S. R., Miller, S. J., Bradley, D., & Dempster, M. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1219 · Cited by 5
Influencing Climate Change Attitudes in the United States: A Systematic Review and Meta-AnalysisPreprint
Rode, J. B., Dent, A. L., Benedict, C. N., Brosnahan, D., Martinez, R. L., & Ditto, P. H. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/bjen5 · Cited by 11
“…make distributional assumptions about effect sizes (Hedges et al., 2010), provides valid estimates about meta-regression coefficients even in small samples (with a small sample adjustment; Hedges et al., 2010; Tipton, 2015), and includes meta-regression moderator tests (Tipton & Pustejovsky, 2015).”
BackgroundMethodsimhelpers: Helper Functions for Simulation Studies
Joshi, M., & Pustejovsky, J. E. (2020) · doi:10.32614/cran.package.simhelpers · Cited by 6
Self-citationPark, S., Lee, Y. R., Nelson, G., Cook, M. A., & Doabler, C. T. (2025) · Education Sciences · doi:10.3390/educsci15091177 · Cited by 5
What can we learn from 1,000 meta-analyses across 10 different disciplines?
Wu, W., Duan, J., Reed, W. R., & Tipton, E. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10035 · Cited by 10
Self-citationRibeiro, L. A., & Keles, S. (2025) · Journal of Research on Adolescence · doi:10.1111/jora.70088 · Cited by 7
“CHE models are based on robust variance estimation (RVE), which addresses the handling of effect sizes and variance estimates related to both within-and between-study dependencies (Fisher & Tipton, 2015; Tipton & Pustejovsky, 2015; see also Pustejovsky & Tipton, 2021).”
Sex differences in self-reported spatial abilities and affect: a systematic review and meta-analysis
Matthews, V., Ramirez, C., Metcalfe, K. B., Wiseman, M., & Voyer, D. (2023) · Spatial Cognition and Computation · doi:10.1080/13875868.2023.2250537 · Cited by 7
“Additionally, the CR2 small samples correction derived from the clubSandwich package (Pustejovsky, 2020) was applied, as recommended by Tipton and Pustejovsky (2015).”
MethodDalgaard, N. T., Jensen, M. C. F., Bengtsen, E., Krassel, K. F., & Vembye, M. H. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1254 · Cited by 12
“…Multiple time points When the results are measured at multiple time points, we plan to model time differences via appropriate CHE models so that we can reliably estimate and compare confidence intervals and mean differences among time points (Pustejovsky & Tipton, 2021; Tipton & Pustejovsky 2015).”
MethodInfluentialMini-review: effect sizes and meta-analysis for antifouling research
Guerin, A., & Clare, A. (2018) · Biofouling (Print) · doi:10.1080/08927014.2018.1550196 · Cited by 5
“…variance estimation’ methods have been developed, which can be used where there is statistical non-dependence, even when there is insufficient information to use some of the more complex approaches (Fisher and Tipton 2015; Hedges et al. 2010; Tanner-Smith et al. 2016; Tipton and Pustejovsky 2015).”
Dutcher, E. G., Salmon, T. C., & Saral, K. J. (2024) · Experimental Economics · doi:10.1007/s10683-024-09844-y · Cited by 31
“28 In the replication package, we provide Tobit regressions to correct for censoring and specifications and also use sandwich regressions (McCaffrey and Bell 2002; Tipton and Pustejovsky 2015; Pustejovsky and Tipton 2018) to cluster at the group level despite the small number of groups.”
PROTOCOL: Testing frequency and student achievement: A systematic review
Thomsen, M. K., Seerup, J. K., Dietrichson, J., Bondebjerg, A., & Viinholt, B. C. A. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1212 · Cited by 11
“We chose this threshold because Tipton (2015) suggested that RVE standard errors are unreliable when the degrees of freedom are below 4.”
PROTOCOL: Street outreach conflict mediation programs for reducing violence
Maguire, E., Telep, C. W., Abt, T., & Adams, E. B. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1388 · Cited by 5
“…and reliant on the same intervention, the results will be treated as multiple outcomes in the same study and will be reported using robust variance estimation procedures designed for handling dependent effect sizes (Hedges et al., 2010; Pustejovsky & Tipton, 2022; Tipton & Pustejovsky, 2015).”
MethodSchindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2022) · doi:10.31234/osf.io/8y6me · Cited by 5
Oliveira, C. M., Henderson, L. M., & Hayiou‐Thomas, M. E. (2022) · doi:10.31234/osf.io/ndtg6 · Cited by 5
Fishstrom, S., Capin, P., Bhat, B. H., Dahl‐Leonard, K., Payne, S. B., Wang, H.-H., Dille, J. T., & Vaughn, S. R. (2025) · Journal of School Psychology · doi:10.1016/j.jsp.2025.101433 · Cited by 8
Feng, S., Liu, R., Thompson, C., & Colwell, B. (2025) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaf111 · Cited by 4
Guo, D., Feng, L., & Hodges, T. S. (2022) · Learning Disability Quarterly · doi:10.1177/07319487221085277 · Cited by 4
“To resolve the issue of the dependency among effect sizes, we used robust variance estimation with the small sample size correction (Hedges et al., 2010; Tipton & Pustejovsky, 2015).”
MethodRecent developments in inference: practicalities for applied economics
Michler, J. D., & Josephson, A. (2022) · Edward Elgar Publishing eBooks · doi:10.4337/9781800372054.00019 · Cited by 6
“This approach has been refined by Tipton and Pustejovsky (2015) and Pustejovsky and Tipton (2018).”
MethodDietrichson, J., Thomsen, M. K., Seerup, J. K., Strandby, M. W., Viinholt, B. C. A., & Bengtsen, E. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1262 · Cited by 5
“These standard errors are adjusted for small‐sample bias as suggested by Tipton (2015) and Tipton and Pustejovsky (2015).”
MethodResultIndirect and Physiological Approaches to Assessing Deviant Sexual Interests
Nunes, K. L., & Pedneault, C. I. (2020) · Wiley eBooks · doi:10.1002/9781119439325.ch7 · Cited by 4
Winters, K. L., Jasso, J., Pustejovsky, J. E., & Byrd, C. T. (2022) · doi:10.31234/osf.io/bcky8 · Cited by 4
Self-citationInteractive Features of E-Texts’ Effects on Learning: A Systematic Review and Meta-analysisPreprint
Clinton‐Lisell, V. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/zu4wd · Cited by 6
Shin, M., Bryant, D. P., Powell, S. R., Jung, P., Ok, M. W., & Hou, F. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/qjtg6 · Cited by 4
The Effect of LLM Assistance on Diagnostic Accuracy: A Systematic Review and Meta-AnalysisPreprint
Tretow, I., Ergon, E., Schwebel, M., Treffers, T., Welpe, I. M., & Feuerriegel, S. (2025) · medRxiv · doi:10.64898/2025.12.11.25341476 · Cited by 9
Identifying moderators of physical literacy interventions using meta-regression
Ng, J. Y. Y., Fung, Y., Sum, R. K. W., Chan, C. H. S., He, Q., Peng, B., Jiang, S., Carl, J., Barnett, L. M., & Ha, A. S. (2025) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2025.102848 · Cited by 6
Williams, C., Yang, Y., Warton, D. I., & Nakagawa, S. (2025) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.70156 · Cited by 11
“…cluster numbers are small (less than 50 studies), which is likely in meta-analysis in ecology and evolution, the CR0 method is downwardly biased for variance components as well as having high Type I error rates for associated hypothesis tests (Tipton & Pustejovsky, 2015; Viechtbauer et al., 2015 ).”
Big Five personality traits and voting: A systematic review, meta-analysis, and mega-analysis
Stahlmann, A. G., Hopwood, C. J., Orth, U., Smillie, L. D., & Bleidorn, W. (2025) · European Journal of Personality · doi:10.1177/08902070251383955 · Cited by 3
“Con fi dence intervals are corrected using a sandwich estimator for the standard errors and Satterthwaite approximation for the critical values (Satterthwaite, 1946; Tipton & Pustejovsky, 2015). to other samples.”
Digital Tools for Reading Success: Meta-analyses of Digital Interventions
Pasqualotto, A., Cunningham, E. G., Holman, C., Bediou, B., & Bavelier, D. (2025) · Journal of Cognitive Enhancement · doi:10.1007/s41465-025-00336-2 · Cited by 9
“…In addition, it also relies on recent state-of-the-art multilevel and robust meta-analytic methods designed to handle the presence of dependent effects, both correlated and hierarchi - cal dependencies, which are typical of interventions studies (Tipton et al., 2019; Tipton & Pustejovsky, 2015).”
Gómez‐Soria, I., Borja, J. E. A., Aguilar‐Latorre, A., Calatayud, E., Ferreira-Sánchez, M. D. R., & Cuenca-Zaldivar, J. N. (2026) · Clinical Gerontologist · doi:10.1080/07317115.2025.2608767 · Cited by 3
“…principle when analyzing repeated measures in the studies (post-treatment and follow-up), standard errors were adjusted using the robust variance estimator (RVE) proposed by Pustejovsky and Tipton (2018) (Tipton & J, 2015) and applying the Satterthwaite adjustment to the degrees of freedom.”
Knight, R., Valappil, A. C., Bruton, A. M., & Greene, A. (2026) · Sports Biomechanics · doi:10.1080/14763141.2025.2603396 · Cited by 5
“Further, RVE was used in conjunction with a small sample adjustment that mitigates type 1 error inflation due to low or moderate study numbers (Tipton, 2015; Tipton & Pustejovsky, 2015).”
Chen, P., Tung, H., Wu, Y., & Sie, J. (2026) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.70134 · Cited by 4
“Omnibus tests used cluster-robust Wald tests with Hotelling – Zhang adjustments (Tipton & Pustejovsky, 2015) to control Type I error.”
Protocol: Interventions aimed at preventing out-of-home placement of children: A systematic review
Dalgaard, N. T., Bondebjerg, A., Bengtsen, E., Dietrichson, J., & Bach‐Mortensen, A. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1395 · Cited by 3
“The confidence intervals will be based on standard errors, which are adjusted for small ‐ sample bias as suggested by Tipton (2015) and Tipton and Pustejovsky (2015).”
MethodBourke, M., Judge-Mears, B., Guo, B., & Fortnum, K. (2025) · Health Psychology Review · doi:10.1080/17437199.2025.2590489 · Cited by 4
“Because subgroup analyses with very small degrees of freedom are unreliable (Tipton & Pustejovsky, 2015), subgroups were only included in the moderation analysis if the estimated effect was based on at least three degrees of freedom.”
Balloo, K., & Winstone, N. (2021) · Frontline Learning Research · doi:10.14786/flr.v9i2.675 · Cited by 5
“Therefore, there may be hierarchical dependence between effect sizes (Tipton & Pustejovsky, 2015), because although different individuals contribute to each effect size, there may be a shared influence on patterns of continuation within a particular programme (e.g. because the same staff may teach…”
BackgroundMethodYu, Z., An, Q., Hawkins, J., & Zhang, A. (2023) · Research on Social Work Practice · doi:10.1177/10497315231171374 · Cited by 3
“RVE can produce more robust statistical inferences when multiple samples are derived from the same manuscript (Hedges et al., 2010; Tipton & Pustejovsky, 2015)”
BackgroundEffects of Instability Resistance Training on Physical and Cognitive Function in Adults
Cavalcante, B. R., Falck, R. S., da Silva, D. T., Campos, I. R. A., Silva, M. S., Eckardt, N., Batista, G. A., Ferreira, F. M., Pirauá, A. L. T., de Souza, M. F., et al. (2025) · International Journal of Sports Medicine · doi:10.1055/a-2674-4030 · Cited by 5
Does parochial cooperation exist in childhood and adolescence? A meta-analysisPreprint
Lazić, A., Purić, D., & Krstić, K. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uexmr · Cited by 3
Bordes, C., Ammeter, M., Lu, C., Beukeboom, R., Goll, Y., Bourdiol, J., & Ilany, A. (2025) · Journal of Animal Ecology · doi:10.1111/1365-2656.70122 · Cited by 2
“We inferred robust confidence intervals for meta-regression es - timates using the sandwich estimator including an adjustment for small sample (Tipton & Pustejovsky, 2015 ).”
A Framework for Learning From Erroneous Examples and Meta-Analysis of Empirical Research
Alemdağ, E., Eichelmann, A., & Narciss, S. (2025) · Review of Educational Research · doi:10.3102/00346543251390901 · Cited by 3
What Works for Teaching Word Reading to Adolescents? A Meta-Analytic Investigation
Yi, J. J., Aitken, A., Sharp, E., Nusrat, A., & Sandbank, M. P. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10124-9 · Cited by 2
“…separate sub - group summary effects by outcome type, which we then statistically compared using Approximate Hotelling’s T2 tests with CR2-type robust variance estimation, which are default recommendations for current working models of RVE (Pustejovsky & Tipton, 2022; Tipton & Pustejovsky, 2015 ).”
High-Quality Special Education Professional Development Improves Teacher and Student Outcomes
Didion, L. A., Filderman, M. J., Hart, E. J., Payne, S. B., Vitale, J., Olmstead, C. L., & Wexler, J. (2026) · The Journal of Special Education · doi:10.1177/00224669261423924 · Cited by 6
“To address the statistical dependencies arising from studies contributing multiple effect sizes due to reporting several teacher outcomes, we used random effects robust variance estimation (Hedges et al., 2010; Tipton & Pustejovsky, 2015).”
Social and Emotional Learning Programs and Students’ Prosocial Behavior: A Meta-Analysis
Hung, C., Brass, N. R., Brockmeier, L., Bergin, C. C., Imler, M. H., & Luper, S. (2026) · Review of Educational Research · doi:10.3102/00346543261438462 · Cited by 8
Robust confidence intervals for meta-regression with interaction effects
Thurow, M., Welz, T., Knop, E. S., Friede, T., & Pauly, M. (2024) · Computational Statistics · doi:10.1007/s00180-024-01530-0 · Cited by 4
“A typical choice for the degrees of freedom is df = k − m − 1 .”
Froerer, A. S., Bolghan‐Abadi, M., Chen, M., & Zhang, A. (2024) · Journal of Marital and Family Therapy · doi:10.1111/jmft.12759 · Cited by 3
Peer Support and Community Engagement Interventions for Chinese Shidu Parents: A Meta-Analysis
Wang, H., Wang, F. Y., & Ji, X. (2025) · Research on Social Work Practice · doi:10.1177/10497315251329520 · Cited by 2
“Robust variance estimation (RVE) was performed in the meta-regression process, as recommended by Hedges et al. (2010).”
InfluentialCosta, C., Ronconi, L., Ferracuti, S., Schincariol, A., & Scarpazza, C. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1568244 · Cited by 2
“Omnibus tests were based on an F-distribution, with degrees of freedom approximated using an approximate Hotelling ’ s T-squared distribution (77 – 79).”
Miller, S. J., Keenan, C., Hanratty, J., Hamilton, J., Coughlan, C., Mackie, P., Fitzpatrick, S., & Maddock, A. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1118 · Cited by 1
“The correction for small samples (Tipton & Pustejovsky, 2015) will be implemented MILLER ET AL .”
Equivalences between ad hoc strategies and meta-analytic models for dependent effect sizesPreprint
Pustejovsky, J. E., & Chen, M. (2023) · doi:10.31222/osf.io/pw54r · Cited by 2
Self-citationBrunner, M., Preckel, F., Goetz, T., Lüdtke, O., & Keller, L. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6pmhr · Cited by 3
Bruntsch, M., Cooper, S. E., Abend, R., Boor, M., Chalkia, A., Ehlers, M. R., Czeszumski, A., Johnson, D. C., Klingelhöfer-Jens, M., Morriss, J., et al. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/unx7w · Cited by 5
Song, J., Torres‐Carmona, E., Abdolizadeh, A., Kambari, Y., Amaev, A., Ueno, F., Koizumi, T., Remington, G. J., Luca, V. D., Graff‐Guerrero, A., et al. (2025) · Brain Topography · doi:10.1007/s10548-025-01140-z · Cited by 1
“Under this scenario, robust variance estimate (RVE) is generally more suitable to aggregate meta-regres - sion coefficients to minimize Type I error (Hedges et al. 2010 ; Tipton and Pustejovsky 2015 ; Pustejovsky and Tip - ton 2022).”
Wu, H., & Shen, J. (2025) · Journal of School Leadership · doi:10.1177/10526846251374239 · Cited by 3
“Model-based standard errors, t-tests, and con fi dence intervals were reported when degrees of freedom of CHE models were less than 4 (Tipton, 2015; Tipton & Pustejovsky, 2015).”
Lv, Z., Pan, Z., Huang, Y., Yang, H., & Li, X. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1673712 · Cited by 1
Biogeographic Ancestry and Socioeconomic Status: A Meta-Analysis of Epidemiological Findings
Fuerst, J. G. R., Connor, G., Hu, M., & Menie, M. W. O. (2025) · Evolutionary Psychological Science · doi:10.1007/s40806-025-00452-4 · Cited by 2
“In the “middle term” of the Huber-White sandwich estimator the covariance sub - matrix within the same-sample diagonal blocks is proxied by the outer product of the demeaned observations; see Tipton & Pustejovsky ( 2015 , p. 608).”
Bourke, M., Harrison, Z., Staton, S., Rossa, K. R., & Smith, S. S. (2026) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2026.102232 · Cited by 5
Smith, T. E., LaBrot, Z. C., Kim, A. M., Lawson, A., & Maxime, E. (2026) · Journal of Educational and Psychological Consultation · doi:10.1080/10474412.2026.2615941 · Cited by 1
Jiao, M., Yan, H., Zhang, B., Zhao, X., Li, J., & Harun, M. T. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1772836 · Cited by 1
“To further address statistical dependence, we used cluster-robust variance estimation (CRVE) based on the variance–covariance matrix and applied a small-sample correction to obtain unbiased standard errors under correlated effects (65).”
Du, J.-M., Wu, J.-C., Yan, Z.-J., & Yu, L.-Q. (2026) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2026.106164 · Cited by 1
Zhu, C., Lu, Y., Tao, M., Yin, T., Li, J., Thompson, S. W., & Gu, N. (2026) · Frontiers in Physiology · doi:10.3389/fphys.2026.1750821 · Cited by 1
“As a sensitivity analysis, we used cluster-robust variance estimation methods (Hedges et al., 2010) with small-sample adjustments (Tipton and Pustejovsky, 2015) to adjust the within-study standard errors for correlations between effect sizes.”
Tao, M., Yin, M., Zhong, Y., Xu, K., Zhu, C., Deng, J., Song, Y., Lyu, M., Chen, Z., Nassis, G. P., et al. (2026) · International Journal of Sports Science & Coaching · doi:10.1177/17479541261428779 · Cited by 1
“The contour-enhanced funnel plot, 60 in conjunction with Egger ’ s asymmetry test 61,62 was employed to assess publication bias (tests were only conducted when k indicates the number of included studies ≥ 10 63 ), and the p > 0.05 was considered without risk of publication bias.”
InfluentialMeta-regression with categorical moderators and dependent effect sizes: A simulation study
Fernández-Castilla, B., López-López, J. A., & Rubio‐Aparicio, M. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10092 · Cited by 1
Zuber, J., & Gnambs, T. (2026) · The Arts in Psychotherapy · doi:10.1016/j.aip.2026.102482 · Cited by 1
Activation parenting in mothers and fathers: A systematic review and meta‐analysis
Laquerre, D., Gagné, A., Beaulieu, L., Gariépy, M., Feldman, J. S., Bureau, J., & Deneault, A. (2026) · Infant Mental Health Journal · doi:10.1002/imhj.70101 · Cited by 1
“We estimated robust confidence intervals and p-values with the robust variance estimation (RVE) with a small-sample adjustment (Tipton & Pustejovsky, 2015).”
Zeng, J., Li, H., & Zhang, X. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1878053 · Cited by 2
“Meta-analyses were conducted in R version 4.3.1 using the meta-for and clubSandwich packages (28–30).”
InfluentialHuang, J., Pervez, A., Qu, X., & Zhang, T. (2026) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2026.105198 · Cited by 1
Doussard, C. (2023) · Urban forestry & urban greening · doi:10.1016/j.ufug.2023.128004 · Cited by 3
https://stratfordjournal.org/journals/index.php/journal-of-procurement-supply/article/view/2434
Gachigo, J. (2024) · Journal of Procurement & Supply Chain · doi:10.53819/81018102t2457 · Cited by 1
Digital Transformation in Public Procurement: Blockchain Solutions and Legal Frameworks
Gachigo, J. (2024) · Journal of Procurement & Supply Chain · doi:10.53819/81018102t2456 · Cited by 1
Student Motivation and Academic Dishonesty: A Meta-Analytic Investigation
Krou, M. R. (2019) · Cited by 2
Spry, E., Aarsman, S. R., Youssef, G. J., Patton, G. C., Macdonald, J. A., Sanson, A. V., Thomson, K., Hutchinson, D. M., Letcher, P., & Olsson, C. A. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/y9s7w · Cited by 2
Supplemental Material for Effects of Action Video Game Play on Cognitive Skills: A Meta-Analysis
Bediou, B., Rodgers, M. A., Tipton, E., Mayer, R. E., Green, C. S., & Bavelier, D. (2023) · Technology Mind and Behavior · doi:10.1037/tmb0000102.supp · Cited by 4
Self-citationConnor, G., Fuerst, J. G. R., & Hu, M. (2024) · Qeios · doi:10.32388/5oulh0.2 · Cited by 1
González-González, A. M., Pellitero, M. T., Tejerina, N. L., Soto-Goñi, X., Cuenca‐Zaldívar, J. N., & Romero, E. A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-6388848/v1 · Cited by 1
Wetland proximity: a meta-analysis of hedonic property values
Bastola, S., Penn, J., & Yoo, J. (2025) · Environmental Economics and Policy Studies · doi:10.1007/s10018-025-00441-8
Gunnerud, H. L. (2021) · University of Stavanger eBooks · doi:10.31265/usps.72
Gnambs, T., & Schroeders, U. (2022) · doi:10.31234/osf.io/hte6z
Metadata
- Authors
- Tipton, E., & Pustejovsky, J. E.
- Affiliations
- Columbia University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2015-12
- Volume
- 40
- Issue
- 6
- Pages
- 604-634
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2276186231
- Cited by
- 362
- References
- 31
- Citations by year
- 2026: 66 · 2025: 49 · 2024: 46 · 2023: 47 · 2022: 49 · 2021: 30 · 2020: 31 · 2019: 21 · 2018: 14 · 2017: 6 · 2016: 3
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- robust variance estimation, meta-regression, type I error inflation, F-statistic, moderators, dependent effect sizes, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.