Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression
The DOI matches its Crossref record; 30 of the 31 references it cites were found in a registry or an index.
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since there is uncertainty around the correlation value used for imputation, robust inference methods with an adjustment for small samples was used, so that the interpretation of fi xed effects were unbiased [51].”
From The effect of caffeine on subsequent sleep: A systematic review and meta-analysis (2023) · cites this paper as [51]
In the citing paper
“Given the limited number of studies included, we employed small sample correction to adjust for inflated Type I errors in test statistics based on RVE using t tests (Tipton, 2015;Tipton & Pustejovsky, 2015).”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Tipton & Pustejovsky, 2015
In the citing paper
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky (2015)
In the citing paper
“Like the bias of the CRVE estimator itself, the accuracy of the asymptotic approximations depends on design features such as the degree of imbalance across clusters, skewness or leverage of the covariates, and the similarity of cluster sizes (Tipton and Pustejovsky 2015; McCaffrey, Bell, and Botts 2001; MacKinnon and Webb 2016; Carter, Schnepel, and Steigerwald 2013).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
In the citing paper
“However, simulation studies reported here and elsewhere (Tipton 2015; Tipton and Pustejovsky 2015) have demonstrated that the approach is actually robust to a high degree of misspecification in the working model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“The multivariate meta-analysis model requires good estimates of the correlation between dependent effect sizes (Gleser & Olkin, 2009), which typically are not available to the meta-analyst.”
MethodAspects of multivariate statistical theory
Muirhead, R. J. 1. (1982) · Cited by 3,032
Aspects of Multivariate Statistical Theory
Muirhead, R. J. (1982) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316559 · Cited by 2,260 · cited 2×
“For a random q q matrix D such that ηD follows a Wishart distribution with η degrees of freedom and scale matrix Iq, the covariances among the entries of D satisfy: ηCovðdst; duvÞ ¼ Iðs ¼ uÞIðt ¼ vÞ þ Iðs ¼ vÞIðt ¼ uÞ; ð11Þ for s,t,u,v¼ 1, . . . , q (Muirhead, 1982, section 3.2.2).”
BackgroundInfluentialRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 20×
“However, evidence from several simulation studies indicates that, even with these corrections, the test has inflated Type I error when there are fewer than 40 independent studies (Hedges et al., 2010; Tipton, 2013, 2015; Williams, 2012). Tipton (2015) proposed alternative methods for testing single meta-regression coe…”
BackgroundMethodSelf-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899 · cited 1×
“…(e.g., Wilson, Lipsey, Tanner-Smith, Huang, & Steinka-Fry, 2011), psychology (e.g., Samson, Tipton and Pustejovsky 605 Ojanen, & Hollo, 2012; Uttal et al., 2013), and intervention science (e.g., De Vibe, Bjørndal, Tipton, Hammerstrøm, & Kowalski, 2012; Tanner-Smith, Wilson, & Lipsey, 2013).”
Self-citationSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 28×
“Another approach, introduced by Bell and McCaffrey (2002) and followed by Tipton (2015), is to instead use the same working model that was used to develop the weighting matrices Wj and adjustment matrices Aj.”
BackgroundMethodResultSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588 · cited 4×
“The per-group sample size per study (nj) ranged from 32 to 130 and included the same values as in the second simulation from Tipton (2015). We chose to use a variable number of effect sizes per study and variable sample sizes in order to emulate conditions likely to be observed in practice as well as to induce a great…”
MethodResultSelf-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.05.006 · Cited by 331 · cited 2×
“As with the t-test developed by Tipton (2015) for single-parameter hypothesis tests, we seek a test that performs well in small samples, thus enabling broad application in meta-analysis.”
Method
Research works printed but not linked: 5
- 10.3102/10769986019002091 · doi:10.3102/10769986019002091 · Paper page
- Bell R. M., McCaffrey D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.
- Cooper H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.
- 10.4073/csr.2012.3 · DOI · Paper page
- McCaffrey D. F. Bell R. M. Botts C. H. (2001 August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.
Cited by
Items 101–200 of 367
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 43
McLouth, L. E. S., Ford, C. G., Pustejovsky, J. E., Park, C. L., Sherman, A. C., Trevino, K. M., & Salsman, J. M. (2020) · Psycho-Oncology · doi:10.1002/pon.5562 · Cited by 48
Self-citationComparing the Ability of MMPI-2 and MMPI-2-RF Validity Scales to Detect Feigning: A Meta-Analysis
Aparcero, M., Picard, E. H., Nijdam-Jones, A., & Rosenfeld, B. (2022) · Assessment (Odessa, Fla.) · doi:10.1177/10731911211067535 · Cited by 21
“Of note, when the Satterthwaite degrees of freedom of the RVE coefficients (i.e., small sample corrected degrees of freedom) are lower than four, the p values for the associated t tests underestimate the type I error (Tipton, 2015; Tipton & Pustejovsky, 2015).”
BackgroundLiu, Y.-Z., Groen, M., & Cain, K. (2023) · Educational Research and Reviews · doi:10.1016/j.edurev.2023.100571 · Cited by 55
Clinton‐Lisell, V. (2021) · Journal of Research in Reading · doi:10.1111/1467-9817.12372 · Cited by 55
Joshi, M., Pustejovsky, J. E., & Beretvas, S. N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1554 · Cited by 43
“This study's data generation process closely followed that of Tipton and Pustejovsky.7 Like Tipton and Pustejovsky,7 we simulated standardized mean differences (SMDs), the most common effect size measure in meta-analyses of social science intervention research.”
BackgroundMethodResultInfluentialSelf-citationCommon, E. A., Lane, K. L., Pustejovsky, J. E., Johnson, A. H., & Johl, L. E. (2017) · Remedial and Special Education · doi:10.1177/0741932517693320 · Cited by 45
“Furthermore, we applied smallsample corrections for RVE due to the limited number of included studies (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citationRisk of poisoning in children and adolescents with ADHD: a systematic review and meta-analysis
Ruiz-Goikoetxea, M., Cortese, S., Magallón, S., Aznárez-Sanado, M., Álvarez Zallo, N., Luis, E. O., de Castro-Manglano, P., Soutullo, C., & Arrondo, G. (2018) · Scientific Reports · doi:10.1038/s41598-018-25893-9 · Cited by 37
Effective writing instruction for students in grades 6 to 12: a best evidence meta-analysis
Graham, S., Cao, Y., Kim, Y. G., Lee, J., Tate, T. P., Collins, P., Cho, M., Moon, Y., Chung, H. Q., & Olson, C. B. (2024) · Reading and Writing · doi:10.1007/s11145-024-10539-2 · Cited by 71
“When the number of treatment/control comparisons is small, the test statistics and confidence intervals based on RVE can result in inflated Type I errors (Hedges et al., 2010; Tipton & Pustejovsky, 2015).”
Keenan, C., Miller, S. J., Hanratty, J., Pigott, T., Hamilton, J., Coughlan, C., Mackie, P., Fitzpatrick, S., & Cowman, J. (2021) · Campbell Systematic Reviews · doi:10.1002/cl2.1165 · Cited by 48
Blumenstein, K. G., Brose, A., Kemp, C., Meister, D., Walling, E., DuVall, A. S., & Zhang, A. (2022) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2022.103709 · Cited by 48
The Child Opportunity Index and Children’s Health: A Meta-analysis
Tyris, J., Putnick, D., Keller, S., Parikh, K., & Yeung, E. (2025) · Pediatrics · doi:10.1542/peds.2024-067873 · Cited by 31
Does neuroticism disrupt the psychological benefits of nostalgia? a meta-analytic test
Frankenbach, J., Wildschut, T., Juhl, J., & Sedikides, C. (2021) · European Journal of Personality · doi:10.1002/per.2276 · Cited by 34
“We tested single regression coefficients using t-tests with Satterthwaite-adjusted degrees of freedom (Tipton, 2015) and multiple regression coefficients with the approximate Hotelling-Zhang test (AHZ; Tipton & Pustejovsky, 2015).”
BackgroundMethodYanagisawa, A., & Webb, S. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000577 · Cited by 50
Listening Ears or Reading Eyes: A Meta-Analysis of Reading and Listening Comprehension Comparisons
Clinton-Lisell, V. (2021) · Review of Educational Research · doi:10.3102/00346543211060871 · Cited by 34
“It should be noted that none of the degrees of freedom were fewer than four in the metaregressions, which is reassuring as results with degrees of freedom fewer than four in these types of analyses should not be trusted (Tipton & Pustejovsky, 2015).”
BackgroundMethodThe effect of notetaking method on academic performance: A systematic review and meta-analysis
Voyer, D., Ronis, S. T., & Byers, N. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.102025 · Cited by 54
The acute affective response to physical activity in people with depression: A meta-analysis
Bourke, M., Patten, R. K., Klamert, L., Pogrmilović, B. K., Dash, S. R., & Pascoe, M. C. (2022) · Journal of Affective Disorders · doi:10.1016/j.jad.2022.05.089 · Cited by 41
How effective is feedback for L1, L2, and FL learners’ writing? A meta-analysis
Scherer, S., Graham, S., & Busse, V. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101961 · Cited by 76
Dietrichson, J., Filges, T., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Jensen, U. H. (2020) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1081 · Cited by 33
“To calculate the F statistic, the degrees of freedom, and the (two‐sided) p value in columns 2–4, we used the test described in Tipton and Pustejovsky (2015) and implemented it using an extension to the function Wald_test and the “HTZ” small‐sample correction procedure contained in the R package…”
Meade, R. D., Akerman, A. P., Notley, S. R., McGarr, G. W., McCourt, E. R., Kirby, N. V., Costello, J. T., Cotter, J. D., Crandall, C. G., Zanobetti, A., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57868-6 · Cited by 43
“After con fi rming the models were well speci fi ed, robust con fi dence intervals and signi fi cance tests were obtained using a cluster-robust (sandwich) estimator with small-sample correction 471,472 .”
Taylor, J. A., Kowalski, S. M., Polanin, J. R., Askinas, K., Stuhlsatz, M. A. M., Wilson, C. D., Tipton, E., & Wilson, S. J. (2018) · AERA Open · doi:10.1177/2332858418791991 · Cited by 32
“These intervals are calculated using Est +/- t .025, SE Estη( ) ( ), (5) where both the degrees of freedom (η) and standard error (i.e., SE[Est]) are estimated based on the small-sample F test (Tipton & Pustejovsky, 2015).”
BackgroundMethodInfluentialSelf-citationThe processing advantage of multiword sequences: A meta-analysis
Yi, W., & Zhong, Y. (2023) · Studies in Second Language Acquisition · doi:10.1017/s0272263123000542 · Cited by 35
“Main effects of the moderators in three-level models were examined by a conservative type of Wald test (i.e., HTZ test; Tipton & Pustejovsky, 2015) from which F statistics were obtained.”
MethodKowalski, S. M., Taylor, J. A., Askinas, K. M., Wang, Q., Zhang, Q., Maddix, W. P., & Tipton, E. (2020) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2020.1726538 · Cited by 26
“To account for dependence among effect size estimates, we estimated our meta-regression models using RVE (Hedges et al., 2010a) with small sample corrections (Tipton, 2015; Tipton & Pustejovsky, 2015) and implemented these methods using the R package robumeta (Fisher, Tipton, & Hou, 2017).”
MethodSelf-citationHa, C., McCarthy, M. F., Strambler, M. J., & Cipriano, C. (2025) · Review of Educational Research · doi:10.3102/00346543251367769 · Cited by 64
“The analysis methods produce results that are more accurate than meta-analysis by accounting for correlated effect sizes derived from the same intervention (Cipriano, Strambler, et al., 2023b; Tipton & Pustejovsky, 2015).”
Performance on Reading Comprehension Assessments and College Achievement: A Meta-Analysis
Clinton-Lisell, V., Taylor, T. O., Carlson, S. E., Davison, M., & Seipel, B. (2022) · Journal of College Reading and Learning · doi:10.1080/10790195.2022.2062626 · Cited by 38
“Flow diagram of the systematic review process. variation of sample sizes, small sample size correction was used in analyses (Tipton & Pustejovsky, 2015).”
MethodRode, J. B., Dent, A. L., & Ditto, P. H. (2022) · Environmental Communication · doi:10.1080/17524032.2022.2101501 · Cited by 33
“For moderation by political affiliation, we used the Approximate Hotelling-Zhang (AHZ) test (Tipton & Pustejovsky, 2015).”
MethodThe Effects of Mindfulness Interventions on Older Adults’ Cognition: A Meta-Analysis
Mirabito, G., & Verhaeghen, P. (2022) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbac143 · Cited by 37
Thomas, K. H., Dalili, M. N., Cheng, H., Dawson, S., Donnelly, N., Higgins, J. P. T., & Hickman, M. (2024) · Addiction · doi:10.1111/add.16616 · Cited by 45
Zainal, N. H., Soh, C. P., Doren, N. V., & Benjet, C. L. (2024) · Clinical Psychology Review · doi:10.1016/j.cpr.2024.102518 · Cited by 43
Wahman, C. L., Pustejovsky, J. E., Ostrosky, M. M., & Santos, R. M. (2019) · Topics in Early Childhood Special Education · doi:10.1177/0271121419855692 · Cited by 42
“We used cluster-robust variance estimation methods (CRVE; Hedges, Tipton, & Johnson, 2010) with small-sample adjustments (Tipton & Pustejovsky, 2015) to calculate standard errors and confidence intervals for overall average effect size estimates, with clustering at the study level.”
Self-citationThe Impact of Summer Programs on Student Mathematics Achievement: A Meta-Analysis
Lynch, K., An, L., & Mancenido, Z. (2022) · Review of Educational Research · doi:10.3102/00346543221105543 · Cited by 32
“***p < .01. also provided information on content hours) (Tipton & Pustejovsky, 2015).”
BackgroundMethodResultInfluentialIdentifying positive impression management using the MMPI-2 and the MMPI-2-RF: A meta-analysis
Picard, E. H., Aparcero, M., Nijdam-Jones, A., & Rosenfeld, B. (2022) · Clinical Neuropsychologist · doi:10.1080/13854046.2022.2077237 · Cited by 12
“Recent meta-analytic methods, such as the Robust Variance Estimation (RVE) method (Tipton, 2015; Tipton & Pustejovsky, 2015), adjusts for dependent effect sizes and small samples and permit a more rigorous analysis of the impact of methodological variables on the utility of the scales used to…”
MethodAddressing Dependency in Meta-Analysis: A Companion to Assink and Wibbelink (2016)
Assink, M., & Wibbelink, C. J. M. (2024) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.20.1.p001 · Cited by 35
“Put simply, combining a working model with the RVE technique may address the dependency in effect size data more accurately, which presumably leads to more precise and accurate model coefficients than using RVE alone (Tipton, 2015; Tipton & Pustejovsky, 2015).”
BackgroundMethodInfluentialA Meta-Analysis of Mathematics Interventions: Examining the Impacts of Intervention Characteristics
Rojo, M. M., Gersib, J. A., Powell, S., Shen, Z., King, S. G., Akther, S. S., Arsenault, T. L., Bos, S. E., Lariviere, D. O., & Lin, X. (2024) · Educational Psychology Review · doi:10.1007/s10648-023-09843-0 · Cited by 32
“To account for dependencies within studies, we used robust variance estimation, which incorporates Sattherthwaite degrees of freedom and standard errors for regression coefficients to correct for small samples (Tipton & Pustejovsky, 2015).”
Meta‐analysis reveals impacts of disturbance on reptile and amphibian body condition
Macdonald, K. J., Driscoll, D. A., Macdonald, K. J., Hradsky, B. A., & Doherty, T. S. (2023) · Global Change Biology · doi:10.1111/gcb.16852 · Cited by 29
“Additionally, a robust variance estimator was used with ‘study ID’ as the cluster to account for multiple effect sizes from individual studies (Tipton & Pustejovsky, 2015).”
MethodBratlie, S. S., Brinchmann, E. I., Melby-Lervåg, M., & Torkildsen, J. V. K. (2022) · Review of Educational Research · doi:10.3102/00346543211073186 · Cited by 26
“For details about computations and formulas, see Fisher and Tipton (2015) as well as Tipton and Pustejovsky (2015).”
MethodAdolescent Pregnancy Prevention: Meta-Analysis of Federally Funded Program Evaluations
Juras, R., Tanner‐Smith, E. E., Kelsey, M., Lipsey, M. W., & Layzer, J. I. (2019) · American Journal of Public Health · doi:10.2105/ajph.2018.304925 · Cited by 32
White, R. M. B., Pasco, M. C., Korous, K. M., & Causadías, J. M. (2019) · Journal of Adolescence · doi:10.1016/j.adolescence.2019.12.005 · Cited by 31
Austin, C. R., Wanzek, J., Scammacca, N. K., Vaughn, S., Gesel, S. A., Donegan, R. E., & Engelmann, M. L. (2019) · Exceptional Children · doi:10.1177/0014402918796164 · Cited by 25
“We conducted tests of statistical significance for the difference between coefficients for the categorical moderators using Wald tests as recommended in Tipton and Pustejovsky (2015) and implemented in the clubSandwich package for R (Pustejovsky, 2015).”
MethodPower Approximations for Overall Average Effects in Meta-Analysis With Dependent Effect Sizes
Vembye, M. H., Pustejovsky, J., & Pigott, T. (2022) · Journal of educational and behavioral statistics · doi:10.3102/10769986221127379 · Cited by 22
“We focus on the CR2 estimator because it has been found to outperform other robust variance estimators and is recommended for general use in the context of meta-analysis of dependent effect sizes (Tipton, 2015; Tipton and Pustejovsky, 2015).”
BackgroundMethodSelf-citationFranklin, C., Ding, X., Kim, J., Zhang, A., Hai, A. H., Jones, K. V., Nachbaur, M., & O’Connor, A. (2023) · Research on Social Work Practice · doi:10.1177/10497315231162611 · Cited by 32
“Data analyses were conducted in R statistical software (version 3.5.3), including the metafor and robumeta packages to implement robust variance estimation (RVE) in meta-regression.”
MethodInfluentialChen, Q., Yan, Z., Moeyaert, M., & Bangert‐Drowns, R. L. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108432 · Cited by 41
Gefen, E., Launder, N. H., Davey, C. G., Nahum, M., Gilboa, Y., Finke, C., Gavelin, H. M., Lautenschlager, N. T., & Lampit, A. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01872-6 · Cited by 27
A meta‐analysis on the effects of just‐below versus round prices
Troll, E. S., Frankenbach, J., Friese, M., & Loschelder, D. D. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1353 · Cited by 28
Xu, F., Gage, N. A., Zeng, S., Zhang, M., Iun, A., O'Riordan, M., & Kim, E. (2024) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-024-06563-4 · Cited by 23
“The overall effect size was calculated using a robust variance estimation (RVE; Fisher & Tipton, 2015; Hedges et al., 2010; Tipton, 2015; Tipton & Pustejovsky, 2015) model to address the fact that there were multiple dependent effect sizes from the same studies, as various assessment tools were…”
A Meta-Analysis Examining the Effect of E-Book Use on Literacy Outcomes for Students in Grades K–12
Swanson, E. A., Austin, C. R., Stewart, A. A., & Scammacca, N. K. (2020) · Reading & Writing Quarterly · doi:10.1080/10573569.2019.1696724 · Cited by 34
“Because the number of studies in this meta-analysis was less than 40, we implemented the small-sample correction in the model as a means of avoiding inflated Type I error (Tipton, 2015; Tipton & Pustejovsky, 2015).”
Does parochial cooperation exist in childhood and adolescence? A meta‐analysis
Lazić, A., Purić, D., & Krstić, K. (2021) · International Journal of Psychology · doi:10.1002/ijop.12791 · Cited by 27
Muharib, R., Dowdy, A., Rajaraman, A., & Jessel, J. (2021) · Autism · doi:10.1177/13623613211065540 · Cited by 24
“…Moeyaert et al. (2021)’s recommendation of modeling potential moderating variables using a multilevel approach; furthermore, metaregres-sion models consisting of cluster-robust variance estimation and small sample corrections were applied to account for small samples (Tipton & Pustejovsky, 2015).”
MethodAffective and cognitive impact of social overinclusion: a meta-analytic review of cyberball studies
Hay, D. E., Bleicher, S., Azoulay, R., Kivity, Y., & Gilboa‐Schechtman, E. (2023) · Cognition & Emotion · doi:10.1080/02699931.2022.2163619 · Cited by 31
“This approach has been shown to perform well in recent simulations (e.g. Tipton & Pustejovsky, 2015).”
MethodShen, J., & Wu, H. (2024) · Educational Administration Quarterly · doi:10.1177/0013161x241286527 · Cited by 57
Zainal, N. H., Wang, V. Y., Garthwaite, B., & Curtiss, J. E. (2025) · Health Psychology Review · doi:10.1080/17437199.2025.2547610 · Cited by 37
“All models applied robust variance estimation with small-sample corrections to adjust effect size estimates and standard errors ( SE ; Fernandez-Castilla et al., 2020; Tipton & Pustejovsky, 2015).”
Schudde, L. T., & Brown, R. S. (2019) · Texas Digital Library (University of Texas) · doi:10.26153/tsw/9473 · Cited by 22
“…sizes, we calculate robust variance estimates with small sample adjustments on regression coefficients (Tipton 2015) and F-statistics for categorical effect size moderators from an approximate Hotelling T-squared test when we use multiple dummy or indicator variables (Tipton and Pustejovsky 2015).”
MethodA Meta-analysis on the Effect of Expertise on Eye Movements during Music Reading
Perra, J., Latimier, A., Poulin-Charronnat, B., Baccino, T., & Drai-Zerbib, V. (2022) · Journal of Eye Movement Research · doi:10.16910/jemr.15.4.1 · Cited by 22
“This estimator, and the associated degrees of freedom, were adjusted for small sample sizes as recommended by Tipton and Pustejovsky (2015).”
MethodLynch, K., Gonzalez, K., Hill, H., & Merritt, R. (2025) · AERA Open · doi:10.1177/23328584251335302 · Cited by 21
“The use of CHE with RVE guards against model misspecification, given the complexities inherent in datasets with dependencies among effect sizes (Tipton & Pustejovsky, 2015).”
Say something nice: A meta-analytic review of peer reporting interventions
Collins, T. A., Drevon, D. D., Brown, A. M., Villarreal, J. N., Newman, C. L., & Endres, B. E. (2020) · Journal of School Psychology · doi:10.1016/j.jsp.2020.10.002 · Cited by 24
Shin, M., Bryant, D. P., Powell, S. R., Jung, P., Ok, M. W., & Hou, F. (2020) · Remedial and Special Education · doi:10.1177/0741932520964918 · Cited by 25
“…measures within a study, we also applied a robust variance estimation (RVE) method to combine effect sizes across studies and estimated standard errors and confidence intervals (CIs) around the average intervention effect estimates (Hedges et al., 2010; Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodManipulating belief in free will and its downstream consequences: A meta-analysisPreprint
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2021) · doi:10.31234/osf.io/quwgr · Cited by 21
Ojha, M., Naidu, D. G. T., & Bagchi, S. (2022) · Journal of Ecology · doi:10.1111/1365-2745.13841 · Cited by 19
“2.4.2 | RVE models We used the RVE approach to account for non- independence when multiple data points from a study can be correlated (Nakagawa, Senior, et al., 2021; Tipton & Pustejovsky, 2015).”
MethodInfluentialGómez‐Soria, I., Iguacel, I., Cuenca‐Zaldívar, J. N., Aguilar‐Latorre, A., Peralta, P., Latorre, E., & Calatayud, E. (2023) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2023.105114 · Cited by 25
Reading Interventions for Students in Grades 3–12 With Significant Word Reading Difficulties
Boucher, A. N., Bhat, B. H., Clemens, N. H., Vaughn, S. R., & O’Donnell, K. (2023) · Journal of Learning Disabilities · doi:10.1177/00222194231207556 · Cited by 27
“We also applied small-sample corrections for both the residuals and the degrees of freedom (Tipton, 2015).”
BackgroundMethodInfluentialAre Observed Classroom Practices Related to Student Language/Literacy Achievement?
Cao, Y., Kim, Y. G., & Cho, M. (2022) · Review of Educational Research · doi:10.3102/00346543221130687 · Cited by 24
“When the number of studies is small, the test statistics and confidence intervals based on RVE can have inflated Type I error (Hedges et al., 2010; Tipton & Pustejovsky, 2015).”
Loneliness and the Big Five Personality Traits: A Meta-AnalysisPreprint
Buecker, S., Maes, M., Denissen, J. J. A., & Luhmann, M. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/fx5bq · Cited by 21
Sermier Dessemontet, R., Geyer, M., Linder, A.-L., Atzemian, M., Martinet, C., Meuli, N., Audrin, C., & de Chambrier, A.-F. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100615 · Cited by 17
Patten, R. K., Tacey, A., Bourke, M., Smith, C., Pascoe, M. C., Vogrin, S., Parker, A. G., McKenna, M. J., Tran, P., Gori, M. D., et al. (2022) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2022.07.007 · Cited by 18
Solari, E. J., Kehoe, K. F., Cho, E., Hall, C., Vargas, I., Dahl‐Leonard, K., Richmond, C. L., Henry, A. R., Cook, L., Hayes, L. L., et al. (2022) · Learning Disabilities Research and Practice · doi:10.1111/ldrp.12286 · Cited by 27
“Therefore, we fit multivariate random-effects models assuming a correlation of .80 and further applied the RVE method to standard errors, hypothesis tests, and confidence intervals to make small-sample corrections (Tipton & Pustejovsky, 2015), using study as a clustering unit.”
MethodLaSota, R. R., Polanin, J. R., Perna, L. W., Rodgers, M. A., & Austin, M. J. (2024) · Review of Educational Research · doi:10.3102/00346543241239955 · Cited by 26
“We also used the default small sample correction suggested by Tipton and Pustejovsky (2015), so that the variance-covariance estimator is approximately unbiased when the number of studies is small or moderate.”
Revisiting the effectiveness of HOPE/swift‐certain‐fair supervision programs: A meta‐analytic review
Pattavina, A., Long, J. S., Petrich, D. M., Byrne, J. M., Cullen, F. T., & Taxman, F. S. (2023) · Criminology & Public Policy · doi:10.1111/1745-9133.12635 · Cited by 21
Ribeiro, L. A., Donolato, E., Aguiar, C., Correia, N., & Zachrisson, H. (2023) · Journal of Cognition and Development · doi:10.1080/15248372.2023.2248259 · Cited by 11
“CHE models derive from Robust Variance Estimation (RVE) models for handling effects and variance estimates related to within and between study dependencies (Fisher & Tipton, 2015; Tipton & Pustejovsky, 2015; see also Pustejovsky & Tipton, 2021).”
MethodThe relation between reading and externalizing behavior: a correlational meta-analysis
Pickren, S. E., Torelli, J. N., Miller, A. H., & Chow, J. C. (2024) · Annals of Dyslexia · doi:10.1007/s11881-024-00307-w · Cited by 14
“We performed the analysis in R (version 4.0.2) using the “metafor” and “robumeta” packages (Fisher & Tipton, 2015; Tipton & Pustejovsky, 2015; Viechtbauer, 2010).”
MethodInfluentialShi-jie, J., Zhang, A., Chen, Y., Shen, C., Currin-McCulloch, J. A., & Zhu, C. (2021) · Supportive Care in Cancer · doi:10.1007/s00520-021-06166-0 · Cited by 19
INTERNAL AND EXTERNAL VALIDITY OF THE COMPARATIVE INTERRUPTED TIME‐SERIES DESIGN: A META‐ANALYSIS
Coopersmith, J., Cook, T. D., Zurovac, J., Chaplin, D., & Forrow, L. V. (2022) · Journal of Policy Analysis and Management · doi:10.1002/pam.22361 · Cited by 19
Using Chained or Tandem Schedules With Functional Communication Training: A Systematic Review
Torelli, J. N., & Pickren, S. E. (2022) · Behavior Modification · doi:10.1177/01454455221077420 · Cited by 15
Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M. F., et al. (2023) · doi:10.31219/osf.io/mk35u · Cited by 15
Aghajanian, S., Shafiee, A., Ahmadi, A., & Elsamadicy, A. A. (2023) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2023.06.013 · Cited by 14
Weisburd, D. L., Petersen, K., Telep, C. W., & Fay, S. (2024) · Criminology & Public Policy · doi:10.1111/1745-9133.12665 · Cited by 33
Hostutler, C. A. A., Shahidullah, J. D., Mautone, J. A., Rybak, T. M., Okoroji, C., Bruni, T. P., Stephenson, K. G., Utset, L. V., Freeman, K. A., LaLonde, L., et al. (2024) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsae038 · Cited by 17
Pustejovsky, J. E., & Tipton, E. (2016) · Figshare · doi:10.6084/m9.figshare.4055403 · Cited by 14
Self-citationGnambs, T., & Schroeders, U. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1673 · Cited by 15
“37,38 Importantly, although the robust standard errors are unbiased even if the “ working model ” is not correctly specified, the precision of the resulting estimates increases the closer the true dependency structure is approximated.”
BackgroundJaldin, M. A., Balbim, G., Pinto, J., Negrete, M., Motl, R., Bustamante, E., Aguiñaga, S., Kaushal, N., Castillo, C., Khanna, S., et al. (2024) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000003585 · Cited by 7
“We performed approximate Hotelling – Zhang tests (HTZ) for categorical moderators with three or more levels and t -tests for continuous moderators and binary moderators (49).”
Interactions between faces and visual context in emotion perception: A meta-analysis
Steward, B. A., Mewton, P., Palermo, R., & Dawel, A. (2025) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02678-6 · Cited by 22
“While the use of RVE was not specified in our pre-registered protocol, this approach better estimates unbiased effects (Hedges et al., 2010; Tipton & Pustejovsky, 2015) and meta-analytic models with and without RVE produced the same direction of results (see supplemental r-markdown).”
Nudging smokers away from lighting up: A meta-analysis of framing effect in current smokers
Waheed, H. (2023) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2023.101998 · Cited by 14
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · BMC Public Health · doi:10.1186/s12889-024-18907-x · Cited by 34
“As suggested by Tipton and Pustejovsky (2015) the approximate Hotelling’s T 2 Z test was used to test moderators for perceived capabilities.”
O’Keefe, K., Chen, M., Lesser, K., DuVall, A. S., & Dils, A. T. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21070881 · Cited by 19
“To synthesize the effect size estimates across the included studies, meta-regression with robust variance estimation (RVE) was employed [25,26].”
MethodMadsø, K. G., Molde, H., Hynninen, M. J., & Nordhus, I. H. (2021) · Clinical Gerontologist · doi:10.1080/07317115.2021.1978121 · Cited by 20
Nakagawa, S., Yang, Y., Macartney, E. L., Spake, R., & Lagisz, M. (2023) · doi:10.32942/x2kk5n · Cited by 13
Spurious Precision in Meta-Analysis of Observational ResearchPreprint
Iršová, Z., Bom, P. R. D., Havránek, T., & Rachinger, H. J. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/3qp2w · Cited by 21
Löffler, M. T., Daikeler, J., Wagner, L., Natalucci, G., & Daum, M. M. (2025) · PEDIATRICS · doi:10.1542/peds.2024-070477 · Cited by 16
Bourke, M., Maddren, C. I., Sippel, F., & Thomas, G. (2026) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2025.6490 · Cited by 25
A meta-analysis on the effectiveness of functional literacy interventions for adults
Kindl, J., & Lenhard, W. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100569 · Cited by 18
Bline, A. P., Ellis, L. B., Pelch, K. E., Lam, J., Sen, Ś., Zlatnik, M. G., & Varshavsky, J. (2024) · Environment International · doi:10.1016/j.envint.2024.108711 · Cited by 14
A meta-analysis of the association of death anxiety with psychological distress and psychopathology
Dumitru, E. P., Cardoș, R. A.-I., Milea, I., Gambarota, F., Altoè, G., David, D. O., & Cristea, I. A. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02115-7 · Cited by 20
“RVE was implemented in the R package clubSandwich 64,65 (version 0.5.11), and we used the Cluster Robust-2 small-sample adjustment 64 .”
InfluentialLasauskaite, R., Wüst, L. N., Schöllhorn, I., Richter, M. M., & Cajochen, C. (2025) · LEUKOS The Journal of the Illuminating Engineering Society of North America · doi:10.1080/15502724.2025.2493669 · Cited by 20
“Meta-analyses were performed and forest plots were generated using the metafor package for R (Tipton 2015; Tipton and Pustejovsky 2015), which allowed performing a multilevel meta-analysis.”
González-González, A. M., Pellitero, M. T., Tejerina, N. L., Soto-Goñi, X., Cuenca‐Zaldívar, J. N., & Romero, E. A. S. (2025) · BMC Oral Health · doi:10.1186/s12903-025-06789-3 · Cited by 13
“Because of the violation of the principle of independence when analyzing repeated measures in the studies, the standard errors were adjusted using the robust variance estimator (RVE) proposed by Pustejovsky and Tip-ton [18] and applying the Satterthwaite adjustment to the degrees of freedom.”
Teacher Professional Development for Reading: A Review of the State of the Research
Filderman, M. J., Didion, L. A., Austin, C. R., Payne, S. B., Silvert, C., & Wexler, J. (2025) · Reading Research Quarterly · doi:10.1002/rrq.70054 · Cited by 19
“All eligible, independent effect sizes were included from each study using random effects robust variance estimation to account for the statistical dependencies of correlated effects of some studies contributing multiple effect sizes (Hedges et al. 2010; Tipton and Pustejovsky 2015).”
A panel analysis of groundwater use in California
Ke, J., & Blum, H. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.129357 · Cited by 12
Oliveira, C. M., Henderson, L. M., & Hayiou‐Thomas, M. E. (2023) · Cognitive Science · doi:10.1111/cogs.13310 · Cited by 15
Täschner, J., Dicke, T., Reinhold, S., & Holzberger, D. (2023) · doi:10.31234/osf.io/cds45 · Cited by 11
Jernslett, M., Kilmer, R. P., Avraam, D., & Anastassiou‐Hadjicharalambous, X. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102617 · Cited by 13
Cho, E., Dahl‐Leonard, K., Kehoe, K. F., Capin, P., Hall, C., & Solari, E. J. (2023) · Topics in Language Disorders · doi:10.1097/tld.0000000000000312 · Cited by 21
“(2020) Y EM None Y Torgesen et al. (2010)—RWT Y IT, G, PC None N Torgesen et al.”
BackgroundDo jihadist terrorist attacks cause changes in institutional trust? A multi‐site natural experiment
Nägel, C., Nivette, A. E., & Czymara, C. S. (2023) · European Journal of Political Research · doi:10.1111/1475-6765.12612 · Cited by 14
“We use the bias-reduced linearisation method (‘CR2’) recommended for small samples to calculate the confidence intervals and p-values (Tipton & Pustejovsky, 2015).”
Method
Metadata
- Authors
- Tipton, E., & Pustejovsky, J. E.
- Affiliations
- Columbia University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2015-12
- Volume
- 40
- Issue
- 6
- Pages
- 604-634
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2276186231
- Cited by
- 362
- References
- 31
- Citations by year
- 2026: 66 · 2025: 49 · 2024: 46 · 2023: 47 · 2022: 49 · 2021: 30 · 2020: 31 · 2019: 21 · 2018: 14 · 2017: 6 · 2016: 3
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- robust variance estimation, meta-regression, type I error inflation, F-statistic, moderators, dependent effect sizes, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.