Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression
The DOI matches its Crossref record; 29 of the 31 references it cites were found in a registry or an index.
Abstract
Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Since there is uncertainty around the correlation value used for imputation, robust inference methods with an adjustment for small samples was used, so that the interpretation of fi xed effects were unbiased [51].”
From The effect of caffeine on subsequent sleep: A systematic review and meta-analysis (2023) · cites this paper as [51]
In the citing paper
“Given the limited number of studies included, we employed small sample correction to adjust for inflated Type I errors in test statistics based on RVE using t tests (Tipton, 2015;Tipton & Pustejovsky, 2015).”
From Teacher Professional Development and Student Mathematics Achievement: A Meta-Analysis of the Effects and Moderators (2025) · cites this paper as Tipton & Pustejovsky, 2015
In the citing paper
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky (2015)
In the citing paper
“Like the bias of the CRVE estimator itself, the accuracy of the asymptotic approximations depends on design features such as the degree of imbalance across clusters, skewness or leverage of the covariates, and the similarity of cluster sizes (Tipton and Pustejovsky 2015; McCaffrey, Bell, and Botts 2001; MacKinnon and Webb 2016; Carter, Schnepel, and Steigerwald 2013).”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
In the citing paper
“However, simulation studies reported here and elsewhere (Tipton 2015; Tipton and Pustejovsky 2015) have demonstrated that the approach is actually robust to a high degree of misspecification in the working model.”
From Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models (2018) · cites this paper as Tipton and Pustejovsky 2015
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,723
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“The multivariate meta-analysis model requires good estimates of the correlation between dependent effect sizes (Gleser & Olkin, 2009), which typically are not available to the meta-analyst.”
MethodAspects of multivariate statistical theory
Muirhead, R. J. 1. (1982) · Cited by 3,032
Aspects of Multivariate Statistical Theory
Muirhead, R. J. (1982) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316559 · Cited by 2,260 · cited 2×
“For a random q q matrix D such that ηD follows a Wishart distribution with η degrees of freedom and scale matrix Iq, the covariances among the entries of D satisfy: ηCovðdst; duvÞ ¼ Iðs ¼ uÞIðt ¼ vÞ þ Iðs ¼ vÞIðt ¼ uÞ; ð11Þ for s,t,u,v¼ 1, . . . , q (Muirhead, 1982, section 3.2.2).”
BackgroundInfluentialRobust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 20×
“However, evidence from several simulation studies indicates that, even with these corrections, the test has inflated Type I error when there are fewer than 40 independent studies (Hedges et al., 2010; Tipton, 2013, 2015; Williams, 2012). Tipton (2015) proposed alternative methods for testing single meta-regression coe…”
BackgroundMethodSelf-citationThe malleability of spatial skills: A meta-analysis of training studies.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N. S. (2013) · Psychological Bulletin · doi:10.1037/a0028446 · Cited by 1,899 · cited 1×
“…(e.g., Wilson, Lipsey, Tanner-Smith, Huang, & Steinka-Fry, 2011), psychology (e.g., Samson, Tipton and Pustejovsky 605 Ojanen, & Hollo, 2012; Uttal et al., 2013), and intervention science (e.g., De Vibe, Bjørndal, Tipton, Hammerstrøm, & Kowalski, 2012; Tanner-Smith, Wilson, & Lipsey, 2013).”
Self-citationSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · cited 28×
“Another approach, introduced by Bell and McCaffrey (2002) and followed by Tipton (2015), is to instead use the same working model that was used to develop the weighting matrices Wj and adjustment matrices Aj.”
BackgroundMethodResultSelf-citationTanner-Smith, E. E., & Tipton, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1091 · Cited by 588 · cited 4×
“The per-group sample size per study (nj) ranged from 32 to 130 and included the same values as in the second simulation from Tipton (2015). We chose to use a variable number of effect sizes per study and variable sample sizes in order to emulate conditions likely to be observed in practice as well as to induce a great…”
MethodResultSelf-citationResearch Synthesis and Meta-Analysis
Cooper, H., Patall, E., & Lindsay, J. (2009) · The SAGE Handbook of Applied Social Research Methods · doi:10.4135/9781483348858.n11 · Cited by 397
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2012.05.006 · Cited by 331 · cited 2×
“As with the t-test developed by Tipton (2015) for single-parameter hypothesis tests, we seek a test that performs well in small samples, thus enabling broad application in meta-analysis.”
Method
Research works printed but not linked: 6
- 10.3102/10769986019002091 · doi:10.3102/10769986019002091 · Paper page
- Bell R. M., McCaffrey D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28, 169–181.
- Cooper H. M. (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: SAGE.
- 10.4073/csr.2012.3 · DOI · Paper page
- Fisher Z. Tipton E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Retrieved from arXiv preprint arXiv:1503.02220 · arXiv
- McCaffrey D. F. Bell R. M. Botts C. H. (2001 August). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association.
Cited by
Meta-analysis and the science of research synthesis
Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018) · Nature · doi:10.1038/nature25753 · Cited by 1,709
Childhood maltreatment and characteristics of adult depression: Meta-analysis
Nelson, J., Klumparendt, A., Doebler, P., & Ehring, T. (2017) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.180752 · Cited by 822
“We used an extension of the RVE method by Tipton & Pustejovsky 32 to compare the effects of different types of maltreatment with the grand mean effect (averaged over all maltreatment types) for research questions 1–3.”
Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018) · Psychological Bulletin · doi:10.1037/bul0000130 · Cited by 807
Self-citationMeta-analysis with robust variance estimation: Expanding the range of working models
Pustejovsky, J. E., & Tipton, E. (2022) · Prevention Science · doi:10.1007/s11121-021-01246-3 · Cited by 737
“Specifically, using a working model that more accurately captures the dependence structure will lead to meta-analysis and meta-regression coefficient estimates that are more precise and accurate (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodSelf-citationCipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., Sehgal, K., Zieher, A. K., Eveleigh, A., McCarthy, M., et al. (2023) · Child Development · doi:10.1111/cdev.13968 · Cited by 636
“We then applied a sandwich estimator using the “clubSandwich” package (Pustejovsky, 2022) to employ RVE for obtaining robust confidence intervals and p - values (Hedges et al., 2010; Tipton, 2015; Tipton & Pustejovsky, 2015).”
Pustejovsky, J. E., & Tipton, E. (2018) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2016.1247004 · Cited by 492
“These findings follow closely those from Tipton and Pustejovsky (2015), which indicated that the Type I error of the AHT test was close to nominal even when the working model was quite discrepant from the data-generating model.”
InfluentialSelf-citationFalck, R. S., Davis, J. C., Best, J. R., Crockett, R. A., & Liu-Ambrose, T. (2019) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2019.03.007 · Cited by 437
Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R
Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-016-0026-5 · Cited by 390
Self-citationPhysical activity and subjective well-being in healthy individuals: a meta-analytic review
Buecker, S., Simacek, T., Ingwersen, B., Terwiel, S., & Simonsmeier, B. A. (2021) · Health Psychology Review · doi:10.1080/17437199.2020.1760728 · Cited by 374
“If the number of included studies is small or moderate, coefficients and confidence intervals based on RVE may have an inflated Type I error (Tipton & Pustejovsky, 2015).”
BackgroundLoneliness and the Big Five Personality Traits: A Meta–Analysis
Buecker, S., Maes, M., Denissen, J. J. A., & Luhmann, M. (2020) · European Journal of Personality · doi:10.1002/per.2229 · Cited by 361
“However, if the number of studies included in a meta-analysis is small or moderate, test statistics and confidence intervals (CIs) based on RVE may suffer from inflated type I error (Tipton & Pustejovsky, 2015).”
BackgroundDoes Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 334
The effect of caffeine on subsequent sleep: A systematic review and meta-analysis
Gardiner, C., Weakley, J., Burke, L. M., Roach, G. D., Sargent, C., Maniar, N., Townshend, A., & Halson, S. L. (2023) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2023.101764 · Cited by 357
Principles for the Validation and Use of Personnel Selection Procedures
(2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2018.195 · Cited by 300
Dealing with effect size multiplicity in systematic reviews and meta‐analyses
López‐López, J. A., Page, M. J., Lipsey, M. W., & Higgins, J. P. T. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1310 · Cited by 294
Is bilingualism related to a cognitive advantage in children? A systematic review and meta-analysis.
Gunnerud, H. L., ten Braak, D., Reikerås, E. K. L., Donolato, E., & Melby-Lervåg, M. (2020) · Psychological Bulletin · doi:10.1037/bul0000301 · Cited by 281
Meta-analytic structural equation modeling with moderating effects on SEM parameters.
Jak, S., & Cheung, M. W. L. (2020) · Psychological Methods · doi:10.1037/met0000245 · Cited by 262
Risk factors for recurrent falls in older adults: A systematic review with meta-analysis
Jehu, D. A., Davis, J. C., Falck, R. S., Bennett, K. J., Tai, D., Souza, M. F., Cavalcante, B. R., Zhao, M., & Liu-Ambrose, T. (2021) · Maturitas · doi:10.1016/j.maturitas.2020.10.021 · Cited by 248
The association between teacher leadership and student achievement: A meta-analysis
Shen, J., Wu, H., Reeves, P., Zheng, Y., Ryan, L., & Anderson, D. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100357 · Cited by 238
The development of gender differences in spatial reasoning: A meta-analytic review.
Lauer, J. E., Yhang, E., & Lourenco, S. F. (2019) · Psychological Bulletin · doi:10.1037/bul0000191 · Cited by 239
Automatic imitation: A meta-analysis.
Cracco, E., Bardi, L., Desmet, C., Genschow, O., Rigoni, D., De Coster, L., Radkova, I., Deschrijver, E., & Brass, M. (2018) · Psychological Bulletin · doi:10.1037/bul0000143 · Cited by 228
“To address this issue, small sample corrections have been proposed (Tipton, 2015; Tipton & Pustejovsky, 2015).”
BackgroundDignath, C., Rimm-Kaufman, S., van Ewijk, R., & Kunter, M. (2022) · Educational Psychology Review · doi:10.1007/s10648-022-09695-0 · Cited by 219
“…standard errors (in a similar way as, for example, heteroscedasticity and autocorrelation consistent (HAC) standard errors), and (c) it does not require exact knowledge on the covariances between the effect sizes from the same clusters (Tanner-Smith & Tipton, 2014; Tipton & Pustejovsky, 2015).”
BackgroundclubSandwich: Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections
Pustejovsky, J. E. (2016) · CRAN: Contributed Packages · doi:10.32614/cran.package.clubsandwich · Cited by 216
Self-citationNakagawa, S., Yang, Y., Macartney, E. L., Spake, R., & Lagisz, M. (2023) · Environmental Evidence · doi:10.1186/s13750-023-00301-6 · Cited by 210
Achievement Motivation and Academic Dishonesty: A Meta-Analytic Investigation
Krou, M. R., Fong, C. J., & Hoff, M. A. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09557-7 · Cited by 194
“For hypothesis testing, we also used smallsample adjustments for t tests (Tipton 2015).”
MethodPhysical activity during pregnancy and postpartum depression: systematic review and meta- analysis
Nakamura, A., van der Waerden, J., Melchior, M., Bolze, C., El-Khoury, F., & Pryor, L. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.009 · Cited by 186
Hall, C., Dahl‐Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L., Vargas, I., et al. (2023) · Reading Research Quarterly · doi:10.1002/rrq.477 · Cited by 163
Strengthening the research base that informs STEM instructional improvement efforts: A meta-analysis
Lynch, K., Hill, H. C., Gonzalez, K. E., & Pollard, C. (2019) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373719849044 · Cited by 160
Task characteristics influence facial emotion recognition age-effects: A meta-analytic review.
Hayes, G. S., McLennan, S. N., Henry, J. D., Phillips, L. H., Terrett, G., Rendell, P. G., Pelly, R. M., & Labuschagne, I. (2020) · Psychology and Aging · doi:10.1037/pag0000441 · Cited by 159
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145
“If a study reported multiple effect sizes from the same sample that could not be treated as independent from each other, we accounted for this non-independence by using the cluster-robust inference method (also called robust variance estimation; RVE; Sidik and Jonkman, 2006; Hedges et al., 2010; Tipton and Pustejovsky…”
MethodInfluencing climate change attitudes in the United States: A systematic review and meta-analysis
Rode, J. B., Dent, A. L., Benedict, C. N., Brosnahan, D., Martinez, R. L., & Ditto, P. H. (2021) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2021.101623 · Cited by 177
Jukic, I., Castilla, A. P., Ramos, A. G., Van Hooren, B., McGuigan, M. R., & Helms, E. R. (2023) · Sports Medicine · doi:10.1007/s40279-022-01754-4 · Cited by 138
“In addition, cluster-robust variance estimation methods [50] with small-sample adjustments [51] were implemented to calculate standard errors of the overall effect size estimates, with clustering at the study level.”
MethodHow Effective Are Intentional Vocabulary‐Learning Activities? A Meta‐Analysis
WEBB, S., YANAGISAWA, A., & UCHIHARA, T. (2020) · The Modern Language Journal · doi:10.1111/modl.12671 · Cited by 100
“To account for correlated ESs, RVE (Hedges et al., 2010) with correlated weights and small-sample adjustments (Tipton, 2015; Tipton & Pustejovsky, 2015) was used.”
Park, C. L., Pustejovsky, J. E., Trevino, K., Sherman, A. C., Esposito, C., Berendsen, M., & Salsman, J. M. (2019) · Cancer · doi:10.1002/cncr.32078 · Cited by 125
“7, as well as small-sample corrections to standard errors, hypothesis tests, and confidence intervals.(53,54)”
BackgroundSelf-citationMerluzzi, T. V., Pustejovsky, J. E., Philip, E. J., Sohl, S. J., Berendsen, M., & Salsman, J. M. (2019) · Psycho-Oncology · doi:10.1002/pon.5148 · Cited by 122
“More specifically, we used a “correlated effects” working model, assuming a correlation of 0.7, as well as small‐sample corrections to standard errors, hypothesis tests, and CIs.45,46 To measure the extent of heterogeneity among the effect sizes, we reported the restricted maximum likelihood estimate of the between‐st…”
BackgroundSelf-citationSiegel, R., König, C. J., & Lazar, V. (2022) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2022.100227 · Cited by 121
Zhang, A., Borhneimer, L. A., Weaver, A., Franklin, C., Hai, A. H., Guz, S., & Li, S. (2019) · Journal of Behavioral Medicine · doi:10.1007/s10865-019-00046-z · Cited by 147
Zhang, A., Park, S., Sullivan, J. E., & Jing, S. (2018) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2018.01.170270 · Cited by 111
“We chose to conduct analyses in R, rather than software specific to metaanalysis (eg, RevMan), because R allowed for more flexibility in statistical modeling (eg, small sample size correction).(34) Sensitivity analysis using Robust Variance Estimation did not significantly alter results estimated with the typical appr…”
MethodClinton‐Lisell, V., Taylor, T. O., Bajpayee, S., Davison, M. L., Carlson, S. E., & Seipel, B. (2020) · Reading and Writing · doi:10.1007/s11145-020-10044-2 · Cited by 142
Hancock, J., Liu, S. X., Luo, M., & Mieczkowski, H. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4053961 · Cited by 101
Donolato, E., Cardillo, R., Mammarella, I. C., & Melby‐Lervåg, M. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13536 · Cited by 137
Täschner, J., Dicke, T., Reinhold, S., & Holzberger, D. (2024) · Review of Educational Research · doi:10.3102/00346543231221499 · Cited by 193
“Additionally, we applied cluster robust inference methods with a small sample correction, also known as robust variance estimation (RVE) (Pustejovsky & Tipton, 2022; Tanner-Smith et al., 2016; Tipton & Pustejovsky, 2015).”
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1338 · Cited by 97
Self-citationDessemontet, R. S., Martinet, C., de Chambrier, A.-F., Martini‐Willemin, B., & Audrin, C. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.01.001 · Cited by 128
Maggin, D. M., Pustejovsky, J. E., & Johnson, A. H. (2017) · Remedial and Special Education · doi:10.1177/0741932517716900 · Cited by 118
“Given the relatively small number of included samples, small-sample adjustments to RVE were used to maintain close-to-nominal error rates for hypothesis tests and confidence intervals (CIs; Tipton & Pustejovsky, 2015).”
MethodSelf-citationA Meta-Analytic Review of the Benefit of Spacing out Retrieval Practice Episodes on Retention
Latimier, A., Peyre, H., & Ramus, F. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09572-8 · Cited by 96
“Given the relatively small number of included samples, small-sample adjustments for hypothesis tests and confidence intervals (CIs; Tipton & Pustejovsky, 2015) were used for our analyses.”
MethodClinton‐Lisell, V., & Khan, S. (2019) · AERA Open · doi:10.1177/2332858419872212 · Cited by 128
The Competitive Effects of School Choice on Student Achievement: A Systematic Review
Jabbar, H., Fong, C. J., Germain, E., Li, D., Sánchez, J. D., Sun, W.-L., & Devall, M. (2019) · Educational Policy · doi:10.1177/0895904819874756 · Cited by 126
“We also adjusted for small-sample t tests to determine whether there was a relationship between focal variables and effect sizes in the population (Tipton & Pustejovsky, 2015).”
MethodHandling effect size dependency in meta-analysis
Gucciardi, D. F., Lines, R. L. J., & Ntoumanis, N. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1946835 · Cited by 99
Yanagisawa, A., Webb, S., & Uchihara, T. (2020) · Studies in Second Language Acquisition · doi:10.1017/s0272263119000688 · Cited by 118
“In order to appropriately deal with the potential Type I error inflation due to these dependent ESs, we applied the cluster-robust variance estimation (Hedges, Tipton, & Johnson, 2010) with small-sample adjustments (Tipton, 2015; Tipton & Pustejovsky, 2015).”
Hooren, B. V., Jukić, I., Cox, M., Frenken, K. G., Bautista, I. J., & Moore, I. S. (2024) · Sports Medicine · doi:10.1007/s40279-024-01997-3 · Cited by 126
Effects of action video game play on cognitive skills: A meta-analysis.
Bediou, B., Rodgers, M. A., Tipton, E., Mayer, R. E., Green, C. S., & Bavelier, D. (2023) · Technology Mind and Behavior · doi:10.1037/tmb0000102 · Cited by 132
Self-citationIs bilingualism associated with better working memory capacity? A meta-analysis
Monnier, C., Boiché, J., Armandon, P., Baudoin, S., & Bellocchi, S. (2021) · International Journal of Bilingual Education and Bilingualism · doi:10.1080/13670050.2021.1908220 · Cited by 108
“In case of 3-modality moderator variables (e.g. age groups) the Wald_test function from the ClubSandwich package provides an F-type test with degrees of freedom estimated using the approximate Hotelling’s T2Z method.”
MethodInfluentialZhang, A., Franklin, C., Jing, S., Bornheimer, L. A., Hai, A. H., Himle, J. A., Kong, D., & Ji, Q. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.008 · Cited by 100
Daucourt, M. C., Erbeli, F., Little, C. W., Haughbrook, R., & Hart, S. A. (2019) · Scientific Studies of Reading · doi:10.1080/10888438.2019.1631827 · Cited by 82
“(Continued ) study sample, all RVE analyses were conducted using a small-sample adjustment to minimize the risk for alpha-inflation (Tipton & Pustejovsky, 2015).”
MethodThe sleep benefit in episodic memory: An integrative review and a meta-analysis.
Berres, S., & Erdfelder, E. (2021) · Psychological Bulletin · doi:10.1037/bul0000350 · Cited by 74
Xu, K., Blazevich, A. J., Boullosa, D. A., Ramírez‐Campillo, R., Yin, M., Zhong, Y., Tian, Y., Finlay, M. J., Byrne, P. J., Cuenca‐Fernández, F., et al. (2025) · Sports Medicine · doi:10.1007/s40279-024-02170-6 · Cited by 97
Barton, E. E., Pustejovsky, J. E., Maggin, D. M., & Reichow, B. (2017) · Remedial and Special Education · doi:10.1177/0741932517729508 · Cited by 74
“Given the relatively small number of included samples, small-sample adjustments for hypothesis tests and confidence intervals (CIs; Tipton, 2015; Tipton & Pustejovsky, 2015) were used for all analyses.”
MethodSelf-citationFranklin, C., Kim, J. S., Beretvas, S. N., Zhang, A., Guz, S., Park, S., Montgomery, K. L., Chung, S., & Maynard, B. R. (2017) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-017-0235-4 · Cited by 96
“Because this review did not meet the above criteria, 24 primary studies and 123 effect size estimates, we used small sample size correction (TannerSmith and Tipton 2014; Tipton 2015) running RVE to control for a possibly inflated Type I error of test statistics and confidence intervals (Tipton and Pustejovsky 2015).”
BackgroundMethodYanagisawa, A., & Webb, S. (2021) · Language Learning · doi:10.1111/lang.12444 · Cited by 109
Martín, E. L., Gutiérrez-de-Rozas, B., González-Benito, A., & Casas, E. E. (2023) · International Journal of Educational Research · doi:10.1016/j.ijer.2023.102199 · Cited by 171
Stevens, E. A., Austin, C. R., Moore, C., Scammacca, N. K., Boucher, A. N., & Vaughn, S. R. (2021) · Exceptional Children · doi:10.1177/0014402921993406 · Cited by 120
“Because the meta-analyses included fewer than 40 studies, we implemented a small-sample correction to avoid inflating Type I error (Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodBalbim, G. M., Falck, R. S., Barha, C. K., Starkey, S. Y., Bullock, A., Davis, J. C., & Liu-Ambrose, T. Y. L. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2021-104955 · Cited by 78
Meta-Analytic Findings on Reading in Children With Cochlear Implants
Wang, Y., Sibaii, F., Lee, K., Gill, M. J., & Hatch, J. L. (2021) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/deafed/enab010 · Cited by 78
Meta-analysis in vocational behavior: A systematic review and recommendations for best practices
Rudolph, C. W., Chang, C. K., Rauvola, R. S., & Zacher, H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103397 · Cited by 69
Stucke, N. J., & Doebel, S. (2024) · Psychological Bulletin · doi:10.1037/bul0000445 · Cited by 69
Salmerón, L., Altamura, L., Delgado, P., Karagiorgi, A., & Vargas, C. (2023) · Journal of Educational Psychology · doi:10.1037/edu0000830 · Cited by 104
Altamura, L., Vargas, C., & Salmerón, L. (2025) · Review of Educational Research · doi:10.3102/00346543231216463 · Cited by 59
“Thus, we calculated cluster-robust standard errors, statistical tests, and confidence intervals for our estimates by using the “CR2” method (Tipton & Pustejovsky, 2015).”
Their own worst enemy? Collective narcissists are willing to conspire against their in‐group
Biddlestone, M., Cichocka, A. K., Główczewski, M., & Cisłak, A. (2022) · British Journal of Psychology · doi:10.1111/bjop.12569 · Cited by 60
Dignath, C., van Ewijk, R. J. G., Perels, F., & Fabriz, S. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09718-4 · Cited by 98
“RVE builds on the adjustment of standard errors and does not require precise information on the covariances between the effect sizes from the same clusters (Tipton & Pustejovsky, 2015).”
BackgroundMeta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 67
The effect of alcohol on subsequent sleep in healthy adults: A systematic review and meta-analysis
Gardiner, C., Weakley, J. J. S., Burke, L. M., Roach, G. D., Sargent, C., Maniar, N., Huynh, M., Miller, D. J., TOWNSHEND, A. D., & Halson, S. L. (2024) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2024.102030 · Cited by 83
“Owing to uncertainty around this correlation value, we used robust inference methods with an adjustment for small samples, so that our interpretation of fixed effects were unbiased [52].”
Winters, K. L., Jasso, J., Pustejovsky, J. E., & Byrd, C. T. (2022) · Journal of Speech Language and Hearing Research · doi:10.1044/2022_jslhr-22-00017 · Cited by 75
Self-citationNag, S., Vagh, S. B., Dulay, K. M., Snowling, M. J., Donolato, E., & Melby‐Lervåg, M. (2024) · Psychological Bulletin · doi:10.1037/bul0000417 · Cited by 93
Dietrichson, J., Filges, T., Seerup, J. K., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Eiberg, M. (2021) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1152 · Cited by 54
RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
“The participants were members of panels that had been recruited through non-probability sampling methods to complete online questionnaires in return for small amounts of money or redeemable ‘points’ 40,41 .”
BackgroundMethodSelf-citationRetractedZhao, Y., Li, Y., Ma, S., Xu, Z., & Zhang, B. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105279 · Cited by 100
Velayudhan, L., McGoohan, K., & Bhattacharyya, S. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003524 · Cited by 58
Schudde, L. T., & Brown, R. S. (2019) · Sociology of Education · doi:10.1177/0038040719848445 · Cited by 60
Socio‐Economic Status and Creativity: A Meta‐Analysis
Acar, S., Tadık, H., Uysal, R., Myers, D., & Inetas, B. (2022) · The Journal of Creative Behavior · doi:10.1002/jocb.568 · Cited by 65
Smith, T. E., Holmes, S. R., Romero, M. E., & Sheridan, S. M. (2022) · School Mental Health · doi:10.1007/s12310-022-09510-9 · Cited by 105
“The R package robumeta (Fisher et al., 2017), including small-sample corrections by Tipton and Pustejovsky (2015), was used for all RVE analyses.”
MethodEnhancing mathematical problem posing competence: a meta-analysis of intervention studies
Zhang, L., Stylianides, G. J., & Stylianides, A. J. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00507-1 · Cited by 51
“We used small-sample adjusted t -test to determine if there was a relationship between moderators and effect sizes in the population as well as adjusted F -test to assess model fit (Tipton & Pustejovsky, 2015).”
MethodMok, S. Y., Rupp, D., & Holzberger, D. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100552 · Cited by 73
Denton, C. A., Hall, C., Cho, E., Cannon, G., Scammacca, N. K., & Wanzek, J. (2021) · Learning and Individual Differences · doi:10.1016/j.lindif.2021.102062 · Cited by 102
Fong, C. J., Krou, M. R., Johnston-Ashton, K., Hoff, M. A., Lin, S., & Gonzales, C. (2021) · Educational Research Review · doi:10.1016/j.edurev.2021.100407 · Cited by 70
Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683221087527 · Cited by 72
“As a solution, small-sample (i.e., a limited number of studies in the meta-analysis) corrections have been proposed for both t tests (Tipton, 2015) and F tests (Tipton & Pustejovsky, 2015).”
BackgroundMethodYang, Y., Macleod, M. R., Pan, J., Lagisz, M., & Nakagawa, S. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105016 · Cited by 60
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
Self-citationInteractive features of E-texts’ effects on learning: a systematic review and meta-analysis
Clinton-Lisell, V., Seipel, B., Gilpin, S., & Litzinger, C. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1943453 · Cited by 48
“In addition, a small sample size correction for RVE was included in the analysis (Tipton, 2015; see Tipton & Pustejovsky, 2015).”
MethodRoberts, G. J., Cho, E., Garwood, J. D., Goble, G. H., Robertson, T., & Hodges, A. (2020) · Educational Psychology Review · doi:10.1007/s10648-019-09485-1 · Cited by 47
Online Versus Offline Peer Feedback in Higher Education: A Meta-Analysis
Jongsma, M. V., Scholten, D. J., van Muijlwijk‐Koezen, J. E., & Meeter, M. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331221114181 · Cited by 72
“…for the dependence among effect size estimates in certain studies in this meta-analysis, a correlate effects model with robust variance estimates and small-sample corrections was conducted using R studio with the use of the R package “ robumeta ” (Fisher & Tipton, 2015; Tipton & Pustejovsky, 2015).”
MethodPietsch, M., Aydın, B., & Gümüş, S. (2023) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737231197434 · Cited by 99
“The adjustment was based on the Satterthwaite approximation by declaring countries, the highest level, as clusters (Tipton, 2015; Tipton & Pustejovsky, 2015).”
Single-case synthesis tools II: Comparing quantitative outcome measures
Zimmerman, K. N., Pustejovsky, J. E., Ledford, J. R., Barton, E. E., Severini, K. E., & Lloyd, B. P. (2018) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2018.02.001 · Cited by 52
Self-citationInterventions for academically underachieving students: A systematic review and meta-analysis
Snyder, K. E., Fong, C. J., Painter, J. K., Pittard, C. M., Barr, S. M., & Patall, E. A. (2019) · Educational Research Review · doi:10.1016/j.edurev.2019.100294 · Cited by 70
Vembye, M. H., Weiss, F., & Bhat, B. H. (2023) · Review of Educational Research · doi:10.3102/00346543231186588 · Cited by 104
“To tackle this challenge, we applied robust variance estimation ( RVE ; Hedges et al., 2010; Pustejovsky & Tipton, 2021; Tipton & Pustejovsky, 2015), which has shown to be the most accurate method for meta-analyzing dependent effect sizes (Fernández-Castilla, Aloe, et al., 2020; Vembye et al.,…”
MethodInfluentialWilliams, R. T., Citkowicz, M., Miller, D. I., Lindsay, J., & Walters, K. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2021.2009072 · Cited by 59
“…per study), we used robust variance estimation to adjust the standard errors and degrees of freedom for regression coefficients, using the small-sample correction based on the Satterthwaite approximation (Tipton, 2015; Tipton & Pustejovsky, 2015) and the clubSandwich R package (Pustejovsky, 2018).”
MethodSandbank, M. P., Pustejovsky, J. E., Bottema‐Beutel, K., Caldwell, N., Feldman, J. I., Crowley, S., & Woynaroski, T. G. (2024) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2024.1832 · Cited by 57
Self-citationEffects of the 5E Instructional Model: A Systematic Review and Meta-Analysis
Polanin, J. R., Austin, M. J., Taylor, J. A., Steingut, R. R., Rodgers, M. A., & Williams, R. T. (2024) · AERA Open · doi:10.1177/23328584241269866 · Cited by 94
“Of the six confirmatory moderator analyses we conducted, two moderators—instructional model and outcome construct—yielded statistically significant F -statistics (Tipton & Pustejovsky, 2015).”
Smith, T. E., Thompson, A. M., & Maynard, B. R. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1223 · Cited by 104
Spry, E., Aarsman, S. R., Youssef, G. J., Patton, G. C., Macdonald, J. A., Sanson, A. V., Thomson, K., Hutchinson, D. M., Letcher, P., & Olsson, C. A. (2020) · Developmental Review · doi:10.1016/j.dr.2020.100934 · Cited by 54
Is retell a valid measure of reading comprehension?
Cao, Y., & Kim, Y. G. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100375 · Cited by 59
Metadata
- Authors
- Tipton, E., & Pustejovsky, J. E.
- Affiliations
- Columbia University (US)
- The University of Texas at Austin (US)
- Venue
- Journal of Educational and Behavioral Statistics
- Published
- 2015-12
- Volume
- 40
- Issue
- 6
- Pages
- 604-634
- ISSN
- 1076-9986 (ISSN-L)
- 1935-1054
- OpenAlex
- W2276186231
- Cited by
- 362
- References
- 31
- Citations by year
- 2026: 66 · 2025: 49 · 2024: 46 · 2023: 47 · 2022: 49 · 2021: 30 · 2020: 31 · 2019: 21 · 2018: 14 · 2017: 6 · 2016: 3
- FWCI
- 7.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Educational and Behavioral Statistics (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- robust variance estimation, meta-regression, type I error inflation, F-statistic, moderators, dependent effect sizes, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.