A history of meta‐regression: Technical, conceptual, and practical developments between 1974 and 2018
Open accessThe DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
At the beginning of the development of meta‐analysis, understanding the role of moderators was given the highest priority, with meta‐regression provided as a method for achieving this goal. Yet in current practice, meta‐regression is not as commonly used as anticipated. This paper seeks to understand this mismatch by reviewing the history of meta‐regression methods over the past 40 years. We divide this time span into four periods and examine three types of methodological developments within each period: technical, conceptual, and practical. Our focus is broad and includes development of methods in the fields of education, psychology, and medicine. We conclude the paper with a discussion of five consensus points, as well as open questions and areas of research for the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 2×
“This focus clearly built on the early promise of meta‐analysis as conceived of by Glass.27 In practice, however, Glass's vision was not being realized.”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 2×
“While RVE does not require specification of a covariance structure, the efficiency of the method can be improved by specifying a working model for the covariance structure.(83) Furthermore, Tipton(87) and Tipton and Pustejovsky(88) show that using a working model for the”
MethodSelf-citationImproved tests for a random effects meta‐regression with a single covariate
Knapp, G., & Hartung, J. (2003) · Statistics in Medicine · doi:10.1002/sim.1482 · Cited by 2,075 · cited 1×
“Knapp and Hartung developed hypothesis testing methods based on an alternative variance estimator and a t reference distribution with k − p degrees of freedom, which contained Type‐I error better than standard methods.(49) Later simulation work by Higgins and Thompson pitted three tests of meta‐ regression against one…”
MethodBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Meta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701
Methods to estimate the between‐study variance and its uncertainty in meta‐analysis
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P., Langan, D., & Salanti, G. (2016) · Research Synthesis Methods · doi:10.1002/jrsm.1164 · Cited by 1,685
QUANTITATIVE METHODS IN THE REVIEW OF EPIDEMIOLOGIC LITERATURE1
GREENLAND, S. (1987) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036298 · Cited by 1,644
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
Controlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 2×
“20 with 5 covariates and 100 studies—and that this result holds both for a sequence of separate simple meta‐regressions and for a single meta‐regression containing all of the covariates.(50) In their previous work, they argued that subgroup and meta‐regression analyses should be prespecified and conducted with Bonferr…”
BackgroundMethodSchwarzer, G., Carpenter, J. R., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0 · Cited by 1,230
Writing meta-analytic reviews.
Rosenthal, R. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.118.2.183 · Cited by 1,108 · cited 1×
“Rosenthal echoed this concern, noting that meta‐ analyses could be computed in existing software and worrying that making user‐friendly packages available could lead to poor practice.(47)”
BackgroundSmall sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810
Self-citationMETA‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (1989) · Journal of Economic Surveys · doi:10.1111/j.1467-6419.1989.tb00064.x · Cited by 805
META‐REGRESSION ANALYSIS: A QUANTITATIVE METHOD OF LITERATURE SURVEYS
Stanley, T. D., & Jarrell, S. B. (2005) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2005.00249.x · Cited by 805
Stijnen, T., Hamza, T. H., & Özdemir, P. (2010) · Statistics in Medicine · doi:10.1002/sim.4040 · Cited by 757 · cited 1×
“model approach that can perform better for outcomes that are rare events.(71) Guolo developed and tested a second‐order likelihood test, demonstrating that this test improves upon both the standard test and the Knapp‐ Hartung test.”
MethodThe power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672 · cited 2×
“In contrast, Hedges and Pigott developed analytic methods of calculating the power of tests for moderators in meta‐regression.(57) This work covered power for both fixed effect and mixed effects models, and for omnibus tests as well as tests of individual coefficients.”
BackgroundMethodCook, T. D., Shadish, W. R., & Wong, V. C. (2008) · Journal of Policy Analysis and Management · doi:10.1002/pam.20375 · Cited by 604
Multivariate meta‐analysis: Potential and promise
Jackson, D., Riley, R., & White, I. R. (2011) · Statistics in Medicine · doi:10.1002/sim.4172 · Cited by 530
Fixed effects and variance components estimation in three-level meta-analysis
Konstantopoulos, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.35 · Cited by 521
Invited commentary: a critical look at some popular meta-analytic methods
Greenland, S. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117248 · Cited by 511
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490 · cited 1×
“For example, Steiner and colleagues have shown that in order to reduce bias, models need to include covariates directly related to selection and that simple demographics are inadequate.(111) Importantly, in order to study these problems, novel designs(112) and sophisticated simulation studies are used.”
BackgroundPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · cited 1×
“The most recent period has seen dawning recognition and growing concern about the irreproducibility of primary study findings in a wide range of disciplines, including clinical medicine,(94,95) psychology,(96) and cognitive sciences.(97) The reproducibility crisis has brought concerns with Type I error and confounding…”
BackgroundPigott, T. D. (2012) · doi:10.1007/978-1-4614-2278-5 · Cited by 453
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450
Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R
Tanner-Smith, E. E., Tipton, E., & Polanin, J. R. (2016) · Journal of Developmental and Life-Course Criminology · doi:10.1007/s40865-016-0026-5 · Cited by 390 · cited 1×
“Additionally, software‐oriented papers and examples abound, providing researchers with examples and code snippets for implementation.(109) While there appear to be several methods to choose from, the best approach may be to combine them.”
BackgroundSelf-citationTipton, E., & Pustejovsky, J. E. (2015) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998615606099 · Cited by 362
Self-citationEmpirical bayes meta-analysis.
Raudenbush, S. W., & Bryk, A. S. (1985) · Journal of Educational Statistics · doi:10.3102/10769986010002075 · Cited by 299 · cited 1×
“To the extent the treatments interact with any of these factors, averaging can be misleading.(16)”
BackgroundThose confounded moderators in meta-analysis: good, bad, and ugly
Lipsey, M. W. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716202250791 · Cited by 283 · cited 1×
“Concerns regarding the confounding of substantive moderators with one another and with methodological characteristics have been echoed again and again over time.(32) A considerable body of methodological work indicates that confounding variables should be considered, whether using regression or subgroup analysis metho…”
BackgroundEstimation in Parallel Randomized Experiments
Rubin, D. B. (1981) · Journal of Educational Statistics · doi:10.3102/10769986006004377 · Cited by 269 · cited 5×
“Rubin showed that this model provides a way to quantify the degree of heterogeneity and to test hypotheses regarding differences between subgroups of effect sizes.(14) During the same period, Glass and colleagues introduced the use of ordinary least squares (OLS) regression methods to explore trends in effect size var…”
MethodInfluentialA comparison of procedures to test for moderators in mixed-effects meta-regression models
Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F. (2015) · Psychological Methods · doi:10.1037/met0000023 · Cited by 241
Estimating the Difference Between Published and Unpublished Effect Sizes: A Meta-Review
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016) · Review of Educational Research · doi:10.3102/0034654315582067 · Cited by 234 · cited 1×
“Over the past 40 years, the number of meta‐ analyses published has increased about fivefold every 10 years, and in the fields of education and psychology, nearly half of meta‐analyses now include over 100 studies.(4,5) Methods for meta‐analysis have also grown rapidly over time.”
BackgroundSalsman, J. M., Pustejovsky, J. E., Jim, H. S. L., Munoz, A. R., Merluzzi, T. V., George, L., Park, C. L., Danhauer, S. C., Sherman, A. C., Snyder, M. A., et al. (2015) · Cancer · doi:10.1002/cncr.29350 · Cited by 232
Self-citationA multivariate mixed linear model for meta-analysis.
Kalaian, H. A., & Raudenbush, S. W. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.3.227 · Cited by 229 · cited 2×
“This work was extended to meta‐regression models of correlations and partial‐regression coefficients by Becker,20 to other dependence structures for different types of covariation (eg, multiple end‐points and shared comparison groups) by Gleser and Olkin,21 and to mixed‐effects models with residual heterogeneity by Ka…”
MethodCombinations of techniques that effectively change health behavior: evidence from Meta-CART analysis
Dusseldorp, E., van Genugten, L., van Buuren, S., Verheijden, M. W., & van Empelen, P. (2014) · Health Psychology · doi:10.1037/hea0000018 · Cited by 138 · cited 1×
“Dusseldorp and colleagues proposed a CART technique for meta‐ analysis, demonstrating the approach with an example from health psychology.(90) Li and colleagues developed and studied the statistical properties of the CART technique.”
MethodA Review of Meta-Analysis Packages in R
Polanin, J. R., Hennessy, E. A., & Tanner-Smith, E. E. (2017) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998616674315 · Cited by 92
Cited by
Meta-analysis with robust variance estimation: Expanding the range of working models
Pustejovsky, J. E., & Tipton, E. (2022) · Prevention Science · doi:10.1007/s11121-021-01246-3 · Cited by 737
Self-citationO'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
“We expect that multilevel models for effect size data will become more commonly used in meta-analysis (Tipton et al., 2019a).”
BackgroundMethodInfluentialHow to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218
Transparency and reproducibility of meta-analyses in psychology: A meta-review
Polanin, J. R., Hennessy, E. A., & Tsuji, S. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620906416 · Cited by 100
Metadata
- Authors
- Tipton, E., Pustejovsky, J. E., & Ahmadi, H.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2019-02-20
- Volume
- 10
- Issue
- 2
- Pages
- 161-179
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2907033972
- Cited by
- 97
- References
- 41
- FWCI
- 4.62
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.