The DOI matches its Crossref record; 59 of the 62 references it cites were found in a registry or an index.
Fixed or random effects meta-analysis? Common methodological issues in systematic reviews of effectiveness
Abstract
Systematic review aims to systematically identify, critically appraise, and summarize all relevant studies that match predefined criteria and answer predefined questions. The most common type of systematic review is that assessing the effectiveness of an intervention or therapy. In this article, we discuss some of the common methodological issues that arise when conducting systematic reviews and meta-analyses of effectiveness data, including issues related to study designs, meta-analysis, and the use and interpretation of effect sizes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, it was recommended that the protocol should define the methods to be used for risk of bias assessment, meta-analysis and exploration of heterogeneity [41, 42, 44].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [44]
In the citing paper
“The articles recommended that the protocol should state the objectives, hypotheses to be tested and the rationale of the review, [29] and that it should describe eligibility criteria [29, 33, 35, 36, 38, 39, 41], define the type of studies to be included [35, 37, 42, 44], and give the reason when including observational studies in the review [35, 37, 40].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [44]
In the citing paper
“Twenty-five (27%) articles gave recommendations on item “considering different study designs” [8, 29, 34, 35, 37–39, 42, 44, 49, 51, 56, 61, 65, 70, 73, 75, 76, 77–83], with publication dates between 1995 and 2016 (median year of publication 2007).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [44]
In the citing paper
“A variation of this recommendation was to exclude studies that are at high risk of bias [42, 44, 54, 73, 82].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [44]
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
(2000) · Cited by 9,198
A basic introduction to fixed-effect and random-effects models for meta-analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.12 · Cited by 6,793
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,312
Meta-analyses involving cross-over trials: methodological issues
Elbourne, D. R., Altman, D. G., Higgins, J. P., Curtin, F., Worthington, H. V., & Vail, A. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.140 · Cited by 2,305
Hayes, R., & Moulton, L. H. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781584888178 · Cited by 881
15 Design and Analysis of Cross-Over Trials
Kenward, M. G., & Jones, B. C. (2007) · Handbook of statistics · doi:10.1016/s0169-7161(07)27015-4 · Cited by 873
Petitti, D. B. (1994) · Medical Entomology and Zoology · Cited by 869
Cross-Over Trials in Clinical Research
Senn, S. J. (1994) · Technometrics · doi:10.1080/00401706.1994.10485798 · Cited by 777
Issues in the meta-analysis of cluster randomized trials
Donner, A., & Klar, N. (2002) · Statistics in Medicine · doi:10.1002/sim.1301 · Cited by 734
JBI's Systematic Reviews: Study Selection and Critical Appraisal
Porritt, K., Gomersall, J., & Lockwood, C. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000450430.97383.64 · Cited by 644
Developing the Review Question and Inclusion Criteria
Stern, C., Jordan, Z., & McArthur, A. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000445689.67800.86 · Cited by 629
The Systematic Review: An Overview
Aromataris, E., & Pearson, A. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000444496.24228.2c · Cited by 598
Self-citationA comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573
Munn, Z., Tufănaru, C., & Aromataris, E. (2014) · AJN American Journal of Nursing · doi:10.1097/01.naj.0000451683.66447.89 · Cited by 543
Self-citationThe handbook of research synthesis and meta-analysis, 2nd ed.
Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009) · Cited by 541
Systematic Reviews: Constructing a Search Strategy and Searching for Evidence
Aromataris, E., & Riitano, D. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000446779.99522.f6 · Cited by 523
Self-citationThe Relative Merits of Risk Ratios and Odds Ratios
Cummings, P. (2009) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpediatrics.2009.31 · Cited by 515
Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change
Cornell, J. E., Mulrow, C. D., Localio, R., Stack, C. B., Meibohm, A. R., Guallar, E., & Goodman, S. N. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2886 · Cited by 453
Pereira, C., & Castilho, E. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwp016 · Cited by 365
A Practical Guide to Cluster Randomised Trials in Health Services Research
Eldridge, S. M., & Kerry, S. (2012) · Wiley eBooks · doi:10.1002/9781119966241 · Cited by 356
Methods for Estimating Adjusted Risk Ratios
Cummings, P. (2009) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900201 · Cited by 326
Analysis of Pretest-Posttest Designs
Bonate, P. L. (2000) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420035926 · Cited by 305
A Quality-Effects Model for Meta-Analysis
Doi, S. A. R., & Thalib, L. (2008) · Epidemiology · doi:10.1097/ede.0b013e31815c24e7 · Cited by 265
Neither fixed nor random: weighted least squares meta‐analysis
Stanley, T. D., & Doucouliagos, H. (2015) · Statistics in Medicine · doi:10.1002/sim.6481 · Cited by 258
Measures of effect size for categorical data.
Fleiss, J. L. (1994) · Cited by 248
Meta‐analysis combining parallel and cross‐over clinical trials. I: Continuous outcomes
Curtin, F., Altman, D. G., & Elbourne, D. R. (2002) · Statistics in Medicine · doi:10.1002/sim.1205 · Cited by 208
Barratt, A. L. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1021197 · Cited by 202
Bateson, M. C. (2004) · PubMed Central · Cited by 199
Hedges, L. V. (1992) · Journal of Educational Statistics · doi:10.3102/10769986017004279 · Cited by 186
Meta-Analysis for Public Management and Policy
Ringquist, E. J., & Anderson, M. R. (2013) · Cited by 171
Shrier, I., Boivin, J.-F., Steele, R. J., Platt, R. W., Furlan, A., Kakuma, R., Brophy, J. M., & Rossignol, M. (2007) · American Journal of Epidemiology · doi:10.1093/aje/kwm189 · Cited by 161
Statistical methods for the meta-analysis of cluster randomization trials
Donner, A. P., Piaggio, G. G., & Villar, J. (2001) · Statistical Methods in Medical Research · doi:10.1177/096228020101000502 · Cited by 159
Meta-Analysis of Published Data Using a Linear Mixed-Effects Model
Stram, D. O. (1996) · Biometrics · doi:10.2307/2532893 · Cited by 155
Meta‐analysis combining parallel and cross‐over clinical trials. II: Binary outcomes
Curtin, F., Elbourne, D. R., & Altman, D. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1206 · Cited by 127
Katikireddi, S. V., Egan, M., & Petticrew, M. (2015) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2014-204711 · Cited by 126
Meta‐analysis combining parallel and cross‐over clinical trials. III: The issue of carry‐over
Curtin, F., Elbourne, D. R., & Altman, D. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1207 · Cited by 120
khalaf, M. M. A., Thalib, L., & Doi, S. A. R. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.01.009 · Cited by 114
Trying to be precise about vagueness.
Senn, S. (2007) · Statistics in Medicine · doi:10.1002/sim.2639 · Cited by 105
Burgess, S. (2003) · Australasian Journal of Paramedicine · doi:10.33151/ajp.1.3.194 · Cited by 104
A proposed method of bias adjustment for meta-analyses of published observational studies
Thompson, S., Ekelund, U., Jebb, S., Lindroos, A. K., Mander, A., Sharp, S., Turner, R., & Wilks, D. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyq248 · Cited by 101
Meta-analysis of heterogeneous clinical trials: An empirical example
Doi, S. A. R., Barendregt, J. J., & Mozurkewich, E. (2010) · Contemporary Clinical Trials · doi:10.1016/j.cct.2010.12.006 · Cited by 92
Confidence intervals for a random‐effects meta‐analysis based on Bartlett‐type corrections
Noma, H. (2011) · Statistics in Medicine · doi:10.1002/sim.4350 · Cited by 81
Meta-analysis using linear mixed models
Sheu, C., & Suzuki, S. (2001) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195354 · Cited by 64
A simple method for inference on an overall effect in meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2007) · Statistics in Medicine · doi:10.1002/sim.2883 · Cited by 59
Potential and limitations of research synthesis
Cooper, H. (1994) · Medical Entomology and Zoology · Cited by 58
8. Statistical Models and Approaches to Inference
Mengersen, K. L., Schmid, C. H., Jennions, M. D., & Gurevitch, J. (2013) · Princeton University Press eBooks · doi:10.1515/9781400846184-010 · Cited by 56
Methods for the bias adjustment of meta‐analyses of published observational studies
Doi, S. A. R., Barendregt, J. J., & Onitilo, A. A. (2012) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2012.01890.x · Cited by 27
An Updated Method for Risk Adjustment in Outcomes Research
Doi, S. A. R., Barendregt, J. J., & Rao, C. (2014) · Value in Health · doi:10.1016/j.jval.2014.05.003 · Cited by 19
Senn, S. J. (1998) · UCL Discovery (University College London) · Cited by 15
Furuya‐Kanamori, L., & Doi, S. A. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.10.010 · Cited by 13
Statistics for Systematic Review Authors
Tufănaru, C., Huang, W. J., Tsay, S., & Chou, S. (2012) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 4
Meta-analysis: formulating, evaluating, combining, and reporting
Normand, S.-L. T. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990215)18:3
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Research works printed but not linked: 3
- Walter, 1998, Odds ratios revisited., Evidence-Based Med, 3, 71
- Olkin, 1998, Odds ratios revisited., Evidence-Based Med, 3, 71
- Altman, 1998, Odds ratios revisited., Evidence-Based Med, 3, 71
Cited by
Sorted by citations
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164
Self-citationQuek, J., Chan, K. E., Wong, Z. Y., Tan, C., Tan, B., Lim, W. H., Tan, D. J. H., Tang, A. S. P., Tay, P., Xiao, J., et al. (2023) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(22)00317-x · Cited by 749
Barker, T. H., Migliavaca, C. B., Stein, C., Colpani, V., Falavigna, M., Aromataris, E., & Munn, Z. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01381-z · Cited by 738
“As with traditional comparative methods of metaanalysis, there are different options in terms of model choice when performing a proportional meta-analysis (discussed above) [39].”
Self-citationMueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
“proach takes clinical as well as statistical considerations into account when deciding on pooling and the choice of the model [31, 38, 41, 44, 47, 61, 116], because “statistical”
BackgroundMethodInfluentialEn Li Cho, E., Ang, C. Z., Quek, J., Fu, C. E., Lim, L. K. E., Heng, Z. E. Q., Tan, D. J. H., Lim, W. H., Yong, J. N., Zeng, R., et al. (2023) · Gut · doi:10.1136/gutjnl-2023-330110 · Cited by 380
Metadata
- Authors
- Tufanaru, C., Munn, Z., Stephenson, M., & Aromataris, E.
- Affiliations
- The University of Adelaide (AU)
- Venue
- International Journal of Evidence-Based Healthcare
- Published
- 2015-09
- Volume
- 13
- Issue
- 3
- Pages
- 196-207
- ISSN
- 1744-1595 (ISSN-L)
- 1744-1609
- OpenAlex
- W2180006680
- Cited by
- 1,057
- References
- 62
- Citations by year
- 2026: 77 · 2025: 151 · 2024: 204 · 2023: 167 · 2022: 157 · 2021: 101 · 2020: 65 · 2019: 57 · 2018: 48 · 2017: 22 · 2016: 8
- FWCI
- 4.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Evidence-Based Healthcare (publishedVersion)
- PubMed (publishedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- fixed-effects meta-analysis, random-effects meta-analysis, systematic review methodology, intervention effectiveness, effect size, research design
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Odds Ratio, Guidelines as Topic, Cross-Over Studies, Evidence-Based Medicine, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
