Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program
Abstract
Objective This article is to establish recommendations for conducting quantitative synthesis, or meta-analysis, using study-level data in comparative effectiveness reviews (CERs) for the Evidence-based Practice Center (EPC) program of the Agency for Healthcare Research and Quality.Study design and setting We focused on recurrent issues in the EPC program and the recommendations were developed using group discussion and consensus based on current knowledge in the literature.Results We first discussed considerations for deciding whether to combine studies, followed by discussions on indirect comparison and incorporation of indirect evidence. Then, we described our recommendations on choosing effect measures and statistical models, giving special attention to combining studies with rare events; and on testing and exploring heterogeneity. Finally, we briefly presented recommendations on combining studies of mixed design and on sensitivity analysis.Conclusion Quantitative synthesis should be conducted in a transparent and consistent way. Inclusion of multiple alternative interventions in CERs increases the complexity of quantitative synthesis, whereas the basic issues in quantitative synthesis remain crucial considerations in quantitative synthesis for a CER. We will cover more issues in future versions and update and improve recommendations with the accumulation of new research to advance the goal for transparency and consistency.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several subgroups, such as intervention durations for ALM, included fewer than six RCTs, which may not have adequate power for detecting differences among subgroups.
“Finally, inadequate statistical power for subgroup analyses was noted. Several subgroups (such as intervention durations for ALM) included a small number of RCTs (less than six), which may not have adequate power for detecting differences among subgroups [71,72]; the results of such subgroup analyses should be cautiously interpreted.”
Several subgroups (such as participant types for lean body mass) included a small number of RCTs less than six, which may not have adequate power for detecting a difference among subgroups; the results of such subgroup analyses should be cautiously interpreted.
“Several subgroups (such as participant types for lean body mass) included a small number of RCTs less than six, which may not have adequate power for detecting a difference among subgroups [89,90]; the results of such subgroup analyses should be cautiously interpreted.”
From Effects of Protein Supplementation Combined with Exercise Intervention on Frailty Indices, Body Composition, and Physical Function in Frail Older Adults (2018) · cites this paper as [89]
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Practical methods for incorporating summary time-to-event data into meta-analysis.
Tierney, J. F., Stewart, L. A., Ghersi, D., Burdett, S., & Sydes, M. R. (2007) · Trials · doi:10.1186/1745-6215-8-16 · Cited by 6,208 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675 · see citing works
Parmar, M. K. B., Torri, V., & Stewart, L. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981230)17:24 · Cited by 4,519 · see citing works
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · see citing works
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374 · see citing works
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347 · see citing works
Combination of direct and indirect evidence in mixed treatment comparisons
Lu, G., & Ades, A. E. (2004) · Statistics in Medicine · doi:10.1002/sim.1875 · Cited by 2,149 · see citing works
Bucher, H. C., Guyatt, G. H., Griffith, L. E., & Walter, S. D. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00049-8 · Cited by 2,052 · see citing works
Self-citationThe number needed to treat: a clinically useful measure of treatment effect
Cook, R. J., & Sackett, D. L. (1995) · BMJ · doi:10.1136/bmj.310.6977.452 · Cited by 1,885 · see citing works
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734 · see citing works
Simultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600 · see citing works
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Controlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 1× · see citing works
“Use of permutation test for meta-regression can help assess the level of statistical significance of an observed meta-regression finding [52].”
MethodUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · see citing works
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206 · see citing works
Tubach, F., Ravaud, P., Baron, G., Falissard, B., Logeart, I., Bellamy, N., Bombardier, C., Felson, D. T., Hochberg, M., van der Heijde, D. M. F. M., et al. (2004) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2004.022905 · Cited by 1,095 · see citing works
Song, F., Altman, D. G., Glenny, A., & Deeks, J. J. (2003) · BMJ · doi:10.1136/bmj.326.7387.472 · Cited by 944 · see citing works
Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938 · see citing works
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919 · see citing works
Assessing Evidence Inconsistency in Mixed Treatment Comparisons
Lu, G., & Ades, A. E. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001302 · Cited by 756 · see citing works
Indirect comparisons of competing interventions
Glenny, A., Altman, D. G., Song, F., Sakarovitch, C., Deeks, J. J., D’Amico, R., Bradburn, M. J., & Eastwood, A. J. (2005) · Health Technology Assessment · doi:10.3310/hta9260 · Cited by 682 · see citing works
Deeks, J. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1188 · Cited by 660 · see citing works
Summing up evidence: one answer is not always enough
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08468-7 · Cited by 646 · see citing works
Self-citationA comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573 · see citing works
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568 · see citing works
Lambert, P. C., Sutton, A. J., Burton, P. R., Abrams, K. R., & Jones, D. R. (2005) · Statistics in Medicine · doi:10.1002/sim.2112 · Cited by 511 · see citing works
Exact logistic regression: Theory and examples
Mehta, C. R., & Patel, N. R. (1995) · Statistics in Medicine · doi:10.1002/sim.4780141908 · Cited by 484 · see citing works
Friedrich, J. O., Adhikari, N. K. J., & Beyene, J. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-5 · Cited by 460 · see citing works
A comparison of observational studies and randomized, controlled trials
Benson, K., & Hartz, A. J. (2000) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(00)00754-6 · Cited by 456 · see citing works
The binomial distribution of meta-analysis was preferred to model within-study variability
Hamza, T. H., van Houwelingen, H. C., & Stijnen, T. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.016 · Cited by 388 · see citing works
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care
Lau, J., Schmid, C. H., & Chalmers, T. C. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00106-z · Cited by 358 · see citing works
Self-citationUncertain Effects of Rosiglitazone on the Risk for Myocardial Infarction and Cardiovascular Death
Diamond, G. A., Bax, L., & Kaul, S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00182 · Cited by 288 · see citing works
Schmid, C. H., Stark, P. C., Berlin, J. A., Landais, P., & Lau, J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.001 · Cited by 245 · see citing works
Self-citationSchmid, C. H., Lau, J., McIntosh, M. W., & Cappelleri, J. C. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980915)17:17 · Cited by 210 · see citing works
Self-citationAn assessment of the use of the continuity correction for sparse data in meta-analysis
Sankey, S. S., Weissfeld, L. A., Fine, M. J., & Kapoor, W. N. (1996) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919608813357 · Cited by 123 · see citing works
Invited Commentary: Re: “A Critical Look at Some Popular Meta-Analytic Methods”
Olkin, I. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117249 · Cited by 117 · see citing works
Baker, S. G., & Kramer, B. S. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-13 · Cited by 114 · see citing works
Shuster, J. J., Jones, L. S., & Salmon, D. A. (2007) · Statistics in Medicine · doi:10.1002/sim.3060 · Cited by 105 · see citing works
Chou, R., Fu, R. F., Huffman, L. H., & Korthuis, P. T. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69638-4 · Cited by 104 · see citing works
Self-citationIndirect comparisons: the mesh and mess of clinical trials
Ioannidis, J. P. A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69615-3 · Cited by 94 · see citing works
Song, F., Glenny, A., & Altman, D. G. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00055-6 · Cited by 78 · see citing works
Droitcour, J. A., Silberman, G., & Chelimsky, E. (1993) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300004694 · Cited by 76 · see citing works
Meta-Analysis of Migraine Headache Treatments: Combining Information from Heterogeneous Designs
Dominici, F., Parmigiani, G., Wolpert, R. L., & Hasselblad, V. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473815 · Cited by 75 · see citing works
Schulzer, M., & Mancini, G. B. J. (1996) · International Journal of Epidemiology · doi:10.1093/ije/25.4.704 · Cited by 75 · see citing works
Dahabreh, I. J. (2008) · Clinical Trials · doi:10.1177/1740774508090212 · Cited by 61 · see citing works
AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program
Slutsky, J. R., Atkins, D. C., Chang, S., & Sharp, B. A. C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.009 · Cited by 49 · see citing works
AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program
Helfand, M., & Balshem, H. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.05.005 · Cited by 47 · see citing works
Duchateau, L., Collette, L., Sylvester, R. J., & Pignon, J. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00886.x · Cited by 35 · see citing works
Meta-Analysis of Migraine Headache Treatments: Combining Information from Heterogeneous Designs
Dominici, F., Parmigiani, G., Wolpert, R. L., & Hasselblad, V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669674 · Cited by 18 · see citing works
The exact analysis of contingency tables in medical research
Mehta, C. R. (1995) · Cancer treatment and research · doi:10.1007/978-1-4615-2009-2_9 · Cited by 15 · see citing works
Engels, E. (2000) · doi:10.1002/1097-0258(20000715)19:13 · Cited by 2 · see citing works
Self-citationHahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8
Flagged first, then by citations
2 of the 62 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Racism as a Determinant of Health: A Systematic Review and Meta-Analysis
Paradies, Y., Ben, J., Denson, N., Elias, A., Priest, N., Pieterse, A., Gupta, A., Kelaher, M., & Gee, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138511 · Cited by 2,610 · see citing works
“This threshold was used in prior meta-analyses, and based on minimum thresholds established in the literature [60, 61].”
MethodMindfulness-based interventions for psychiatric disorders: A systematic review and meta-analysis
Goldberg, S. B., Tucker, R. P., Greene, P. A., Davidson, R. J., Wampold, B. E., Kearney, D. J., & Simpson, T. L. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.011 · Cited by 1,145 · see citing works
Meta-analyses of factors motivating climate change adaptation behaviour
van Valkengoed, A. M., & Steg, L. (2019) · Nature Climate Change · doi:10.1038/s41558-018-0371-y · Cited by 940 · see citing works
Global prevalence of gaming disorder: A systematic review and meta-analysis
Stevens, M. W., Dorstyn, D., Delfabbro, P. H., & King, D. L. (2021) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867420962851 · Cited by 863 · see citing works
Gastrointestinal Symptoms in Autism Spectrum Disorder: A Meta-analysis
McElhanon, B. O., McCracken, C., Karpen, S., & Sharp, W. G. (2014) · Pediatrics · doi:10.1542/peds.2013-3995 · Cited by 831 · see citing works
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01117-5 · Cited by 793 · see citing works
Hoaglin, D. C., Hawkins, N., Jansen, J. P., Scott, D. A., Itzler, R. F., Cappelleri, J. C., Boersma, C., Thompson, D., Larholt, K. M., Díaz, M., et al. (2011) · Value in Health · doi:10.1016/j.jval.2011.01.011 · Cited by 771 · see citing works
Bai, S., Hew, K. F., & Huang, B. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100322 · Cited by 689 · see citing works
Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022) · Perspectives on Psychological Science · doi:10.1177/1745691620968771 · Cited by 520 · see citing works
“To strengthen the precision of our results, we adopted the Agency for Healthcare Research and Quality recommendation of requiring at least four primary studies (Fu et al., 2011).”
MethodWong, Y. J., Ho, M.-H. R., Wang, S.-Y., & Miller, I. S. K. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000176 · Cited by 507 · see citing works
Henderson, J. T., Whitlock, E. P., O’Connor, E., Senger, C. A., Thompson, J. H., & Rowland, M. G. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2844 · Cited by 500 · see citing works
Wang, M.-T., L. Degol, J., Amemiya, J., Parr, A., & Guo, J. (2020) · Developmental Review · doi:10.1016/j.dr.2020.100912 · Cited by 470 · see citing works
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Ansari, M., Berkman, N., Chang, S. M., Hartling, L., McPheeters, M., Santaguida, P., Shamliyan, T., Singh, K., Tsertsvadze, A., et al. (2012) · doi:10.23970/ahrqepcmethguide2 · Cited by 731 · see citing works
Fortmann, S. P., Burda, B. U., Senger, C. A., Lin, J. S., & Whitlock, E. P. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-159-12-201312170-00729 · Cited by 442 · see citing works
Determinants of take-over time from automated driving: A meta-analysis of 129 studies
Zhang, B., de Winter, J., Varotto, S. F., Happee, R., & Martens, M. (2019) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2019.04.020 · Cited by 437 · see citing works
Donald, J. N., Sahdra, B. K., Van Zanden, B., Duineveld, J. J., Atkins, P. W. B., Marshall, S. L., & Ciarrochi, J. (2019) · British Journal of Psychology · doi:10.1111/bjop.12338 · Cited by 371 · see citing works
Byrne, A. W., McEvoy, D., Collins, Á. B., Hunt, K., Casey-Bryars, M., Barber, A. M., Butler, F., Griffin, J., Lane, E. A., McAloon, C. G., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-039856 · Cited by 374 · see citing works
Ericsson, K. A., & Harwell, K. W. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02396 · Cited by 370 · see citing works
“…Macnamara et al. (2014) found significant moderator effects for the domain of performance, but due to the low representation of effects from some of the performance domains we could not replicate this categorical moderator analysis with our present selection of effects (see Fu et al., 2011).”
BackgroundHow Sedentary Are University Students? A Systematic Review and Meta-Analysis
Castro, Ó., Bennie, J. A., Vergeer, I., Bosselut, G., & Biddle, S. (2020) · Prevention Science · doi:10.1007/s11121-020-01093-8 · Cited by 392 · see citing works
Prevalence and factors associated with polypharmacy: a systematic review and meta-analysis
Delara, M., Murray, L., Jafari, B., Bahji, A., Goodarzi, Z., Kirkham, J. G., Chowdhury, M., & Seitz, D. P. (2022) · BMC Geriatrics · doi:10.1186/s12877-022-03279-x · Cited by 348 · see citing works
Galante, J., Friedrich, C., Dawson, A. F., Modrego-Alarcón, M., Gebbing, P., Delgado-Suárez, I., Gupta, R., Dean, L., Dalgleish, T., White, I. R., et al. (2021) · PLOS Medicine · doi:10.1371/journal.pmed.1003481 · Cited by 319 · see citing works
“We used random–effects multivariable meta–regression within multivariate meta–analyses and interpreted with caution analyses of categorical subgroup variables with fewer than 5 studies per category [71].”
MethodSharp, W. G., Volkert, V. M., Scahill, L. D., McCracken, C., & McElhanon, B. O. (2016) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2016.10.002 · Cited by 321 · see citing works
Effects of digital game-based STEM education on students’ learning achievement: a meta-analysis
Wang, L.-H., Chen, B., Hwang, G.-J., Guan, J.-Q., & Wang, Y.-Q. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00344-0 · Cited by 316 · see citing works
“Students with different knowledge levels would affect the experiment’s results, leading to effect size heterogeneity (Fu et al., 2011).”
BackgroundHooren, B. V., FULLER, J. T., Buckley, J. D., Miller, J. R., Sewell, K. B., Rao, G., Barton, C. J., Bishop, C., & Willy, R. W. (2019) · Sports Medicine · doi:10.1007/s40279-019-01237-z · Cited by 306 · see citing works
Cognitive–behavioral therapy for Internet gaming disorder: A systematic review and meta‐analysis
Stevens, M. W. R., King, D. L., Dorstyn, D., & Delfabbro, P. H. (2019) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2341 · Cited by 295 · see citing works
The efficacy of metacognitive therapy: A systematic review and meta-analysis
Normann, N., & Morina, N. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02211 · Cited by 289 · see citing works
Grading the strength of a body of evidence when assessing health care interventions: an EPC update
Berkman, N. D., Lohr, K. N., Ansari, M. T., Balk, E. M., Kane, R., McDonagh, M., Morton, S. C., Viswanathan, M., Bass, E. B., Butler, M., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.023 · Cited by 279 · see citing works
Self-citationSherman, D. S., Mauser, J., Nuno, M. A., & Sherzai, D. (2017) · Neuropsychology Review · doi:10.1007/s11065-017-9363-3 · Cited by 279 · see citing works
von Bartheld, C. S., Hagen, M. M., & Butowt, R. (2020) · ACS Chemical Neuroscience · doi:10.1021/acschemneuro.0c00460 · Cited by 283 · see citing works
Effects of Exercise on Falls, Balance, and Gait Ability in Parkinson’s Disease
Shen, X., Wong-Yu, I. S. K., & Mak, K. Y. (2015) · Neurorehabilitation and neural repair · doi:10.1177/1545968315613447 · Cited by 278 · see citing works
Pellicane, M. J., & Ciesla, J. A. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102113 · Cited by 281 · see citing works
Trott, M., Driscoll, R., Iraldo, E., & Pardhan, S. (2022) · eClinicalMedicine · doi:10.1016/j.eclinm.2022.101452 · Cited by 266 · see citing works
Guirguis‐Blake, J., Evans, C. V., Senger, C. A., O’Connor, E., & Whitlock, E. P. (2016) · Annals of Internal Medicine · doi:10.7326/m15-2113 · Cited by 260 · see citing works
Acne vulgaris and risk of depression and anxiety: A meta-analytic review
Samuels, D. V., Rosenthal, R., Lin, R., Chaudhari, S. P., & Natsuaki, M. N. (2020) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2020.02.040 · Cited by 260 · see citing works
Marriage and risk of dementia: systematic review and meta-analysis of observational studies
Sommerlad, A., Ruegger, J., Singh-Manoux, A., Lewis, G., & Livingston, G. (2018) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2017-316274 · Cited by 252 · see citing works
Goldberg, S. B., Lam, S. U., Simonsson, O., Torous, J., & Sun, S. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000002 · Cited by 237 · see citing works
“, psychiatric symptoms, stress, quality of life, addictive behaviors that are included in the DSM-5 [33] such as alcohol and tobacco use) (3) based on a minimum of four randomized controlled trials (RCTs) [34] (4) using comparison conditions that could be categorized as inactive, active, non-specific, specific, or adj…”
BackgroundHowes, O. D., Whitehurst, T., Shatalina, E., Townsend, L., Onwordi, E. C., Mak, T. L. A., Arumuham, A., O’Brien, O., Lobo, M., Vano, L., et al. (2021) · World Psychiatry · doi:10.1002/wps.20822 · Cited by 243 · see citing works
Türk, F., & Waller, G. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101856 · Cited by 257 · see citing works
Hospitalisation rates and predictors in people with dementia: a systematic review and meta-analysis
Shepherd, H., Livingston, G., Chan, J., & Sommerlad, A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1369-7 · Cited by 232 · see citing works
Job burnout in mental health providers: A meta-analysis of 35 years of intervention research
Dreison, K. C., Luther, L., Bonfils, K. A., Sliter, M. T., McGrew, J. H., & Salyers, M. P. (2018) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000047 · Cited by 244 · see citing works
Major, L., Francis, G. A., & Tsapali, M. (2021) · British Journal of Educational Technology · doi:10.1111/bjet.13116 · Cited by 318 · see citing works
Xu, S., Liu, L., & Sayer, E. J. (2013) · Biogeosciences · doi:10.5194/bg-10-7423-2013 · Cited by 239 · see citing works
A meta‐analysis of problem gambling risk factors in the general adult population
Allami, Y., Hodgins, D. C., Young, M. M., Brunelle, N., Currie, S. R., Dufour, M., Flores‐Pajot, M., & Nadeau, L. (2021) · Addiction · doi:10.1111/add.15449 · Cited by 254 · see citing works
Burgdorf, V., Szabó, M., & Abbott, M. J. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01336 · Cited by 234 · see citing works
Jenkinson, P. M., Taylor, L., & Laws, K. R. (2018) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2018.04.005 · Cited by 227 · see citing works
Whitlock, E. P., Burda, B. U., Williams, S. B., Guirguis‐Blake, J., & Evans, C. V. (2016) · Annals of Internal Medicine · doi:10.7326/m15-2112 · Cited by 228 · see citing works
Selvaraj, S., Arnone, D., Cappai, A., & Howes, O. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.06.005 · Cited by 223 · see citing works
Recommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221 · see citing works
Kallitsounaki, A., & Williams, D. M. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-022-05517-y · Cited by 235 · see citing works
Kuss, O. (2015) · Statistics in Medicine · doi:10.1002/sim.6383 · Cited by 206 · see citing works
Çakici, N., van Beveren, N. J. M., Judge-Hundal, G., Koola, M. M., & Sommer, I. E. (2019) · Psychological Medicine · doi:10.1017/s0033291719001995 · Cited by 215 · see citing works
Li, L., Hew, K. F., & Du, J. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-023-10337-7 · Cited by 385 · see citing works
Chiavaroli, L., Lee, D., Ahmed, A., Cheung, A., Khan, T. A., Blanco, S., Mejia, Mirrahimi, A., Jenkins, D. J. A., Livesey, G., et al. (2021) · BMJ · doi:10.1136/bmj.n1651 · Cited by 215 · see citing works
Gartlehner, G., Wagner, G., Matyas, N., Titscher, V., Greimel, J., Lux, L., Gaynes, B. N., Viswanathan, M., Patel, S., & Lohr, K. N. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014912 · Cited by 192 · see citing works
Self-citationA meta-analysis of factors related to recycling
Geiger, J. L., Steg, L., van der Werff, E., & Ünal, A. B. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.05.004 · Cited by 199 · see citing works
Stubbs, B., Vancampfort, D., Veronese, N., Solmi, M., Gaughran, F., Manu, P., Rosenbaum, S., Binder, M. D., & Fornaro, M. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.02.060 · Cited by 195 · see citing works
Liao, C.-D., Chen, H.-C., Huang, S.-W., & Liou, T.-H. (2019) · Nutrients · doi:10.3390/nu11081713 · Cited by 189 · see citing works
Chou, R., Selph, S., Buckley, D., Fu, R. F., Griffin, J. C., Grusing, S., & Gore, J. L. (2016) · The Journal of Urology · doi:10.1016/j.juro.2016.12.090 · Cited by 185 · see citing works
Self-citationWolfowicz, M., Litmanovitz, Y., Weisburd, D. L., & Hasisi, B. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09439-4 · Cited by 184 · see citing works
Esteban-Figuerola, P., Canals, J., Fernández-Cao, J. C., & Arija Val, V. (2019) · Autism · doi:10.1177/1362361318794179 · Cited by 181 · see citing works
Goldberg, S. B., Tucker, R. P., Greene, P. A., Davidson, R. J., Kearney, D. J., & Simpson, T. L. (2019) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1556330 · Cited by 171 · see citing works
Profiling inflammatory signatures of schizophrenia: A cross-sectional and meta-analysis study
Frydecka, D., Krzystek‐Korpacka, M., Lubeiro, A., Stramecki, F., Stańczykiewicz, B., Beszłej, J. A., Piotrowski, P., Kotowicz, K., Szewczuk-Bogusławska, M., Pawlak, E., et al. (2018) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2018.05.002 · Cited by 179 · see citing works
Akhtar, P. S., Ma, L., Waqas, A., Naveed, S., Li, Y., Rahman, A., & Wang, Y. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.03.183 · Cited by 176 · see citing works
Cepoiu-Martin, M., Tam-Tham, H., Patten, S., Maxwell, C. J., & Hogan, D. B. (2016) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4449 · Cited by 175 · see citing works
Strohmaier, S. (2020) · Mindfulness · doi:10.1007/s12671-020-01319-4 · Cited by 163 · see citing works
To Flip or Not to Flip? A Meta-Analysis of the Efficacy of Flipped Learning in Higher Education
Bredow, C. A., Roehling, P. V., Knorp, A. J., & Sweet, A. M. (2021) · Review of Educational Research · doi:10.3102/00346543211019122 · Cited by 227 · see citing works
Aminuddin, H. B., Jiao, N., Jiang, Y., Hong, J., & Wang, W. (2021) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2019.02.003 · Cited by 157 · see citing works
Brazier, J., Ara, R., Azzabi, I., Busschbach, J., Chevrou-séverac, H., Crawford, B., Cruz, L., Karnon, J., Lloyd, A., Paisley, S., et al. (2019) · Value in Health · doi:10.1016/j.jval.2019.01.004 · Cited by 146 · see citing works
Iglay, K., Cartier, S. E., Rosen, V. M., Zarotsky, V., Rajpathak, S. N., Radican, L., & Tunceli, K. (2015) · Current Medical Research and Opinion · doi:10.1185/03007995.2015.1053048 · Cited by 157 · see citing works
Chubak, J., Whitlock, E. P., Williams, S. B., Kamineni, A., Burda, B. U., Buist, D. S. M., & Anderson, M. L. (2016) · Annals of Internal Medicine · doi:10.7326/m15-2117 · Cited by 160 · see citing works
Assessment of Anhedonia in Adults With and Without Mental Illness
Trøstheim, M., Eikemo, M., Meir, R., Hansen, I., Paul, E. R., Kroll, S. L., Garland, E. L., & Leknes, S. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.13233 · Cited by 160 · see citing works
Florean, I. S., Dobrean, A., Poetar, C.-R., Georgescu, R., & Milea, I. (2020) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-020-00326-0 · Cited by 159 · see citing works
Intergroup contact and mental health stigma: a comparative effectiveness meta-analysis
Maunder, R. D., & White, F. A. (2019) · Clinical Psychology Review · doi:10.1016/j.cpr.2019.101749 · Cited by 152 · see citing works
Mukadam, N., Sommerlad, A., & Livingston, G. (2017) · Journal of Alzheimer's Disease · doi:10.3233/jad-170131 · Cited by 86 · see citing works
Physical Activity and School Engagement in Youth: A Systematic Review and Meta-Analysis
Owen, K. B., Parker, P. D., Zanden, B. E. V., MacMillan, F., Astell‐Burt, T., & Lonsdale, C. (2016) · Educational Psychologist · doi:10.1080/00461520.2016.1151793 · Cited by 158 · see citing works
Carron, M., Zarantonello, F., Tellaroli, P., & Ori, C. (2016) · Journal of Clinical Anesthesia · doi:10.1016/j.jclinane.2016.06.018 · Cited by 150 · see citing works
Brasure, M., Fuchs, E. L., MacDonald, R. H., Nelson, V. A., Koffel, E., Olson, C. M., Khawaja, I. S., Diem, S. J., Carlyle, M. H., Wilt, T. J., et al. (2016) · Annals of Internal Medicine · doi:10.7326/m15-1782 · Cited by 148 · see citing works
Self-citationNg, Y.-L., Ma, F., Ho, F. K. W., Ip, P., & Fu, K. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.026 · Cited by 148 · see citing works
Nguyen, M., Jarvis, S. E., Tinajero, M., Yu, J., Chiavaroli, L., Mejía, S. B., Khan, T. A., Tobias, D. K., Willett, W. C., Hu, F. B., et al. (2022) · American Journal of Clinical Nutrition · doi:10.1016/j.ajcnut.2022.11.008 · Cited by 159 · see citing works
Internalization of body shape ideals and body dissatisfaction: A systematic review and meta‐analysis
Paterna, A., Alcaraz‐Ibáñez, M., Fuller‐Tyszkiewicz, M., & Sicilia, Á. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23568 · Cited by 157 · see citing works
Simmons, S. M., Hicks, A., & Caird, J. K. (2015) · Accident Analysis & Prevention · doi:10.1016/j.aap.2015.11.015 · Cited by 143 · see citing works
Sex differences in neuropsychiatric symptoms in Alzheimer’s disease dementia: a meta-analysis
Eikelboom, W., Pan, M., Ossenkoppele, R., Coesmans, M., Gatchel, J., Ismail, Z., Lanctôt, K., Fischer, C. E., Mortby, M., van den Berg, E., et al. (2021) · Alzheimer's Research & Therapy · doi:10.1186/s13195-022-00991-z · Cited by 126 · see citing works
Selph, S., Dana, T., Blazina, I., Bougatsos, C., Patel, H., & Chou, R. (2015) · Annals of Internal Medicine · doi:10.7326/m14-2221 · Cited by 134 · see citing works
Lee, B. E. C., Ling, M., Boyd, L., Olsson, C., & Sheen, J. (2023) · Journal of Affective Disorders · doi:10.1016/j.jad.2023.03.012 · Cited by 120 · see citing works
“Subgroup analyses by study level characteristics Subgroup analyses were conducted on overall pooled prevalence rates when there were four or more studies (Fu et al., 2011), with test of subgroup differences using Q tests for categorical predictors (i.”
MethodAyala, S., Stubbs, B., Gomez, L., Moore, S., Gaughran, F., Flanagan, R. J., MacCabe, J. H., & Lally, J. (2016) · International Journal of Molecular Sciences · doi:10.3390/ijms17060863 · Cited by 134 · see citing works
Duan, Y., Shang, B., Liang, W., Du, G., Yang, M., & Rhodes, R. E. (2021) · Journal of Medical Internet Research · doi:10.2196/23786 · Cited by 130 · see citing works
Clarke, K., King, M., & Prost, A. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001541 · Cited by 122 · see citing works
Waqas, A., Koukab, A., Meraj, H., Dua, T., Chowdhary, N., Fatima, B., & Rahman, A. (2022) · BMC Psychiatry · doi:10.1186/s12888-022-03694-9 · Cited by 138 · see citing works
Stafford, J., Chung, W. T., Sommerlad, A., Kirkbride, J. B., & Howard, R. (2022) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5711 · Cited by 126 · see citing works
Kwee, R. M., & Kwee, T. C. (2019) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.04.023 · Cited by 124 · see citing works
Wang, L., Başer, O., Kutikova, L., Page, J. H., & Barron, R. L. (2015) · Supportive Care in Cancer · doi:10.1007/s00520-015-2686-9 · Cited by 123 · see citing works
Vandenberg, L. N., Ågerstrand, M., Beronius, A., Beausoleil, C., Bergman, A., Bero, L., Bornehag, C.-G., Boyer, C. S., Cooper, G., Cotgreave, I., et al. (2016) · Environmental Health · doi:10.1186/s12940-016-0156-6 · Cited by 113 · see citing works
Jukić, I., García‐Ramos, A., Helms, E. R., McGuigan, M. R., & Tufano, J. J. (2020) · Sports Medicine · doi:10.1007/s40279-020-01344-2 · Cited by 127 · see citing works
The development of parental monitoring during adolescence: A meta-analysis
Lionetti, F., Palladino, B. E., Passini, C. M., Casonato, M., Hamzallari, O., Ranta, M., Dellagiulia, A., & Keijsers, L. (2018) · European Journal of Developmental Psychology · doi:10.1080/17405629.2018.1476233 · Cited by 122 · see citing works
Identifying suicide risk among college students: A systematic review
Li, W., Dorstyn, D., & Jarmon, E. (2019) · Death Studies · doi:10.1080/07481187.2019.1578305 · Cited by 121 · see citing works
Synthesizing Evidence from Diagnostic Accuracy TEsts : the SEDATE guideline
Sotiriadis, A., Papatheodorou, S., & Martins, W. P. (2015) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.15762 · Cited by 119 · see citing works
Kuppens, S., Moore, S. C., Gross, V., Lowthian, E., & Siddaway, A. P. (2020) · Development and Psychopathology · doi:10.1017/s0954579419000749 · Cited by 119 · see citing works
Michaud, T. L., Rho, Y. H., Shamliyan, T., Kuntz, K. M., & Choi, H. K. (2014) · The American Journal of Medicine · doi:10.1016/j.amjmed.2014.06.012 · Cited by 117 · see citing works
Self-citationRoeske, M. J., Konradi, C. L., Heckers, S., & Lewis, A. S. (2020) · Molecular Psychiatry · doi:10.1038/s41380-020-0853-y · Cited by 116 · see citing works
Kennedy, R. S. (2020) · Aggression and Violent Behavior · doi:10.1016/j.avb.2020.101485 · Cited by 128 · see citing works
- Authors
- Fu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T. J., Griffith, L., Oremus, M., Raina, P., Ismaila, A., Santaguida, P., Lau, J., & Trikalinos, T. A.
- Affiliations
- Oregon Health & Science University (US)
- Universität für Weiterbildung Krems (AT)
- Blue Cross Blue Shield Association (US)
- University of Minnesota (US)
- Agency for Healthcare Research and Quality (US)
- McMaster University (CA)
- Tufts Medical Center (US)
- Corresponding author
- Fu, Rongwei
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-11
- Volume
- 64
- Issue
- 11
- Pages
- 1187-1197
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2145171366
- Cited by
- 730
- References
- 62
- Citations by year
- 2026: 46 · 2025: 64 · 2024: 88 · 2023: 75 · 2022: 99 · 2021: 92 · 2020: 56 · 2019: 50 · 2018: 29 · 2017: 27 · 2016: 33 · 2015: 26 · 2014: 18 · 2013: 13 · 2012: 10
- FWCI
- 14.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Agency for Healthcare Research and Quality
- H2020 European Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- meta-analysis, comparative effectiveness research, evidence-based practice centers, Agency for Healthcare Research and Quality, indirect comparison, effect measures, rare events, heterogeneity, sensitivity analysis
- MeSH terms
- Data Interpretation, Statistical, Humans, Review Literature as Topic, Statistics as Topic, United States, Meta-Analysis as Topic, Models, Statistical, Bias, United States Agency for Healthcare Research and Quality, Outcome Assessment, Health Care, Evidence-Based Medicine, Consensus, Comparative Effectiveness Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
