The DOI matches its Crossref record; 60 of the 62 references it cites were found in a registry or an index.
Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program
Abstract
Objective This article is to establish recommendations for conducting quantitative synthesis, or meta-analysis, using study-level data in comparative effectiveness reviews (CERs) for the Evidence-based Practice Center (EPC) program of the Agency for Healthcare Research and Quality.Study design and setting We focused on recurrent issues in the EPC program and the recommendations were developed using group discussion and consensus based on current knowledge in the literature.Results We first discussed considerations for deciding whether to combine studies, followed by discussions on indirect comparison and incorporation of indirect evidence. Then, we described our recommendations on choosing effect measures and statistical models, giving special attention to combining studies with rare events; and on testing and exploring heterogeneity. Finally, we briefly presented recommendations on combining studies of mixed design and on sensitivity analysis.Conclusion Quantitative synthesis should be conducted in a transparent and consistent way. Inclusion of multiple alternative interventions in CERs increases the complexity of quantitative synthesis, whereas the basic issues in quantitative synthesis remain crucial considerations in quantitative synthesis for a CER. We will cover more issues in future versions and update and improve recommendations with the accumulation of new research to advance the goal for transparency and consistency.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, inadequate statistical power for subgroup analyses was noted. Several subgroups (such as intervention durations for ALM) included a small number of RCTs (less than six), which may not have adequate power for detecting differences among subgroups [71,72]; the results of such subgroup analyses should be cautiously interpreted.”
In the citing paper
“Several subgroups (such as participant types for lean body mass) included a small number of RCTs less than six, which may not have adequate power for detecting a difference among subgroups [89,90]; the results of such subgroup analyses should be cautiously interpreted.”
From Effects of Protein Supplementation Combined with Exercise Intervention on Frailty Indices, Body Composition, and Physical Function in Frail Older Adults (2018) · cites this paper as [89]
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Practical methods for incorporating summary time-to-event data into meta-analysis.
Tierney, J. F., Stewart, L. A., Ghersi, D., Burdett, S., & Sydes, M. R. (2007) · Trials · doi:10.1186/1745-6215-8-16 · Cited by 6,208
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Nissen, S. E., & Wolski, K. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa072761 · Cited by 4,675
Parmar, M. K. B., Torri, V., & Stewart, L. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19981230)17:24 · Cited by 4,519
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563
Random-effects model for meta-analysis of clinical trials: An update
DerSimonian, R., & Kacker, R. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.04.004 · Cited by 2,374
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347
Combination of direct and indirect evidence in mixed treatment comparisons
Lu, G., & Ades, A. E. (2004) · Statistics in Medicine · doi:10.1002/sim.1875 · Cited by 2,172
Bucher, H. C., Guyatt, G. H., Griffith, L. E., & Walter, S. D. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00049-8 · Cited by 2,052
Self-citationThe number needed to treat: a clinically useful measure of treatment effect
Cook, R. J., & Sackett, D. L. (1995) · BMJ · doi:10.1136/bmj.310.6977.452 · Cited by 1,885
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734
Simultaneous comparison of multiple treatments: Combining direct and indirect evidence.
Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005) · BMJ · doi:10.1136/bmj.331.7521.897 · Cited by 1,600
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Controlling the risk of spurious findings from meta-regression
Higgins, J. P. T., & Thompson, S. G. (2004) · Statistics in Medicine · doi:10.1002/sim.1752 · Cited by 1,267 · cited 1×
“Use of permutation test for meta-regression can help assess the level of statistical significance of an observed meta-regression finding [52].”
MethodUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Network meta‐analysis for indirect treatment comparisons
Lumley, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1201 · Cited by 1,206
Tubach, F., Ravaud, P., Baron, G., Falissard, B., Logeart, I., Bellamy, N., Bombardier, C., Felson, D. T., Hochberg, M., van der Heijde, D. M. F. M., et al. (2004) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2004.022905 · Cited by 1,095
Song, F., Altman, D. G., Glenny, A., & Deeks, J. J. (2003) · BMJ · doi:10.1136/bmj.326.7387.472 · Cited by 947
Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events
Bradburn, M. J., Deeks, J. J., Berlin, J. A., & Russell Localio, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2528 · Cited by 938
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919
Assessing Evidence Inconsistency in Mixed Treatment Comparisons
Lu, G., & Ades, A. E. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001302 · Cited by 756
Indirect comparisons of competing interventions
Glenny, A., Altman, D. G., Song, F., Sakarovitch, C., Deeks, J. J., D’Amico, R., Bradburn, M. J., & Eastwood, A. J. (2005) · Health Technology Assessment · doi:10.3310/hta9260 · Cited by 681
Deeks, J. J. (2002) · Statistics in Medicine · doi:10.1002/sim.1188 · Cited by 660
Summing up evidence: one answer is not always enough
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08468-7 · Cited by 655
Self-citationA comparison of statistical methods for meta‐analysis
Brockwell, S. E., & Gordon, I. R. (2001) · Statistics in Medicine · doi:10.1002/sim.650 · Cited by 573
Bayesian approaches to random-effects meta-analysis: a comparative study
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995) · Statistics in Medicine · doi:10.1002/sim.4780142408 · Cited by 568
Lambert, P. C., Sutton, A. J., Burton, P. R., Abrams, K. R., & Jones, D. R. (2005) · Statistics in Medicine · doi:10.1002/sim.2112 · Cited by 511
Exact logistic regression: Theory and examples
Mehta, C. R., & Patel, N. R. (1995) · Statistics in Medicine · doi:10.1002/sim.4780141908 · Cited by 484
Friedrich, J. O., Adhikari, N. K. J., & Beyene, J. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-5 · Cited by 460
A comparison of observational studies and randomized, controlled trials
Benson, K., & Hartz, A. J. (2000) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(00)00754-6 · Cited by 456
The binomial distribution of meta-analysis was preferred to model within-study variability
Hamza, T. H., van Houwelingen, H. C., & Stijnen, T. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.016 · Cited by 388
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care
Lau, J., Schmid, C. H., & Chalmers, T. C. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00106-z · Cited by 350
Self-citationUncertain Effects of Rosiglitazone on the Risk for Myocardial Infarction and Cardiovascular Death
Diamond, G. A., Bax, L., & Kaul, S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00182 · Cited by 288
Schmid, C. H., Stark, P. C., Berlin, J. A., Landais, P., & Lau, J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.001 · Cited by 245
Self-citationSchmid, C. H., Lau, J., McIntosh, M. W., & Cappelleri, J. C. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980915)17:17 · Cited by 211
Self-citationAn assessment of the use of the continuity correction for sparse data in meta-analysis
Sankey, S. S., Weissfeld, L. A., Fine, M. J., & Kapoor, W. N. (1996) · Communications in Statistics - Simulation and Computation · doi:10.1080/03610919608813357 · Cited by 123
Invited Commentary: Re: “A Critical Look at Some Popular Meta-Analytic Methods”
Olkin, I. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117249 · Cited by 115
Baker, S. G., & Kramer, B. S. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-13 · Cited by 114
Shuster, J. J., Jones, L. S., & Salmon, D. A. (2007) · Statistics in Medicine · doi:10.1002/sim.3060 · Cited by 105
Chou, R., Fu, R. F., Huffman, L. H., & Korthuis, P. T. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69638-4 · Cited by 104
Self-citationIndirect comparisons: the mesh and mess of clinical trials
Ioannidis, J. P. A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69615-3 · Cited by 94
Song, F., Glenny, A., & Altman, D. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00055-6 · Cited by 85
Droitcour, J. A., Silberman, G., & Chelimsky, E. (1993) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300004694 · Cited by 76
Meta-Analysis of Migraine Headache Treatments: Combining Information from Heterogeneous Designs
Dominici, F., Parmigiani, G., Wolpert, R. L., & Hasselblad, V. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473815 · Cited by 75
Schulzer, M., & Mancini, G. B. J. (1996) · International Journal of Epidemiology · doi:10.1093/ije/25.4.704 · Cited by 75
Dahabreh, I. J. (2008) · Clinical Trials · doi:10.1177/1740774508090212 · Cited by 62
AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program
Slutsky, J. R., Atkins, D. C., Chang, S., & Sharp, B. A. C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.009 · Cited by 49
AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program
Helfand, M., & Balshem, H. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.05.005 · Cited by 47
Duchateau, L., Collette, L., Sylvester, R. J., & Pignon, J. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00886.x · Cited by 35
Meta-Analysis of Migraine Headache Treatments: Combining Information from Heterogeneous Designs
Dominici, F., Parmigiani, G., Wolpert, R. L., & Hasselblad, V. (1999) · Journal of the American Statistical Association · doi:10.2307/2669674 · Cited by 18
The exact analysis of contingency tables in medical research
Mehta, C. R. (1995) · Cancer treatment and research · doi:10.1007/978-1-4615-2009-2_9 · Cited by 15
Engels, E. (2000) · doi:10.1002/1097-0258(20000715)19:13 · Cited by 2
Self-citationHahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8
Cited by
Sorted by citations
Racism as a Determinant of Health: A Systematic Review and Meta-Analysis
Paradies, Y., Ben, J., Denson, N., Elias, A., Priest, N., Pieterse, A., Gupta, A., Kelaher, M., & Gee, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0138511 · Cited by 2,610
“This threshold was used in prior meta-analyses, and based on minimum thresholds established in the literature [60, 61].”
MethodMindfulness-based interventions for psychiatric disorders: A systematic review and meta-analysis
Goldberg, S. B., Tucker, R. P., Greene, P. A., Davidson, R. J., Wampold, B. E., Kearney, D. J., & Simpson, T. L. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2017.10.011 · Cited by 1,145
Meta-analyses of factors motivating climate change adaptation behaviour
van Valkengoed, A. M., & Steg, L. (2019) · Nature Climate Change · doi:10.1038/s41558-018-0371-y · Cited by 940
Global prevalence of gaming disorder: A systematic review and meta-analysis
Stevens, M. W., Dorstyn, D., Delfabbro, P. H., & King, D. L. (2021) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867420962851 · Cited by 863
Gastrointestinal Symptoms in Autism Spectrum Disorder: A Meta-analysis
McElhanon, B. O., McCracken, C., Karpen, S., & Sharp, W. G. (2014) · Pediatrics · doi:10.1542/peds.2013-3995 · Cited by 831
Metadata
- Authors
- Fu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T. J., Griffith, L., Oremus, M., Raina, P., Ismaila, A., Santaguida, P., Lau, J., & Trikalinos, T. A.
- Affiliations
- Oregon Health & Science University (US)
- Universität für Weiterbildung Krems (AT)
- Blue Cross Blue Shield Association (US)
- University of Minnesota (US)
- Agency for Healthcare Research and Quality (US)
- McMaster University (CA)
- Tufts Medical Center (US)
- Corresponding author
- Fu, Rongwei
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-11
- Volume
- 64
- Issue
- 11
- Pages
- 1187-1197
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2145171366
- Cited by
- 730
- References
- 62
- Citations by year
- 2026: 46 · 2025: 64 · 2024: 88 · 2023: 75 · 2022: 99 · 2021: 92 · 2020: 56 · 2019: 50 · 2018: 29 · 2017: 27 · 2016: 33 · 2015: 26 · 2014: 18 · 2013: 13 · 2012: 10
- FWCI
- 14.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Agency for Healthcare Research and Quality
- H2020 European Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- meta-analysis, comparative effectiveness research, evidence-based practice centers, Agency for Healthcare Research and Quality, indirect comparison, effect measures, rare events, heterogeneity, sensitivity analysis
- MeSH terms
- Data Interpretation, Statistical, Humans, Review Literature as Topic, Statistics as Topic, United States, Meta-Analysis as Topic, Models, Statistical, Bias, United States Agency for Healthcare Research and Quality, Outcome Assessment, Health Care, Evidence-Based Medicine, Consensus, Comparative Effectiveness Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
