The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Association Between Analytic Strategy and Estimates of Treatment Outcomes in Meta-analyses
Abstract
IMPORTANCE: A persistent dilemma when performing meta-analyses is whether all available trials should be included in the meta-analysis. OBJECTIVES: To compare treatment outcomes estimated by meta-analysis of all trials and several alternative analytic strategies: single most precise trial (ie, trial with the narrowest confidence interval), meta-analysis restricted to the 25% largest trials, limit meta-analysis (a meta-analysis model adjusted for small-study effect), and meta-analysis restricted to trials at low overall risk of bias. DATA SOURCES: One hundred sixty-three meta-analyses published between 2008 and 2010 in high-impact-factor journals and between 2011 and 2013 in the Cochrane Database of Systematic Reviews: 92 (705 randomized clinical trials [RCTs]) with subjective outcomes and 71 (535 RCTs) with objective outcomes. DATA SYNTHESIS: For each meta-analysis, the difference in treatment outcomes between meta-analysis of all trials and each alternative strategy, expressed as a ratio of odds ratios (ROR), was assessed considering the dependency between strategies. A difference greater than 30% was considered substantial. RORs were combined by random-effects meta-analysis models to obtain an average difference across the sample. An ROR greater than 1 indicates larger treatment outcomes with meta-analysis of all trials. Subjective and objective outcomes were analyzed separately. RESULTS: Treatment outcomes were larger in the meta-analysis of all trials than in the single most precise trial (combined ROR, 1.13 [95% CI, 1.07-1.19]) for subjective outcomes and 1.03 (95% CI, 1.01-1.05) for objective outcomes). The difference in treatment outcomes between these strategies was substantial in 47 of 92 (51%) meta-analyses of subjective outcomes (meta-analysis of all trials showing larger outcomes in 40/47) and in 28 of 71 (39%) meta-analyses of objective outcomes (meta-analysis of all trials showing larger outcomes in 21/28). The combined ROR for subjective and objective outcomes was, respectively, 1.08 (95% CI, 1.04-1.13) and 1.03 (95% CI, 1.00-1.06) when comparing meta-analysis of all trials and meta-analysis of the 25% largest trials, 1.17 (95% CI, 1.11-1.22) and 1.13 (95% CI, 0.82-1.55) when comparing meta-analysis of all trials and limit meta-analysis, and 0.94 (95% CI, 0.86-1.04) and 1.03 (95% CI, 1.00-1.06) when comparing meta-analysis of all trials and meta-analysis restricted to trials at low risk of bias. CONCLUSIONS AND RELEVANCE: Estimation of treatment outcomes in meta-analyses differs depending on the strategy used. This instability in findings can result in major alterations in the conclusions derived from the analysis and underlines the need for systematic sensitivity analyses.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The persistent evidence of small study effects, as well as the significant heterogeneity, y casts doubts over the reliability of the large effects observed for anxiety25,37,38.”
From The effectiveness of virtual reality based interventions for symptoms of anxiety and depression: A meta-analysis (2018) · cites this paper as [37]
In the citing paper
“Sensitivity analyses were performed by excluding small-size studies (< 25th percentile) [32] and excluding primary studies having a high risk of bias rated by the Cochrane’s risk of bias 2 tool (RoB 2) for RCTs from the identified associations [19–21, 33].”
From Effects of ketogenic diet on health outcomes: an umbrella review of meta-analyses of randomized clinical trials (2023) · cites this paper as [32]
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationSterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709
Self-citationSchulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309
Self-citationWood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Self-citationPublication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282
Gluud, L. L., Villumsen, J., & Gluud, C. N. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-11-200112040-00010 · Cited by 2,266
Investigating and dealing with publication and other biases in meta-analysis
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001) · BMJ · doi:10.1136/bmj.323.7304.101 · Cited by 2,225
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
Self-citationWhat to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734
Discrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials
LeLorier, J., Grégoire, G., Benhaddad, A. A., Lapierre, J., & Derderian, F. (1997) · New England Journal of Medicine · doi:10.1056/nejm199708213370806 · Cited by 1,164
Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010) · BMJ · doi:10.1136/bmj.c365 · Cited by 1,133
Self-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Self-citationBest evidence synthesis: An intelligent alternative to meta-analysis
Slavin, R. E. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00097-a · Cited by 951
Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Tschannen, B., Altman, D. G., Egger, M., & Juni, P. (2010) · BMJ · doi:10.1136/bmj.c3515 · Cited by 559
Self-citationInfluence of trial sample size on treatment effect estimates: meta-epidemiological study
Dechartres, A., Trinquart, L., Boutron, I., & Ravaud, P. (2013) · BMJ · doi:10.1136/bmj.f2304 · Cited by 544
Self-citationEgger, M., & Smith, G. D. (1995) · BMJ · doi:10.1136/bmj.310.6982.752 · Cited by 508
Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials
Pildal, J., Hrobjartsson, A., Jorgensen, K., Hilden, J., Altman, D., & Gotzsche, P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym087 · Cited by 472
Self-citationPreventing the Onset of Depressive Disorders: A Meta-Analytic Review of Psychological Interventions
Cuijpers, P., van Straten, A., Smit, F., Mihalopoulos, C., & Beekman, A. (2008) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.07091422 · Cited by 456
Prevention of Atrial Fibrillation by Renin-Angiotensin System Inhibition
Schneider, M. P., Hua, T. A., Böhm, M., Wachtell, K., Kjeldsen, S. E., & Schmieder, R. E. (2010) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2010.01.043 · Cited by 417
Sterne, J. A. C., Jüni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1184 · Cited by 382
Self-citationMoreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-2 · Cited by 373
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348
Large Trials vs Meta-analysis of Smaller Trials: How Do Their Results Compare?
Cappelleri, J. C. (1996) · JAMA · doi:10.1001/jama.1996.03540160054033 · Cited by 336
Dechartres, A., Boutron, I., Trinquart, L., Charles, P., & Ravaud, P. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00006 · Cited by 307
Self-citationTreatment-effect estimates adjusted for small-study effects via a limit meta-analysis
Rücker, G., Schwarzer, G., Carpenter, J. R., Binder, H., & Schumacher, M. C. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq046 · Cited by 295
Mupirocin ointment for preventing Staphylococcus aureus infections in nasal carriers
van Rijen, M., Bonten, M. J. M., Wenzel, R. P., & Kluytmans, J. A. J. W. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006216.pub2 · Cited by 286
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248
Self-citationIssues in Comparisons Between Meta-analyses and Large Trials
Ioannidis, J. P. A. (1998) · JAMA · doi:10.1001/jama.279.14.1089 · Cited by 246
Strengths and limitations of meta-analysis: Larger studies may be more reliable
Flather, M. D., Farkouh, M. E., Pogue, J. M., & Yusuf, S. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00024-x · Cited by 237
Predictive ability of meta-analyses of randomised controlled trials
Villar, J., Carroli, G., & Belizán, J. M. (1995) · The Lancet · doi:10.1016/s0140-6736(95)90646-0 · Cited by 223
Moseley, A. M., Elkins, M. R., Herbert, R. D., Maher, C. G., & Sherrington, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.018 · Cited by 202
Hartling, L., Hamm, M. P., Milne, A., Vandermeer, B., Santaguida, P. L., Ansari, M., Tsertsvadze, A., Hempel, S., Shekelle, P., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.07.005 · Cited by 173
Pereira, T. V., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.012 · Cited by 160
Detecting and adjusting for small‐study effects in meta‐analysis
Rücker, G., Carpenter, J. R., & Schwarzer, G. (2011) · Biometrical Journal · doi:10.1002/bimj.201000151 · Cited by 132
Borzak, S. L., & Ridker, P. M. (1995) · Annals of Internal Medicine · doi:10.7326/0003-4819-123-11-199512010-00010 · Cited by 131
Shea, B., Moher, D., Graham, I. D., Pham, B., & Tugwell, P. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001008 · Cited by 109
Moreno, S. G., Sutton, A. J., Thompson, J. R., Ades, A. E., Abrams, K. R., & Cooper, N. J. (2012) · Statistics in Medicine · doi:10.1002/sim.4488 · Cited by 90
Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice
Schriger, D. L., Altman, D. G., Vetter, J. A., Heafner, T., & Moher, D. (2010) · International Journal of Epidemiology · doi:10.1093/ije/dyp370 · Cited by 87
Self-citationHopewell, S., Boutron, I., Altman, D. G., & Ravaud, P. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003342 · Cited by 66
Self-citationMeta-Analyses and Large Randomized, Controlled Trials
Ioannidis, J. P. A., Cappelleri, J. C., & Lau, J. T. F. (1998) · New England Journal of Medicine · doi:10.1056/nejm199801013380112 · Cited by 56
Piscione, F., Piccolo, R., Cassese, S., Galasso, G., D’Andrea, C., Rosa, R. D., & Chiariello, M. (2010) · Heart · doi:10.1136/hrt.2009.183277 · Cited by 46
Tudor‐Smith, C., & Clarkson, J. (1995) · BMJ · doi:10.1136/bmj.311.7007.753b · Cited by 42
Can we rely on the best trial? A comparison of individual trials and systematic reviews
Glasziou, P., Shepperd, S., & Brassey, J. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-23 · Cited by 37
Comparing results of large clinical trials to those of meta‐analyses
Moses, L. E., Mosteller, F. C., & Buehler, J. H. (2002) · Statistics in Medicine · doi:10.1002/sim.1098 · Cited by 35
Large Trials vs Meta-analysis of Smaller Trials
Johnson, B. T. (1997) · JAMA · doi:10.1001/jama.1997.03540290029025 · Cited by 34
Comparing Results From Meta-analyses vs Large Trials
LeLorier, J., & Grégoire, G. (1998) · JAMA · doi:10.1001/jama.280.6.518 · Cited by 13
Meta-analyses and large randomized, controlled trials.
Bent, S., Kerlikowske, K. M., & Grady, D. (1998) · PubMed · Cited by 8
Large Trials vs Meta-analysis of Smaller Trials
Klebanoff, M. A. (1997) · JAMA · doi:10.1001/jama.1997.03540290028024 · Cited by 5
Flagged first, then by citations
1 of the 54 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016) · Evidence Based Medicine · doi:10.1136/ebmed-2016-110401 · Cited by 1,388
Cannabis-based medicines for chronic neuropathic pain in adults
Mücke, M., Phillips, T., Radbruch, L., Petzke, F. W., & Häuser, W. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012182.pub2 · Cited by 606
Gabapentin for chronic neuropathic pain in adults
Wiffen, P. J., Derry, S., Bell, R. F., Rice, A. S., Tölle, T. R., Phillips, T., & Moore, R. A. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007938.pub4 · Cited by 561
Kakoly, N. S., Khomami, M. B., Joham, A. E., Cooray, S. D., Misso, M. L., Norman, R. J., Harrison, C. L., Ranasinha, S., Teede, H. J., & Moran, L. J. (2018) · Human Reproduction Update · doi:10.1093/humupd/dmy007 · Cited by 384
Perioperative intravenous ketamine for acute postoperative pain in adults
Brinck, E. C. V., Tiippana, E., Heesen, M. A., Bell, R. F., Straube, S., Moore, A., & Kontinen, V. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012033.pub4 · Cited by 385
Metadata
- Authors
- Dechartres, A., Altman, D. G., Trinquart, L., Boutron, I., & Ravaud, P.
- Affiliations
- Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France2Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris, Paris, France3Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cit
- University of Oxford (GB)
- Inserm (FR)
- Venue
- JAMA
- Publisher
- American Medical Association (AMA)
- Published
- 2014-08-13
- Volume
- 312
- Issue
- 6
- Pages
- 623
- ISSN
- 0002-9955 (ISSN-L)
- 0098-7484
- OpenAlex
- W2108868101
- Cited by
- 233
- References
- 54
- Citations by year
- 2026: 15 · 2025: 15 · 2024: 25 · 2023: 17 · 2022: 25 · 2021: 15 · 2020: 11 · 2019: 16 · 2018: 18 · 2017: 34 · 2016: 23 · 2015: 11 · 2014: 8
- FWCI
- 24.79
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- JAMA (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Cancer Research UK (16895)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- analytical strategies, treatment effect estimation, small-study effects, risk of bias, trial size, subjective outcomes, objective outcomes, publication bias, Cochrane Database of Systematic Reviews, randomized controlled trials
- MeSH terms
- Humans, Research Design, Meta-Analysis as Topic, Confidence Intervals, Odds Ratio, Randomized Controlled Trials as Topic, Treatment Outcome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
