Investigating patient exclusion bias in meta-analysis
Abstract
BACKGROUND: Trial investigators frequently exclude patients from trial analyses which may bias estimates of the effect of treatment. Combining these estimates in a meta-analysis could aggregate any such biases. METHODS: To investigate how excluding patients from trials can affect the results of both trials and meta-analyses, we used 14 meta-analyses of individual patient data (IPD) that addressed therapeutic questions in cancer. These included 133 randomized controlled trials (RCT) and 21 905 patients. We explored whether exclusions were related to trial characteristics and categorized the reasons for exclusions. For each RCT and meta-analysis, we compared results of an intention-to-treat analysis of all randomized patients with an analysis based on those patients included in the investigators' analysis. RESULTS: In all, 92 trials (69%) excluded between 0.3 and 38% of patients randomized. Trials excluding patients tended to be older and larger than those that did not. Most patients were excluded because of ineligibility or protocol violations. Exclusions varied substantially by meta-analysis, more patients tending to be excluded from the treatment arm. Comparing trial analyses there was no clear indication that exclusion of patients altered the results more in favour of either treatment or control. However, comparing meta-analysis results, there was a tendency for those based on 'included' patients to favour the research treatment (P = 0.03). Inconsistency of trial results was often increased as a result of the investigators' exclusions. CONCLUSIONS: Trials, systematic reviews, and meta-analyses may be prone to bias associated with post-randomization exclusion of patients. Wherever possible, the level of such exclusions should be taken into account when assessing the potential for bias in trials, systematic reviews, and meta-analyses. Ideally, trials, systematic reviews, and meta-analyses should be based on all randomized patients.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such attrition can affect the ability of results to be generalized and it undermines the statistical power of the trial.
“Such attrition can affect the ability of results to be generalized [10,11] and it undermines the statistical power of the trial.”
From A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies (2011) · cites this paper as [10]
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Beta blockade during and after myocardial infarction: An overview of the randomized trials
Yusuf, S., Peto, R., Lewis, J., Collins, R., & Sleight, P. (1985) · Progress in Cardiovascular Diseases · doi:10.1016/s0033-0620(85)80003-7 · Cited by 3,205 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · see citing works
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545 · see citing works
Villalba, J. B. (1996) · Lung Cancer · doi:10.1016/0169-5002(96)85918-6 · Cited by 2,510 · see citing works
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160 · see citing works
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426 · see citing works
Stewart, L. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)08091-1 · Cited by 1,226 · see citing works
Improving the quality of reporting in randomised controlled trials.
Walker, A., & Nixon, J. (2004) · Journal of Wound Care · doi:10.12968/jowc.2004.13.3.26593 · Cited by 1,082 · see citing works
Burdett, S., Parmar, M., Stewart, L. A., Souhami, R. L., Arriagada, R., Girling, D., Pignon, J., Torri, V., Brichet, A., Lafitte, J. J., et al. (1998) · The Lancet · doi:10.1016/s0140-6736(98)06341-7 · Cited by 837 · see citing works
Meta-analysis of the literature or of individual patient data: is there a difference?
Stewart, L. A., & Parmar, M. (1993) · The Lancet · doi:10.1016/0140-6736(93)93004-k · Cited by 771 · see citing works
Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis
Fergusson, D., Aaron, S. D., Guyatt, G., & Hébert, P. (2002) · BMJ · doi:10.1136/bmj.325.7365.652 · Cited by 708 · see citing works
Statistical Considerations in the Intent-to-Treat Principle
Lachin, J. M. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00046-5 · Cited by 652 · see citing works
Sample size slippages in randomised trials: exclusions and the lost and wayward.
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07882-0 · Cited by 642 · see citing works
Randomisation and baseline comparisons in clinical trials
Altman, D. G., & Doré, C. J. (1990) · The Lancet · doi:10.1016/0140-6736(90)90014-v · Cited by 508 · see citing works
Practical methodology of meta‐analyses (overviews) using updated individual patient data
Stewart, L. A., & Cochrane Working Group On Meta‐Analysis Using Individual Patient Data (1995) · Statistics in Medicine · doi:10.1002/sim.4780141902 · Cited by 486 · see citing works
Value of Flow Diagrams in Reports of Randomized Controlled Trials
Egger, M. (2001) · JAMA · doi:10.1001/jama.285.15.1996 · Cited by 317 · see citing works
Chemotherapy in advanced ovarian cancer: An overview of randomised clinical trials
Stewart, L. A., Aabo, K., Adnitt, P. I., Adams, M., Alberts, D. S., Barley, V., Bell, D. R., Bianchi, U., Blessing, J. A., Buyse, M. E., et al. (1991) · Cited by 270 · see citing works
Intention to treat – who should use ITT?
Lewis, J., & Machin, D. (1993) · British Journal of Cancer · doi:10.1038/bjc.1993.402 · Cited by 236 · see citing works
Advanced Ovarian Cancer Trialists' Group, Aabo, K., Adams, M., Adnitt, P. I., Alberts, D. S., Athanazziou, A., Barley, V., Bell, D., Bianchi, U., Bolis, G., et al. (1998) · British Journal of Cancer · doi:10.1038/bjc.1998.710 · Cited by 233 · see citing works
Schulz, K. F., Grimes, D. A., Altman, D. G., & Hayes, R. J. (1996) · BMJ · doi:10.1136/bmj.312.7033.742 · Cited by 163 · see citing works
Arnott, S. J., Duncan, W. D., Gignoux, M., Girling, D. J., Hansen, H. S., Launois, B., Nygaard, K., Parmar, M., Roussel, A., Spiliopoulos, G., et al. (1998) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/s0360-3016(97)00569-5 · Cited by 156 · see citing works
Self-citationThirty years of medical research council randomized trials in solid tumours
Machin, D., Stenning, S. P., Parmar, M., Fayers, P. M., Girling, D. J., Stephens, R. J., Stewart, L. A., & Whaley, J. B. (1997) · Clinical Oncology · doi:10.1016/s0936-6555(05)80448-0 · Cited by 64 · see citing works
Statistical Considerations in the Intent-to-Treat Principle
Lachin, J. M. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00092-1 · Cited by 51 · see citing works
(1996) · International Journal of Gynecology & Obstetrics · doi:10.1016/0020-7292(96)88724-8 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
Practical methods for incorporating summary time-to-event data into meta-analysis.
Tierney, J. F., Stewart, L. A., Ghersi, D., Burdett, S., & Sydes, M. R. (2007) · Trials · doi:10.1186/1745-6215-8-16 · Cited by 6,208 · see citing works
Self-citationAssessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · see citing works
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538 · see citing works
2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663 · see citing works
A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies
Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1772 · Cited by 876 · see citing works
PRISMA for Abstracts: Reporting Systematic Reviews in Journal and Conference Abstracts
Beller, E. M., Glasziou, P. P., Altman, D. G., Hopewell, S., Bastian, H., Chalmers, I., Gøtzsche, P. C., Lasserson, T., Tovey, D., & for the PRISMA for Abstracts Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001419 · Cited by 706 · see citing works
Reporting attrition in randomised controlled trials
Dumville, J. C., Torgerson, D. J., & Hewitt, C. E. (2006) · BMJ · doi:10.1136/bmj.332.7547.969 · Cited by 443 · see citing works
van Ballegooijen, W., Cuijpers, P., van Straten, A., Karyotaki, E., Andersson, G., Smit, J. H., & Riper, H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100674 · Cited by 409 · see citing works
“Evidence” in chronic pain–establishing best practice in the reporting of systematic reviews
Moore, A. R., Eccleston, C., Derry, S., Wiffen, P., Bell, R. F., Straube, S., & McQuay, H. (2010) · Pain · doi:10.1016/j.pain.2010.05.011 · Cited by 370 · see citing works
Akl, E. A., Briel, M., You, J. J., Sun, X., Johnston, B. C., Busse, J. W., Mulla, S., Lamontagne, F., Bassler, D., Vera, C., et al. (2012) · BMJ · doi:10.1136/bmj.e2809 · Cited by 350 · see citing works
Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies
Lundh, A., & Gøtzsche, P. C. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-22 · Cited by 352 · see citing works
Risk of bias versus quality assessment of randomised controlled trials: cross sectional study
Hartling, L., Ospina, M., Liang, Y., Dryden, D. M., Hooton, N., Krebs Seida, J., & Klassen, T. P. (2009) · BMJ · doi:10.1136/bmj.b4012 · Cited by 348 · see citing works
Get real in individual participant data (IPD) meta‐analysis: a review of the methodology
Debray, T. P. A., Moons, K. G. M., van Valkenhoef, G., Efthimiou, O., Hummel, N., Groenwold, R. H. H., Reitsma, J. B., & on behalf of the GetReal methods review group (2015) · Research Synthesis Methods · doi:10.1002/jrsm.1160 · Cited by 348 · see citing works
Dechartres, A., Boutron, I., Trinquart, L., Charles, P., & Ravaud, P. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-1-201107050-00006 · Cited by 307 · see citing works
Nuesch, E., Trelle, S., Reichenbach, S., Rutjes, A. W. S., Burgi, E., Scherer, M., Altman, D. G., & Juni, P. (2009) · BMJ · doi:10.1136/bmj.b3244 · Cited by 248 · see citing works
Donkin, L., & Glozier, N. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2100 · Cited by 212 · see citing works
Forbes, D., Blake, C. M., Thiessen, E. J., Peacock, S., & Hawranik, P. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003946.pub4 · Cited by 194 · see citing works
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171 · see citing works
Abraha, I., Cherubini, A., Cozzolino, F., De Florio, R., Luchetta, M. L., Rimland, J. M., Folletti, I., Marchesi, M., Germani, A., Orso, M., et al. (2015) · BMJ · doi:10.1136/bmj.h2445 · Cited by 165 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Assessing the impact of attrition in randomized controlled trials
Hewitt, C. E., Kumaravel, B., Dumville, J. C., & Torgerson, D. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.01.010 · Cited by 139 · see citing works
Bafeta, A., Dechartres, A., Trinquart, L., Yavchitz, A., Boutron, I., & Ravaud, P. (2012) · BMJ · doi:10.1136/bmj.e813 · Cited by 129 · see citing works
Intention‐to‐treat in randomized controlled trials: Recommendations for a total trial strategy
Polit, D. F., & Gillespie, B. M. (2010) · Research in Nursing & Health · doi:10.1002/nur.20386 · Cited by 129 · see citing works
Holman, C., Piper, S. K., Grittner, U., Diamantaras, A. A., Kimmelman, J., Siegerink, B., & Dirnagl, U. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002331 · Cited by 114 · see citing works
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85 · see citing works
Declining efficacy in controlled trials of antidepressants: effects of placebo dropout
Schalkwijk, S., Undurraga, J., Tondo, L., & Baldessarini, R. J. (2014) · The International Journal of Neuropsychopharmacology · doi:10.1017/s1461145714000224 · Cited by 34 · see citing works
- Authors
- Tierney, J. F.
- Affiliations
- MRC Clinical Trials Unit at UCL (GB)
- Corresponding author
- Tierney, J. F
- ORCID
- Tierney, J. F
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2004-08-27
- Volume
- 34
- Issue
- 1
- Pages
- 79-87
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2137745476
- Cited by
- 221
- References
- 28
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 7 · 2023: 7 · 2022: 9 · 2021: 11 · 2020: 5 · 2019: 9 · 2018: 9 · 2017: 12 · 2016: 13 · 2015: 17 · 2014: 14 · 2013: 8 · 2012: 17
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- meta-analysis bias, intention-to-treat analysis, individual patient data meta-analysis, randomized controlled trials, cancer clinical trials, inclusion and exclusion criteria, protocol violations, bias assessment
- MeSH terms
- Humans, Patient Dropouts, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Bias, Proportional Hazards Models, Randomized Controlled Trials as Topic, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
