Sample size slippages in randomised trials: exclusions and the lost and wayward.
Abstract
Proper randomisation means little if investigators cannot include all randomised participants in the primary analysis. Participants might ignore follow-up, leave town, or take aspartame when instructed to take aspirin. Exclusions before randomisation do not bias the treatment comparison, but they can hurt generalisability. Eligibility criteria for a trial should be clear, specific, and applied before randomisation. Readers should assess whether any of the criteria make the trial sample atypical or unrepresentative of the people in which they are interested. In principle, assessment of exclusions after randomisation is simple: none are allowed. For the primary analysis, all participants enrolled should be included and analysed as part of the original group assigned (an intent-to-treat analysis). In reality, however, losses frequently occur. Investigators should, therefore, commit adequate resources to develop and implement procedures to maximise retention of participants. Moreover, researchers should provide clear, explicit information on the progress of all randomised participants through the trial by use of, for instance, a trial profile. Investigators can also do secondary analyses on, for instance, per-protocol or as-treated participants. Such analyses should be described as secondary and non-randomised comparisons. Mishandling of exclusions causes serious methodological difficulties. Unfortunately, some explanations for mishandling exclusions intuitively appeal to readers, disguising the seriousness of the issues. Creative mismanagement of exclusions can undermine trial validity.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The attrition rate was close to the acceptable threshold of 20% of randomized participants, and the only baseline differences between completers and dropouts were lower levels of anxiety and perceived stress in patients who dropped out of the PEI.
“Although the attrition rate was close to the acceptable threshold, which is 20% of randomized participants (Schulz and Grimes, 2002), the only baseline differences between completers and dropouts, were lower levels of anxiety and perceived stress in patients that dropped out of the PEI.”
From The effectiveness of a body-affective mindfulness intervention for multiple sclerosis patients with depressive symptoms: A randomized controlled clinical trial (2017) · cites this paper as Schulz and Grimes, 2002
Such attrition can affect the ability of results to be generalized and it undermines the statistical power of the trial.
“Such attrition can affect the ability of results to be generalized [10,11] and it undermines the statistical power of the trial.”
From A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies (2011) · cites this paper as [11]
Cites
Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1977) · British Journal of Cancer · doi:10.1038/bjc.1977.1 · Cited by 8,285 · see citing works
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Self-citationThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835 · see citing works
Self-citationPeto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1976) · British Journal of Cancer · doi:10.1038/bjc.1976.220 · Cited by 2,294 · see citing works
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237 · see citing works
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160 · see citing works
Evidence-Based Medicine: How to Practice and Teach EBM
Satya-Murti, S. (2000) · JAMA · doi:10.1001/jama.284.18.2382 · Cited by 1,659 · see citing works
Evidence-based Medicine: How to Practice and Teach EBM
Satya‐Murti, S. (1997) · JAMA · doi:10.1001/jama.1997.03550020100049 · Cited by 974 · see citing works
Allocation concealment in randomised trials: defending against deciphering
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07750-4 · Cited by 923 · see citing works
Self-citationGeneration of allocation sequences in randomised trials: chance, not choice
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07683-3 · Cited by 774 · see citing works
Self-citationPhysical fitness training for stroke patients
Saunders, D. H., Sanderson, M., Hayes, S., Kilrane, M., Greig, C. A., Brazzelli, M., & Mead, G. E. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003316.pub6 · Cited by 759 · see citing works
An overview of clinical research: the lay of the land
Grimes, D. A., & Schulz, K. F. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07283-5 · Cited by 734 · see citing works
Self-citationStatistical Considerations in the Intent-to-Treat Principle
Lachin, J. M. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00046-5 · Cited by 652 · see citing works
Reporting on Methods in Clinical Trials
DerSimonian, R., Charette, L. J., McPeek, B., & Mosteller, F. C. (1982) · New England Journal of Medicine · doi:10.1056/nejm198206033062204 · Cited by 509 · see citing works
Sulfinpyrazone in the Prevention of Sudden Death after Myocardial Infarction
The Anturane Reinfarction Trial Research Group (1980) · New England Journal of Medicine · doi:10.1056/nejm198001313020502 · Cited by 346 · see citing works
Fundamentals of Clinical Trials
Rhoads, G. G. (1986) · JAMA · doi:10.1001/jama.1986.03380080101043 · Cited by 345 · see citing works
Sulfinpyrazone in the Prevention of Cardiac Death after Myocardial Infarction
The Anturane Reinfarction Trial Research Group (1978) · New England Journal of Medicine · doi:10.1056/nejm197802092980601 · Cited by 299 · see citing works
Analysis of clinical trials by treatment actually received: Is it really an option?
Lee, Y. J., Ellenberg, J. H., Hirtz, D. G., & Nelson, K. B. (1991) · Statistics in Medicine · doi:10.1002/sim.4780101011 · Cited by 283 · see citing works
Intention to treat – who should use ITT?
Lewis, J., & Machin, D. (1993) · British Journal of Cancer · doi:10.1038/bjc.1993.402 · Cited by 236 · see citing works
Schulz, K. F., Grimes, D. A., Altman, D. G., & Hayes, R. J. (1996) · BMJ · doi:10.1136/bmj.312.7033.742 · Cited by 163 · see citing works
Self-citationThe FDA's Critique of the Anturane Reinfarction Trial
Temple, R., & Pledger, G. W. (1980) · New England Journal of Medicine · doi:10.1056/nejm198012183032534 · Cited by 150 · see citing works
Clinical Trials: Issues and Approaches.
Fleiss, J. L., Shapiro, S. H., & Louis, T. (1984) · Journal of the American Statistical Association · doi:10.2307/2288302 · Cited by 112 · see citing works
Effect of Nonoxynol-9 Gel on Urogenital Gonorrhea and Chlamydial Infection
Roddy, R. E., Zekeng, L. A., Ryan, K. A., Tamoufé, U., & Tweedy, K. G. (2002) · JAMA · doi:10.1001/jama.287.9.1117 · Cited by 105 · see citing works
Content of reports on clinical trials: A critical review
Meinert, C. L., Tonascia, S., & Higgins, K. (1984) · Controlled Clinical Trials · doi:10.1016/s0197-2456(84)80013-6 · Cited by 60 · see citing works
Roddy, R. E., Zekeng, L. A., Ryan, K. A., Tamoufé, U., & Tweedy, K. G. (2002) · Obstetrical & Gynecological Survey · doi:10.1097/00006254-200206000-00013 · Cited by 9 · see citing works
Meier, P. (1981) · New England Journal of Medicine · doi:10.1056/nejm198103193041212 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies
Donkin, L., Christensen, H., Naismith, S. L., Neal, B., Hickie, I. B., & Glozier, N. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1772 · Cited by 876 · see citing works
Physical fitness training for stroke patients
Saunders, D. H., Sanderson, M., Hayes, S., Kilrane, M., Greig, C. A., Brazzelli, M., & Mead, G. E. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003316.pub6 · Cited by 759 · see citing works
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636 · see citing works
Nery, F. G., Monkul, E. S., Hatch, J. P., Fonseca, M., Zunta‐Soares, G. B., Frey, B. N., Bowden, C. L., & Soares, J. C. (2008) · Human Psychopharmacology: Clinical and Experimental · doi:10.1002/hup.912 · Cited by 298 · see citing works
Murray, J. M., Brennan, S. F., French, D. P., Patterson, C. C., Kee, F., & Hunter, R. F. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.09.021 · Cited by 229 · see citing works
Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis
Brueton, V. C., Tierney, J. F., Stenning, S., Meredith, S., Harding, S., Nazareth, I., & Rait, G. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-003821 · Cited by 253 · see citing works
Donkin, L., & Glozier, N. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2100 · Cited by 212 · see citing works
Investigating patient exclusion bias in meta-analysis
Tierney, J. F. (2004) · International Journal of Epidemiology · doi:10.1093/ije/dyh300 · Cited by 221 · see citing works
Catalogue of bias: attrition bias
Nunan, D., Aronson, J., & Bankhead, C. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110883 · Cited by 204 · see citing works
Teo, E., Hassan, N., Tam, W., & Koh, S. (2022) · Diabetologia · doi:10.1007/s00125-021-05648-4 · Cited by 186 · see citing works
Cognitive behavioural therapy for psychosis prevention: a systematic review and meta-analysis
Hutton, P., & Taylor, P. J. (2014) · Psychological Medicine · doi:10.1017/s0033291713000354 · Cited by 184 · see citing works
Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., & Ades, A. E. (2009) · BMJ · doi:10.1136/bmj.b2981 · Cited by 184 · see citing works
Liyana Amin, N. A., Tam, W. W. S., & Shorey, S. (2018) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2018.03.021 · Cited by 167 · see citing works
Intention-to-treat approach to data from randomized controlled trials: a sensitivity analysis
Wright, C. C., & Sim, J. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00155-0 · Cited by 171 · see citing works
Steroidal contraceptives: effect on carbohydrate metabolism in women without diabetes mellitus
Lopez, L. M., Grimes, D. A., & Schulz, K. F. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006133.pub5 · Cited by 166 · see citing works
Coventry, P. A., Bower, P., Keyworth, C., Kenning, C., Knopp, J., Garrett, C., Hind, D., Malpass, A., & Dickens, C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0060532 · Cited by 157 · see citing works
Assessing the impact of attrition in randomized controlled trials
Hewitt, C. E., Kumaravel, B., Dumville, J. C., & Torgerson, D. J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.01.010 · Cited by 139 · see citing works
Sztein, D. M., Koransky, C. E., Fegan, L., & Himelhoch, S. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x17717402 · Cited by 135 · see citing works
Differential attrition in health behaviour change trials: a systematic review and meta-analysis
Crutzen, R., Viechtbauer, W., Spigt, M., & Kotz, D. (2015) · Psychology & Health · doi:10.1080/08870446.2014.953526 · Cited by 132 · see citing works
Intention‐to‐treat in randomized controlled trials: Recommendations for a total trial strategy
Polit, D. F., & Gillespie, B. M. (2010) · Research in Nursing & Health · doi:10.1002/nur.20386 · Cited by 129 · see citing works
Ameli, R., Sinaii, N., West, C. P., Luna, M. J., Panahi, S., Zoosman, M., Rusch, H. L., & Berger, A. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.13424 · Cited by 94 · see citing works
The use of periodization in exercise prescriptions for inactive adults: A systematic review
Strohacker, K., Fazzino, D., Breslin, W. L., & Xu, X. (2015) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2015.04.023 · Cited by 79 · see citing works
Vetrovsky, T., Borowiec, A., Juřík, R., Wahlich, C., Śmigielski, W., Steffl, M., Tufano, J. J., Drygas, W., Stastny, P., Harris, T., et al. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2021-105198 · Cited by 68 · see citing works
Carletto, S., Tesio, V., Borghi, M., Francone, D., Scavelli, F., Bertino, G., Malucchi, S., Bertolotto, A., Oliva, F., Torta, R., et al. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.02083 · Cited by 49 · see citing works
Jennings, V., Haile, L. T., Simmons, R. G., Spieler, J., & Shattuck, D. (2019) · The European Journal of Contraception & Reproductive Health Care · doi:10.1080/13625187.2019.1581164 · Cited by 41 · see citing works
Declining efficacy in controlled trials of antidepressants: effects of placebo dropout
Schalkwijk, S., Undurraga, J., Tondo, L., & Baldessarini, R. J. (2014) · The International Journal of Neuropsychopharmacology · doi:10.1017/s1461145714000224 · Cited by 34 · see citing works
Crutzen, R., Viechtbauer, W., Kotz, D., & Spigt, M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.019 · Cited by 32 · see citing works
Gandy, M., Woldhuis, T., Wu, W., Youssef, M., Bisby, M. A., Dear, B. F., Heriseanu, A. I., & Scott, A. J. (2024) · Psychological Medicine · doi:10.1017/s0033291724001995 · Cited by 20 · see citing works
Suicide risk screening tools for pediatric patients: a systematic review of test accuracy
Lowry, N. J., Goger, P., Hands Ruz, M., Ye, F., & Cha, C. B. (2024) · Pediatrics · doi:10.1542/peds.2023-064172 · Cited by 18 · see citing works
Morris, L., Mansell, W., Emsley, R., Bates, R., Comiskey, J., Pistorius, E., & McEvoy, P. (2016) · Psychology and Psychotherapy: Theory, Research and Practice · doi:10.1111/papt.12070 · Cited by 14 · see citing works
Harris, R., Vernazza, C., Laverty, L., Lowers, V., Burnside, G., Brown, S., Higham, S., & Ternent, L. (2020) · Health Services and Delivery Research · doi:10.3310/hsdr08030 · Cited by 7 · see citing works
- Authors
- Schulz, K. F., & Grimes, D. A.
- Affiliations
- Family Health International 360 (US)
- Corresponding author
- Schulz, Kenneth F
- Venue
- The Lancet
- Publisher
- Elsevier BV
- Published
- 2002-03
- Volume
- 359
- Issue
- 9308
- Pages
- 781-785
- ISSN
- 0099-5355 (ISSN-L)
- 0140-6736
- OpenAlex
- W2115086661
- Cited by
- 642
- References
- 27
- Citations by year
- 2026: 42 · 2025: 30 · 2024: 39 · 2023: 32 · 2022: 30 · 2021: 49 · 2020: 34 · 2019: 37 · 2018: 35 · 2017: 33 · 2016: 28 · 2015: 26 · 2014: 29 · 2013: 29 · 2012: 17
- FWCI
- 20.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Lancet (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- randomized controlled trials, sample size, exclusion, loss to follow-up, eligibility criteria, intention-to-treat analysis, per-protocol analysis, as-treated analysis, participant retention, generalizability, trial validity, missing data
- MeSH terms
- Follow-Up Studies, Humans, Patient Dropouts, Bias, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
