Early stopping of randomized clinical trials for overt efficacy is problematic.
Abstract
Objective To illustrate controversial issues associated with stopping randomized controlled trials (RCTs) early for apparent benefit.Study design and setting The article presents our review of prior relevant work and our research group's reflections on early stopping.Results Compelling evidence suggests that trials stopped early for benefit systematically overestimate treatment effects, sometimes by a large amount. Unresolved controversies in trials stopped early for benefit include ethical and statistical problems in the interpretation of results.Conclusions The best strategy to minimize the problems associated with early stopping of RCTs for benefit is not to stop early. As an alternative, we suggest a threefold approach: a low P-value as the threshold for stopping at the time of interim analyses, not to look before a sufficiently large number of events has accrued and continuation of enrollment and follow-up for a further period.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The authors determined that a sufficient number of clusters had completed implementation of the planned intervention to allow analysis of the primary outcome but were unable to use the final 3 months of the stepped-wedge design (after the policy change) for the planned analysis.
“The authors determined that a sufficient number of clusters had completed implementation of the planned intervention to allow analysis of the primary outcome but were unable to use the final 3 months of the stepped-wedge design (after the policy change) for the planned analysis.”
In a multicenter stepped-wedge cluster randomized pragmatic trial, regional government enacted a policy change partway through the rollout that mandated widespread use of a modified version of the experimental intervention as the new standard of care.
“In a multicenter stepped-wedge cluster randomized pragmatic trial, regional government enacted a policy change partway through the rollout that mandated widespread use of a modified version of the experimental intervention as the new standard of care.”
The trialists identified this policy change as an extenuating circumstance leading to a premature and unanticipated modification to the implementation schedule at 9 months, before the planned completion at 12 months.
“The trialists identified this policy change as an extenuating circumstance leading to a premature and unanticipated modification to the implementation schedule at 9 months, before the planned completion at 12 months.”
The international survey yielded examples of trials that encountered extenuating circumstances including regulatory changes, new standards of care, and security threats.
“The international survey yielded examples of trials that encountered extenuating circumstances including regulatory changes, new standards of care, and security threats (Table).”
The trial by Vaillancourt et al on out-of-hospital cervical spine immobilization was stopped early when regional authorities enacted a new policy mandating a modified immobilization rule and adopting many aspects of the trial's experimental intervention as standard of care.
“The trial by Vaillancourt et al 20 on out-of-hospital cervical spine immobilization was stopped early when regional authorities enacted a new policy mandating a modified immobilization rule and adopting many aspects of the trial's experimental intervention as standard of care (Table).”
The investigators' mitigating strategies included omission of the final study period from the randomized clinical trial analysis, an extended observational data collection period for 3 months beyond the anticipated stop date, and a revised design and analysis plan to include 3 conditions: usual care, planned intervention, and an uncontrolled modified intervention condition.
“The investigators' mitigating strategies included omission of the final study period from the randomized clinical trial analysis, an extended observational data collection period for 3 months beyond the anticipated stop date, and a revised design and analysis plan to include 3 conditions: usual care, planned intervention, and an uncontrolled modified intervention condition.”
Cites
Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1977) · British Journal of Cancer · doi:10.1038/bjc.1977.1 · Cited by 8,285 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826 · see citing works
A Multiple Testing Procedure for Clinical Trials
O'Brien, P. C., & Fleming, T. R. (1979) · Biometrics · doi:10.2307/2530245 · Cited by 3,311 · see citing works
Sleight, R. B., Sleight, P., Parish, S., Appleby, P. N., Collins, R., & Baigent, C. (1988) · PubMed · Cited by 2,868 · see citing works
急性心筋梗塞の疑いのある17187症例におけるストレプトキナーゼ静脈内投与,アスピリン経口投与,上記2つの併用または無処置の無作偽試験 ISIS-2
Dienstl, F., Backer, G. D., Cairns, J., Pedersen, F., Kala, R., Boissel, J., Schroeder, R., Horgan, J., & Tognoni, G. (1988) · The Lancet · Cited by 2,754 · see citing works
What Makes Clinical Research Ethical?
Emanuel, E. J. (2000) · JAMA · doi:10.1001/jama.283.20.2701 · Cited by 2,710 · see citing works
Auvert, B., Taljaard, D., Lagarde, E., Sobngwi-Tambekou, J., Sitta, R., & Puren, A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020298 · Cited by 2,451 · see citing works
Equipoise and the Ethics of Clinical Research
Freedman, B. (1987) · New England Journal of Medicine · doi:10.1056/nejm198707163170304 · Cited by 2,339 · see citing works
Discrete sequential boundaries for clinical trials
GORDON LAN, K. K., & DEMETS, D. L. (1983) · Biometrika · doi:10.1093/biomet/70.3.659 · Cited by 2,310 · see citing works
Discrete Sequential Boundaries for Clinical Trials
Lan, K. K. G., & DeMets, D. L. (1983) · Biometrika · doi:10.2307/2336502 · Cited by 2,310 · see citing works
Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., Howard, S. V., Mantel, N., McPherson, K., Peto, J., & Smith, P. G. (1976) · British Journal of Cancer · doi:10.1038/bjc.1976.220 · Cited by 2,294 · see citing works
Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial.
Bailey, R. C., Moses, S., Parker, C. B., Agot, K., Maclean, I., Krieger, J. N., Williams, C. F., Campbell, R. T., & Ndinya-Achola, J. O. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60312-2 · Cited by 2,292 · see citing works
Male circumcision for HIV prevention in men in Rakai, Uganda: a randomised trial.
Gray, R. H., Kigozi, G., Serwadda, D., Makumbi, F., Watya, S., Nalugoda, F., Kiwanuka, N., Moulton, L. H., Chaudhary, M. A., Chen, M. Z., et al. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60313-4 · Cited by 2,226 · see citing works
Poldermans, D., Boersma, E. H., Bax, J. J., Thomson, I. R., van de Ven, L. L. M., Blankensteijn, J. D., Baars, H. F., Yo, T. I., Trocino, G., Vigna, C., et al. (1999) · New England Journal of Medicine · doi:10.1056/nejm199912093412402 · Cited by 1,480 · see citing works
Berry, D. A. (2006) · Nature Reviews Drug Discovery · doi:10.1038/nrd1927 · Cited by 728 · see citing works
Why do we need some large, simple randomized trials?
Yusuf, S., Collins, R., & Peto, R. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030421 · Cited by 700 · see citing works
Repeated assessment of results in clinical trials of cancer treatment
Haybittle, J. L. (1971) · British Journal of Radiology · doi:10.1259/0007-1285-44-526-793 · Cited by 519 · see citing works
In Vitro Fertilization with Single Blastocyst-Stage versus Single Cleavage-Stage Embryos
Papanikolaou, E. G. E., Camus, M., Kolibianakis, E. M., Landuyt, L. V., Steirteghem, A. V., & Devroey, P. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa053524 · Cited by 416 · see citing works
Multiplicity in randomised trials II: subgroup and interim analyses
Schulz, K. F., & Grimes, D. A. (2005) · The Lancet · doi:10.1016/s0140-6736(05)66516-6 · Cited by 357 · see citing works
A proposed charter for clinical trial data monitoring committees: helping them to do their job well
(2005) · The Lancet · doi:10.1016/s0140-6736(05)17965-3 · Cited by 324 · see citing works
Ethical issues in stopping randomized trials early because of apparent benefit.
Mueller, P. S., Montori, V. M., Bassler, D., Koenig, B. A., & Guyatt, G. H. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-12-200706190-00009 · Cited by 202 · see citing works
Evolution of treatment effects over time: Empirical insight from recursive cumulative metaanalyses
Ioannidis, J. P. A., & Lau, J. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.98.3.831 · Cited by 190 · see citing works
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time
Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., & Ioannidis, J. P. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.018 · Cited by 177 · see citing works
Practical problems in interim analyses, with particular regard to estimation
Pocock, S., & Hughes, M. D. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90059-7 · Cited by 160 · see citing works
Peltoniemi, O. M., Kari, M. A., Heinonen, K., Saarela, T., Nikolajev, K., Andersson, S., Voutilainen, R., & Hallman, M. (2005) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2004.12.040 · Cited by 139 · see citing works
Stopping rules and estimation problems in clinical trials
Hughes, M. D., & Pocock, S. (1988) · Statistics in Medicine · doi:10.1002/sim.4780071204 · Cited by 128 · see citing works
Smith, M. R., Manola, J. B., Kaufman, D. S., Oh, W. K., Bubley, G. J., & Kantoff, P. W. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2005.03.7804 · Cited by 127 · see citing works
Trials stopped early: too good to be true?
Pocock, S., & White, I. R. (1999) · The Lancet · doi:10.1016/s0140-6736(98)00379-1 · Cited by 113 · see citing works
Data Safety and Monitoring Boards
Slutsky, A. S., & Lavery, J. V. (2004) · New England Journal of Medicine · doi:10.1056/nejmsb033476 · Cited by 101 · see citing works
Devereaux, P. J. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1031619 · Cited by 85 · see citing works
Current controversies in data monitoring for clinical trials
Pocock, S. (2006) · Clinical Trials · doi:10.1177/1740774506073467 · Cited by 82 · see citing works
Systematic reviewers neglect bias that results from trials stopped early for benefit
Bassler, D., González, I. F., Briel, M., Cook, D. J., Devereaux, P. J., Heels-Ansdell, D., Kirpalani, H. M., Meade, M. O., Montori, V. M., Rozenberg, A., et al. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.12.006 · Cited by 63 · see citing works
Sydes, M. R., Altman, D. G., Babikera, A. B., Parmar, M. K., Spiegelhalter, D. J., & DAMOCLES group (2004) · Clinical Trials · doi:10.1191/1740774504cn003oa · Cited by 49 · see citing works
Randomized Trials Stopped Early for Benefit: A Systematic Review
Montori, V. M., Devereaux, P. J., Adhikari, N. K. J., Burns, K. E. A., Eggert, C. H., Briel, M., Lacchetti, C., Leung, T. W., Darling, E., Bryant, D. M., et al. (2005) · JAMA · doi:10.1001/jama.294.17.2203 · Cited by 26 · see citing works
How Should Results from Completed Studies Influence Ongoing Clinical Trials?
Laupacis, A., Connolly, S. J., Gent, M., Roberts, R. S., Cairns, J. A., Joyner, C. D., & CAFA Study Group (1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-115-10-818 · Cited by 21 · see citing works
Thistle, P., Spitzer, R. F., Glazier, R. H., Pilon, R., Arbess, G., Simor, A. E., Boyle, E., Chitsike, I., Chipato, T., Gottesman, M., et al. (2006) · Clinical Infectious Diseases · doi:10.1086/508869 · Cited by 21 · see citing works
The Promise of Single-Embryo Transfer
Schieve, L. A. (2006) · New England Journal of Medicine · doi:10.1056/nejme058321 · Cited by 17 · see citing works
Cyclooxygenase-2 Selective Inhibitors and Prostate Cancer: What Is the Clinical Benefit?
DuBois, R. N. (2006) · Journal of Clinical Oncology · doi:10.1200/jco.2006.05.9709 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Bassler, D. (2010) · JAMA · doi:10.1001/jama.2010.310 · Cited by 613 · see citing works
Orkin, A. M., Gill, P. J., Ghersi, D., Campbell, L., Sugarman, J., Emsley, R., Steg, P. G., Weijer, C., Simes, J., Rombey, T., et al. (2021) · JAMA · doi:10.1001/jama.2021.9941 · Cited by 299 · see citing works
Imberger, G. L., Thorlund, K., Gluud, C. N., & Wetterslev, J. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011890 · Cited by 278 · see citing works
Transarterial (chemo)embolisation for unresectable hepatocellular carcinoma
Oliveri, R. S., Wetterslev, J., & Gluud, C. N. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004787.pub2 · Cited by 266 · see citing works
Dewint, P., Hansen, B. E., Verhey, E., Oldenburg, B., Hommes, D. W., Pierik, M. J., Ponsioen, C. Y., van Dullemen, H. M., Rüssel, M. G., Bodegraven, A. A. V., et al. (2013) · Gut · doi:10.1136/gutjnl-2013-304488 · Cited by 263 · see citing works
Evidence at a glance: error matrix approach for overviewing available evidence
Keus, F., Wetterslev, J., Gluud, C. N., & van Laarhoven, C. J. H. M. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-90 · Cited by 218 · see citing works
“Therefore, the p-values of intervention effect estimates certainly are not suitable for comparison of the risk of random error between different studies [32,34-37].”
Assessment of risk of bias in randomized clinical trials in surgery
Gurusamy, K. S., Gluud, C. N., Nikolova, D., & Davidson, B. R. (2009) · British journal of surgery · doi:10.1002/bjs.6558 · Cited by 215 · see citing works
Sydes, M. R., Parmar, M., James, N. D., Clarke, N. W., Dearnaley, D. P., Mason, M. D., Morgan, R. C., Sanders, K., & Royston, P. J. (2009) · Trials · doi:10.1186/1745-6215-10-39 · Cited by 150 · see citing works
“Many instances of early stopping for benefit appear to occur on a random high [27,28] and the credibility and interpretability of a trial can be adversely affected: caution should be applied in this respect.”
BackgroundBayesian statistics for clinical research
Goligher, E. C., Heath, A., & Harhay, M. O. (2024) · The Lancet · doi:10.1016/s0140-6736(24)01295-9 · Cited by 141 · see citing works
Majumdar, S. R., Kjellstrand, C. M., Tymchak, W. J., Hervas‐Malo, M., Taylor, D. A., & Teo, K. K. (2009) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2009.03.024 · Cited by 123 · see citing works
Chamberlain, P., Bradley, A., Arumugam, B., Hammond, D., Mcnally, J. J., Logan, N. S., JONES, D. A., Ngo, C. S., Peixoto-de-Matos, S. C., Hunt, C., et al. (2022) · Optometry and Vision Science · doi:10.1097/opx.0000000000001873 · Cited by 120 · see citing works
Kirkpatrick, A., Coccolini, F., Ansaloni, L., Roberts, D., Tolonen, M., Mckee, J. L., Leppäniemi, A., Faris, P., Doig, C., Catena, F., et al. (2018) · World Journal of Emergency Surgery · doi:10.1186/s13017-018-0183-4 · Cited by 88 · see citing works
Systematic Reviews of Anesthesiologic Interventions Reported as Statistically Significant
Imberger, G. L., Gluud, C. N., Boylan, J. F., & Wetterslev, J. (2015) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000000892 · Cited by 101 · see citing works
Atorvastatin versus placebo in patients with covid-19 in intensive care: randomized controlled trial
INSPIRATION-S Investigators (2022) · BMJ · doi:10.1136/bmj-2021-068407 · Cited by 84 · see citing works
“30 The data and safety monitoring board had independent access to the study data and conducted prespecified interim safety analyses at 25%, 50%, and 75% of recruitment.”
When to keep it simple – adaptive designs are not always useful
Wason, J. M. S., Brocklehurst, P., & Yap, C. (2019) · BMC Medicine · doi:10.1186/s12916-019-1391-9 · Cited by 83 · see citing works
Bikdeli, B., Talasaz, A. H., Rashidi, F., Sharif-Kashani, B., Farrokhpour, M., Bakhshandeh, H., Sezavar, S. H., Dabbagh, A., Beigmohammadi, M. T., Payandemehr, P., et al. (2020) · Thrombosis Research · doi:10.1016/j.thromres.2020.09.027 · Cited by 82 · see citing works
“To provide the most informative results, without risking the possibility of an overinflated estimate of the effect size, the steering committee decided not to set pre-specified criteria for early termination for efficacy [41].”
BackgroundMultimorbidity: a challenge for evidence-based medicine
Campbell-Scherer, D. L. (2010) · BMJ Evidence-Based Medicine · doi:10.1136/ebm1154 · Cited by 81 · see citing works
Why marketplace experimentation is harder than it seems
Blake, T., & Coey, D. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602837 · Cited by 76 · see citing works
Stopping clinical trials early for benefit: impact on estimation
Freidlin, B., & Korn, E. L. (2009) · Clinical Trials · doi:10.1177/1740774509102310 · Cited by 76 · see citing works
Naci, H., & Ioannidis, J. P. A. (2014) · The Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev-pharmtox-010814-124614 · Cited by 68 · see citing works
Expanding the Orbit of Primary Prevention — Moving beyond JUPITER
Hlatky, M. A. (2008) · New England Journal of Medicine · doi:10.1056/nejme0808320 · Cited by 69 · see citing works
Completion and Publication Rates of Randomized Controlled Trials in Surgery
Rosenthal, R., Kasenda, B., Dell‐Kuster, S., von Elm, E. B., You, J. J., Blümle, A., Tomonaga, Y., Saccilotto, R. T., Amstutz, A., Bengough, T., et al. (2014) · Annals of Surgery · doi:10.1097/sla.0000000000000810 · Cited by 67 · see citing works
Duodenal Infusion of Feces for Recurrent Clostridium difficile
Schooneveld, T. C. V., Gross, A., & Kalil, A. C. (2013) · New England Journal of Medicine · doi:10.1056/nejmc1303919 · Cited by 67 · see citing works
Rivaroxaban Prophylaxis in Noncirrhotic Portal Vein Thrombosis
Plessier, A., Goria, O., Cervoni, J. P., Ollivier‐Hourmand, I., Bureau, C., Poujol‐Robert, A., Minello, A., Houssel‐Debry, P., Rautou, P., Payancé, A., et al. (2022) · NEJM Evidence · doi:10.1056/evidoa2200104 · Cited by 60 · see citing works
Stevely, A. K., Dimairo, M., Todd, S., Julious, S. A., Nicholl, J. P., Hind, D., & Cooper, C. L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0141104 · Cited by 58 · see citing works
“Although much attention has been paid to group sequential RCTs that are stopping early for benefit [62,63,66,68], the consequences could be similar when trials are stopped early for futility, since the evidence can be used to withdraw intervention(s) already in the care pathway.”
Photo induced antibacterial activity of CeO2/GO against wound pathogens
Sharma, G., Prema, D., Venkataprasanna, K. S., Prakash, J., Sahabuddin, S., & Venkatasubbu, G. D. (2020) · Arabian Journal of Chemistry · doi:10.1016/j.arabjc.2020.09.004 · Cited by 56 · see citing works
Hartling, L., Wittmeier, K. D. M., Caldwell, P. H. Y., van der Lee, H., Klassen, T. P., Craig, J. C., Offringa, M., & for the StaR Child Health Group (2012) · PEDIATRICS · doi:10.1542/peds.2012-0055c · Cited by 54 · see citing works
Korn, E. L., Freidlin, B., & Mooney, M. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2008.19.5339 · Cited by 49 · see citing works
Is acute respiratory distress syndrome an iatrogenic disease?
Villar, J., & Slutsky, A. S. (2010) · Critical Care · doi:10.1186/cc8842 · Cited by 46 · see citing works
“Early stopping tends to overestimate treatment eff ects [12,13]; this is particularly true for studies with low event rates, as was the case in this study, in which only 12 patients in total developed ALI.”
ResultHumbert, P. G., Faivre, B., Véran, Y., Debure, C., Truchetet, F., Bécherel, P., Plantin, P., Kerihuel, J., Eming, S. A., Dissemond, J., et al. (2014) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.12400 · Cited by 45 · see citing works
Fernandes, R. M. F., van der Lee, J. H., & Offringa, M. (2009) · BMC Pediatrics · doi:10.1186/1471-2431-9-77 · Cited by 107 · see citing works
“This implies knowing which, if any, statistical monitoring methods and stopping rules or guidelines were used, for which outcome (whether primary or secondary), and at which stage of the trial[11,13] It may be relevant in some cases to check if the results were corrected for multiplicity[147] The ongoing controversies…”
InfluentialNarayan, V. M., Gandhi, S., Chrouser, K. L., Evaniew, N., & Dahm, P. (2018) · British Journal of Urology · doi:10.1111/bju.14210 · Cited by 42 · see citing works
Stopping guidelines for an effectiveness trial: what should the protocol specify?
Tyson, J. E., Pedroza, C., Wallace, D., D’Angio, C. T., Bell, E. F., & Das, A. (2016) · Trials · doi:10.1186/s13063-016-1367-4 · Cited by 41 · see citing works
“Despite long-standing problems in decisions to stop clinical trials [1–6], stopping guidelines are often vague or unspecified in the trial protocol.”
BackgroundMechanical Thrombectomy vs Bridging Therapy for Anterior Circulation Large Vessel Occlusion Stroke
Trifan, G., Biller, J., & Testai, F. D. (2022) · Neurology · doi:10.1212/wnl.0000000000200029 · Cited by 39 · see citing works
Magnitude of effects in clinical trials published in high-impact general medical journals
Siontis, K., Εvangelou, Ε., & Ioannidis, J. P. A. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr095 · Cited by 38 · see citing works
Maitland, K., Kiguli, S., Opoka, R. O., Olupot‐Olupot, P., Engoru, C., Njuguna, P., Bandika, V., Mpoya, A., Bush, A., Williams, T. N., et al. (2018) · Wellcome Open Research · doi:10.12688/wellcomeopenres.12747.2 · Cited by 38 · see citing works
Anand, V., Scales, D. C., Parshuram, C. S., & Kavanagh, B. P. (2014) · Intensive Care Medicine · doi:10.1007/s00134-014-3250-7 · Cited by 37 · see citing works
“the magnitude of the apparent treatment benefit) is frequently overestimated, as has been described in cases of premature trial termination for apparent benefit [17].”
BackgroundYounus, J., Verma, S., Franek, J. A., Coakley, N., & the Sarcoma Disease Site Group of Cancer Care Ontario’s Program in Evidence-Based Care (2010) · Current Oncology · doi:10.3747/co.v17i4.560 · Cited by 36 · see citing works
“Early termination of clinical trials due to benefit often overestimates overall treatment effect as such trials tend to be on a ―random high‖ with subsequent followup data from the same or similar trials showing ―regression to the truth‖ ( 11-14 ).”
BackgroundEarly Stopping of Clinical Trials: Charting the Ethical Terrain
Malmqvist, E., Juth, N., Lynöe, N., & Helgesson, G. (2011) · Kennedy Institute of Ethics journal · doi:10.1353/ken.2011.0002 · Cited by 33 · see citing works
“Many believe that early analyses of small amounts of data are prone to exaggerating the difference between the treatments (Pocock 1992; Montori et al. 2005; Mueller et al. 2007; Bassler et al. 2008).”
BackgroundMaitland, K., Kiguli, S., Opoka, R. O., Olupot‐Olupot, P., Engoru, C., Njuguna, P., Bandika, V., Mpoya, A., Bush, A., Williams, T. N., et al. (2017) · Wellcome Open Research · doi:10.12688/wellcomeopenres.12747.1 · Cited by 33 · see citing works
Dimairo, M., Julious, S. A., Todd, S., Nicholl, J. P., & Boote, J. D. (2015) · Trials · doi:10.1186/s13063-015-1119-x · Cited by 32 · see citing works
“For instance, suboptimal reporting of appropriate statistical methods to obtain unbiased trial results (point estimates, CIs and P-values) following an AD may influence consumers of research findings to view them with suspicion [23 – 25].”
Meta‐analysis of clinical trials with early stopping: an investigation of potential bias
Schou, I. M., & Marschner, I. C. (2013) · Statistics in Medicine · doi:10.1002/sim.5893 · Cited by 32 · see citing works
Pedroza, C., Tyson, J. E., Das, A., Laptook, A. R., Bell, E. F., & Shankaran, S. (2016) · Trials · doi:10.1186/s13063-016-1480-4 · Cited by 32 · see citing works
Stegert, M., Kasenda, B., von Elm, E., You, J. J., Blümle, A., Tomonaga, Y., Saccilotto, R., Amstutz, A., Bengough, T., Briel, M., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.023 · Cited by 30 · see citing works
van den Bogert, C. A., Souverein, P. C., Brekelmans, C. T. M., Janssen, S. W. J., Koëter, G. H., Leufkens, H. G. M., & Bouter, L. M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.001 · Cited by 31 · see citing works
The Impact of Clinical Trials Conducted by Research Networks in Pediatric Critical Care
Choong, K. L. M., Duffett, M., Cook, D. J., & Randolph, A. G. (2016) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000000835 · Cited by 27 · see citing works
“Effects may be overestimated when trials are stopped early for benefit, and some recommend against early stopping for this reason (31).”
BackgroundThe unrestricted global effort to complete the COOL trial
Kirkpatrick, A., Coccolini, F., Tolonen, M., Minor, S. F., Catena, F., Gois, E., Doig, C., Hill, M. D., Ansaloni, L., Chiarugi, M., et al. (2023) · World Journal of Emergency Surgery · doi:10.1186/s13017-023-00500-z · Cited by 17 · see citing works
Stopping Randomized Trials Early for Benefit and Estimation of Treatment Effects
Bassler, D., Briel, M., Montori, V. M., Glasziou, P., Zhou, Q. X., Heels‐Ansdell, D., Walter, S. D., & Guyatt, G. H. (2012) · Cited by 24 · see citing works
Standard 3: Data Monitoring Committees
Ellenberg, S., Fernandes, R., Saloojee, H., Bassler, D., Askie, L., Vandermeer, B., Offringa, M., van der Tweel, I., Altman, D., & van der Lee, J. H. (2012) · Pediatrics · doi:10.1542/peds.2012-0055f · Cited by 23 · see citing works
Paul, M., Ram, R., Kugler, E., Farbman, L., Peck, A., Leibovici, L., Lahav, M., Yeshurun, M., Shpilberg, O., Herscovici, C., et al. (2013) · American Journal of Hematology · doi:10.1002/ajh.23622 · Cited by 22 · see citing works
“Trials stopped early for benefit or harm have been shown to systematically exaggerate effects [17].”
BackgroundBalme, K., Zar, H. J., Swift, D. K., & Mann, M. D. (2015) · Clinical Toxicology · doi:10.3109/15563650.2015.1059943 · Cited by 20 · see citing works
Hoff, W. V., & Offringa, M. (2014) · Archives of Disease in Childhood · doi:10.1136/archdischild-2012-303094 · Cited by 17 · see citing works
Data monitoring committees, interim analysis and early termination in paediatric trials
Fernandes, R., van der Lee, J. H., & Offringa, M. (2011) · Acta paediatrica · doi:10.1111/j.1651-2227.2011.02282.x · Cited by 20 · see citing works
“Many trials were terminated for efficacy or futility, which may place them at a higher risk of having biased results (13,15).”
BackgroundInfluentialOptimal Therapy in Hypertensive Subjects with Diabetes Mellitus
Reboldi, G., Gentile, G., Angeli, F., & Verdecchia, P. (2011) · Current Atherosclerosis Reports · doi:10.1007/s11883-011-0160-9 · Cited by 17 · see citing works
“Therefore, the excellent results of the benazepril/amlodipine combination may potentially represent an overestimation of reality, as has been proven to be the case with early truncation of trials [79].”
Kennedy, A. P., Phillips, K. A., Epstein, D., Reamer, D., Schmittner, J., & Preston, K. (2012) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2012.10.025 · Cited by 17 · see citing works
“Trials stopped early can produce false-positive findings (Bassler et al., 2008), and we cannot rule out the possibility that we would have had different results if we had completed our trial.”
ResultEdwards, S. (2010) · Research Ethics · doi:10.1177/174701611000600207 · Cited by 15 · see citing works
“However, this practice has been criticised for leading to false positive results and some trials have even be repeated at a later date to redress the issue of a small sample size [8].”
BackgroundPublishing interim results of randomised clinical trials in peer-reviewed journals
Counsell, N., Biri, D., Fraczek, J., & Hackshaw, A. K. (2016) · Clinical Trials · doi:10.1177/1740774516664689 · Cited by 15 · see citing works
“Disease area Oncology 38 (60) Neurology 9 (14) Cardiology 6 (10) Osteoporosis 3 (5) Gastroenterology 2 (3) Gynaecology 2 (3) Infectious diseases 2 (3) Respiratory 1 (2) Type of sponsor Commercial 17 (27) Non-commercial 46 (73) Commercial funding support received No 33 (52) Yes 30 (48) Subjects blinded No 39 (62) Yes 2…”
BackgroundProtecting Clinical Trial Participants and Protecting Data Integrity: Are We Meeting the Challenges?
Ellenberg, S. S. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001234 · Cited by 15 · see citing works
“A recent variation on this argument is that early termination of trials should be avoided because it leads to artificially elevated estimates of treatment effect; trials that stop early, of course, show higher estimates of effect than trials of the same regimens that do not stop early [33,34].”
BackgroundJiang, Y., Zhao, W., & Durkalski‐Mauldin, V. L. (2019) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2019.1607368 · Cited by 15 · see citing works
“It is known that observed treatment effect sizes are inflated when trials stop early for overwhelming efficacy (Bassler et al. 2008; Freidlin et al. 2009).”
BackgroundResultBubble CPAP for pneumonia: perils of stopping trials early
Shann, F. A., & Lange, T. (2015) · The Lancet · doi:10.1016/s0140-6736(15)60691-2 · Cited by 14 · see citing works
Vaccarino, V., Bremner, J. D., & Kelley, M. E. (2009) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.109.850404 · Cited by 14 · see citing works
Rizzo, M., Berneis, K. K., Spinas, G. A., Rini, G. B., & Kapur, N. K. (2009) · International Journal of Clinical Practice · doi:10.1111/j.1742-1241.2008.01979.x · Cited by 14 · see citing works
A new anti-cancer drug in the market: Good news for investors or for patients?
Apolone, G., Tafuri, G., Trotta, F., & Garattini, S. K. (2008) · European Journal of Cancer · doi:10.1016/j.ejca.2008.06.021 · Cited by 14 · see citing works
Hartling, L., Wittmeier, K. D. M., Caldwell, P. H. Y., van der Lee, J. H., Klassen, T. P., Craig, J. C., & Offringa, M. (2011) · Clinical Pharmacology & Therapeutics · doi:10.1038/clpt.2011.212 · Cited by 13 · see citing works
Voglibose for prevention of type 2 diabetes mellitus
Scheen, A. J. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60575-4 · Cited by 10 · see citing works
The JUPITER Trial: Myth or Reality?
Morrissey, R. P., Diamond, G. A., & Kaul, S. (2011) · Current Atherosclerosis Reports · doi:10.1007/s11883-011-0197-9 · Cited by 11 · see citing works
“Truncated trials are well known to be associated with overestimation of treatment benefits compared with non-truncated trials due to chance (“random variation”), which drift back towards the null (“regress to the mean”) with continued follow-up [12, 20, 32].”
BackgroundSurveying risk subjects: Public health surveys as instruments of biomedicalization
Norton, A. T. (2013) · BioSocieties · doi:10.1057/biosoc.2013.20 · Cited by 9 · see citing works
Interim monitoring of efficacy data is important and appropriate
Sydes, M. R., & Parmar, M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.08.002 · Cited by 9 · see citing works
Sargent, D. J. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2008.20.8611 · Cited by 9 · see citing works
“As expected, the trials had a wide variety of primary end points; in 70% of the trials, accrual was complete at the time of closing, which has been recommended by some as an important criterion before trial reporting should be considered.(6) In only one case were data released before at least 25% of the protocol speci…”
ResultBesonderheiten von Onkologischen Studien im Rahmen der frühen Nutzenbewertung von Arzneimitteln
Lange, S. (2015) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2015.08.003 · Cited by 8 · see citing works
Early stopping rules in oncology: Considerations for clinicians
Mukherjee, S. D., Goffin, J. R., Taylor, V. H., Anderson, K. K., & Pond, G. R. (2011) · European Journal of Cancer · doi:10.1016/j.ejca.2011.05.019 · Cited by 8 · see citing works
Shimura, M., Maruo, K., & Gosho, M. (2018) · Pharmaceutical Statistics · doi:10.1002/pst.1859 · Cited by 7 · see citing works
Das, S., & Xaviar, S. (2019) · Epilepsy Research · doi:10.1016/j.eplepsyres.2019.106258 · Cited by 7 · see citing works
Muir, O. S., Shinozuka, K., Beutler, B. D., Arenas, A., Cherian, K. N., Evans, V. D., Fasano, C., & Tabaac, B. J. (2024) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001720 · Cited by 6 · see citing works
“27 Psychedelic therapies require significant postmarketing surveillance to understand the complete adverse effect profile and variability between specific population cohorts.”
BackgroundHow to Distinguish between Statistically Significant Results and Clinically Relevant Results
Bennett, D. (2016) · Monographs in clinical neuroscience/Frontiers of neurology and neuroscience/Monographs in neural sciences · doi:10.1159/000445411 · Cited by 6 · see citing works
Culbertson, M. J., McCole, D., & McNamara, P. E. (2014) · Journal of Development Effectiveness · doi:10.1080/19439342.2014.919339 · Cited by 6 · see citing works
“However, this strategy should be used with caution (Montori et al. 2005; Bassler et al. 2008).”
Trial Termination and Drug Misclassification in Sequential Adaptive Clinical Trials
Rojas-Cordova, A. C., & Hosseinichimeh, N. (2018) · Service Science · doi:10.1287/serv.2018.0217 · Cited by 6 · see citing works
Evidence-Based Medicine - A Paradigm Ready To Be Challenged?
Schulte, M.-C. (2020) · J.B. Metzler eBooks · doi:10.1007/978-3-476-05703-7 · Cited by 6 · see citing works
Reducing overestimation of the treatment effect by interim analysis when designing clinical trials
Shimura, M. (2018) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/jcpt.12777 · Cited by 5 · see citing works
The new US and European guidelines in hypertension: A multi-dimensional analysis
Fernandes, M., & Rikkert, M. G. M. O. (2019) · Contemporary Clinical Trials · doi:10.1016/j.cct.2019.04.008 · Cited by 5 · see citing works
Published intimate partner violence studies often differ from their trial registration records
Madden, K., Tai, K., Ali, Z., Schneider, P., Singh, M., Ghert, M. A., & Bhandari, M. (2017) · Women & Health · doi:10.1080/03630242.2017.1421287 · Cited by 4 · see citing works
Optimal Patient Enrollment in Sequential Adaptive Clinical Trials with Binary ResponsePreprint
Rojas-Cordova, A. C., & Bish, E. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3234590 · Cited by 5 · see citing works
Lawson, D. O., Mellor, K., Eddy, S., Lee, C., Kim, K. H., Kim, K., Mbuagbaw, L. C. E., & Thabane, L. (2022) · PubMed · doi:10.1097/phm.0000000000001797 · Cited by 4 · see citing works
Vasopressors and the search for the optimal trial design
Maharaj, R. (2011) · Contemporary Clinical Trials · doi:10.1016/j.cct.2011.07.010 · Cited by 4 · see citing works
“Recent reviews have shown that RCTs stopped early for benefit may be associated with greater effect size than those RCT not stopped early [38–41].”
BackgroundOxygen therapy in acute resuscitation
Frei, D. R., & Young, P. J. (2018) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000000549 · Cited by 4 · see citing works
Monitoring in clinical trials: benefit or bias?
Nardini, C. (2013) · Metamedicine · doi:10.1007/s11017-013-9264-2 · Cited by 4 · see citing works
“[9], the effect magnitude is considered ‘‘implausible’’ with respect to normal expectations in the medical community about treatment effectiveness.”
BackgroundResultWhen should an interim analysis be unblinded to the data monitoring committee?
Knottnerus, J. A., & Spigt, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.07.003 · Cited by 4 · see citing works
Marschner, I. C., Askie, L. M., & Schou, I. M. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1394 · Cited by 4 · see citing works
“Some researchers have argued that there is the potential for overestimation of treatment effects in RCTs that stopped early for benefit, which are often referred to as truncated studies.(5) Of particular concern has been the impact that truncation may have on evidence synthesis when conducting metaanalyses.”
BackgroundTherapeutic hypothermia and the ‘equal air‐time’ solution for controversial randomised trials
Wilkinson, D. J. C. (2010) · Journal of Paediatrics and Child Health · doi:10.1111/j.1440-1754.2010.01864.x · Cited by 4 · see citing works
Schwarzer, R., Schnell‐Inderst, P., Grabein, K., Göhler, A., Stollenwerk, B., Grandi, N. C., Klauß, V., Wasem, J., & Siebert, U. (2009) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2009.05.020 · Cited by 4 · see citing works
Rojas-Cordova, A. C., Bish, E. K., & Hosseinichimeh, N. (2019) · Women in engineering and science · doi:10.1007/978-3-030-11866-2_14 · Cited by 3 · see citing works
Early termination of clinical trials in paediatrics
Camara, C. (2020) · British Journal of Nursing · doi:10.12968/bjon.2020.29.14.814 · Cited by 3 · see citing works
Promoting Women’s Participation in Maternity Care in Lebanon
Kabakian‐Khasholian, T., Shayboub, R., Kanaan, M. N., Mahfoud, Z. R., & Kak, F. E. (2015) · International Journal of Childbirth · doi:10.1891/2156-5287.5.3.166 · Cited by 3 · see citing works
Rosuvastatin, C-reactive protein, LDL cholesterol, and the JUPITER trial
Bloom, J. M. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61232-0 · Cited by 3 · see citing works
Rosuvastatin, C-reactive protein, LDL cholesterol, and the JUPITER trial
Rosenstein, R. S., & Parra, D. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61230-7 · Cited by 3 · see citing works
Fleming, S., Mwandigha, L., & Fanshawe, T. R. (2024) · Diagnostic and Prognostic Research · doi:10.1186/s41512-024-00174-4 · Cited by 2 · see citing works
“A study may also be terminated early if there is sufficient interim evidence that it is very likely to yield a clinically useful result (termination for efficacy), although this practice has been criticised as likely to overestimate effect sizes [13].”
BackgroundMethodische Anforderungen an klinische Studien und ihre Interpretation
Windeler, J., & Lange, S. (2009) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-009-0826-4 · Cited by 2 · see citing works
González‐Padilla, D. A., Subiela, J. D., & Villacampa, F. (2024) · Translational Andrology and Urology · doi:10.21037/tau-23-563 · Cited by 2 · see citing works
“While not directly applicable to this context, it is worth noting that early termination for benefit in randomized controlled trials has historically presented challenges in the interpretation of the data showing implausibly large treatment effects (11).”
The latest therapeutic advances with spontaneous intracerebral hemorrhage
Griffin, M., Kwok, C. S., Qureshi, A. I., & Lip, G. Y. H. (2025) · Expert Review of Neurotherapeutics · doi:10.1080/14737175.2025.2502048 · Cited by 1 · see citing works
“Stopping trials early is known to increase the chance of overestimation of treatment benefit [20,21].”
- Authors
- Bassler, D., Montori, V. M., Briel, M., Glasziou, P., & Guyatt, G.
- Affiliations
- University Children's Hospital Tübingen (DE)
- University of Basel (CH)
- University of Oxford (GB)
- McMaster University (CA)
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2008-03
- Volume
- 61
- Issue
- 3
- Pages
- 241-246
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W2042803707
- Cited by
- 130
- References
- 38
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 7 · 2023: 3 · 2022: 5 · 2021: 3 · 2020: 6 · 2019: 6 · 2018: 9 · 2017: 4 · 2016: 7 · 2015: 9 · 2014: 9 · 2013: 8 · 2012: 9
- FWCI
- 1.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Funders
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Medical Research Council
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- early stopping, randomized controlled trials, interim analysis, overestimation of treatment effects, statistical power, p-value threshold, trial ethics
- MeSH terms
- Decision Making, Humans, Motivation, Bias, Randomized Controlled Trials as Topic, Treatment Outcome, Ethics, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
