Threats to Applicability of Randomised Trials: Exclusions and Selective Participation
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Background: Although the randomised controlled trial (RCT) is regarded as the ‘gold standard’ in terms of evaluating the effectiveness of interventions, it is susceptible to challenges to its external validity if those participating are unrepresentative of the reference population for whom the intervention in question is intended. In the past, reporting on numbers and types of potential subjects that have been excluded by design, and centres, clinicians or patients that have elected not to participate, has generally been poor, and the threat to inference posed by possible selection bias is unclear. Methods: A systematic review was undertaken, based largely on MEDLINE and EMBASE with follow-up of cited references, to assess the extent, nature and importance of excluding potential subjects or the unwillingness of particular centres, clinicians or patients to participate. Results: RCTs vary widely in the extent to which potential future recipients of treatment are included. The reasons cited for excluding certain categories of patient may be medical or scientific. Medical reasons include a high risk of adverse effects and the belief that benefit will be relatively small or absent (or has already been established) in the groups in question. Scientific reasons include more precise estimates of treatment effect because of a relatively homogeneous sample and the reduction of potential bias by excluding those individuals most likely to be lost to follow-up. Many RCTs have blanket exclusions, such as the elderly, women and ethnic minorities, but reasons for these exclusions are seldom given. Evaluative research is undertaken predominantly in university or teaching centres. Non-randomised studies are more likely than RCTs to include non-teaching centres. The effect of patient non-participation appears to depend on whether the RCT is concerned with treatment of an existing condition or with disease prevention. Participants in treatment trials tend to be more severely ill than those who do not participate. In contrast, those who participate in prevention trials are more likely to have adopted a healthy lifestyle than those who decline. Most evaluative studies fail to document adequately the characteristics of those who, while eligible, do not participate. However, subjects included in RCTs (i.e. eligible and participating) tend to have a different prognosis than patients identified from clinical databases. Conclusions: Narrow inclusion criteria may offer benefits such as increased precision and reduced loss to follow-up, but there are important disadvantages, such as uncertainty about extrapolation of results, which may result in denial of effective treatment to groups who might benefit, and delay in obtaining definitive results because of reduced recruitment rate. Selective participation by teaching centres and sicker patients in treatment RCTs may exaggerate the measured treatment effect. Prevention trials, on the other hand, may underestimate effects as participants have less capacity to benefit.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Blood pressure, stroke, and coronary heart disease
Collins, R., Peto, R., MacMahon, S., Godwin, J., Qizilbash, N., Collins, R., MacMahon, S., Hebert, P., Eberlein, K. A., Taylor, J. O., et al. (1990) · The Lancet · doi:10.1016/0140-6736(90)90944-z · Cited by 4,491
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
The Exclusion of the Elderly and Women From Clinical Trials in Acute Myocardial Infarction
Gurwitz, J. H. (1992) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1992.03490110055029 · Cited by 532
Paradise, J. L., Bluestone, C. D., Bachman, R. Z., Colborn, D. K., Bernard, B. S., Taylor, F. H., Rogers, K. D., Schwarzbach, R. H., Stool, S. E., Friday, G. A., et al. (1984) · New England Journal of Medicine · doi:10.1056/nejm198403153101102 · Cited by 518
Coronary artery surgery study (CASS): A randomized trial of coronary artery bypass surgery
(1984) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(84)80437-4 · Cited by 439
Cholesterol lowering and mortality: the importance of considering initial level of risk.
Smith, G. D., Song, F., & Sheldon, T. A. (1993) · BMJ · doi:10.1136/bmj.306.6889.1367 · Cited by 423
Taylor, K. M., Margolese, R. G., & Soskolne, C. L. (1984) · New England Journal of Medicine · doi:10.1056/nejm198405243102106 · Cited by 407
Effect of propranolol after acute myocardial infarction in patients with congestive heart failure.
Chadda, K., Goldstein, S., Byington, R., & Curb, J. D. (1986) · Circulation · doi:10.1161/01.cir.73.3.503 · Cited by 403
An evidence based approach to individualising treatment
Glasziou, P. P., & Irwig, L. M. (1995) · BMJ · doi:10.1136/bmj.311.7016.1356 · Cited by 391
Barker, F. G., & Ogilvy, C. S. (1996) · Journal of Neurosurgery · doi:10.3171/jns.1996.84.3.0405 · Cited by 358
Choosing between randomised and non-randomised studies: a systematic review.
Britton, McKee, Black, McPherson, Sanderson, & Bain (1998) · Health Technology Assessment · doi:10.3310/hta2130 · Cited by 308
The effectiveness of surgery for stress incontinence in women: a systematic review
Black, N. A., & Downs, S. H. (1996) · British Journal of Urology · doi:10.1046/j.1464-410x.1996.01422.x · Cited by 261
Self-citationRecruitment experience in clinical trials: Literature summary and annotated bibliography
Hunninghake, D. B., Darby, C. A., & Probstfield, J. L. (1987) · Controlled Clinical Trials · doi:10.1016/0197-2456(87)90004-3 · Cited by 251
(2002) · European Journal of Cancer · doi:10.1016/s0959-8049(02)00637-8 · Cited by 232
Applying results of randomised trials to clinical practice: impact of losses before randomisation.
Charlson, M. E., & Horwitz, R. I. (1984) · BMJ · doi:10.1136/bmj.289.6454.1281 · Cited by 225
Developing improved observational methods for evaluating therapeutic effectiveness
Horwitz, R. I., Viscoli, C. M., Clemens, J. D., & Sadock, R. T. (1990) · The American Journal of Medicine · doi:10.1016/0002-9343(90)90182-d · Cited by 204
Selection of Patients with Acute Myocardial Infarction for Thrombolytic Therapy
Muller, D. W. M., & Topol, E. J. (1990) · Annals of Internal Medicine · doi:10.7326/0003-4819-113-12-949 · Cited by 197
SCHMOOR, C., OLSCHEWSKI, M., & SCHUMACHER, M. (1996) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19960215)15:3<263::aid-sim165>3.0.co;2-k · Cited by 165
Representation of American Blacks in Clinical Trials of New Drugs
Svensson, C. K. (1989) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1989.03420020117041 · Cited by 163
Accrual to cancer clinical trials: Directions from the research literature
Gotay, C. C. (1991) · Social Science & Medicine · doi:10.1016/0277-9536(91)90214-w · Cited by 162
Hunter, C. P., Rw, F., Ar, F., Bavier, A. R., Wh, D., Ford, L., Henson, D. E., Macfarlane, D. K., Cr, S., & Yancik, R. (1987) · PubMed · Cited by 129
Yusuf, S., Held, P., Teo, K. K., & Toretsky, E. R. (1990) · Statistics in Medicine · doi:10.1002/sim.4780090114 · Cited by 121
Harth, S. C., & Thong, Y. H. (1990) · BMJ · doi:10.1136/bmj.300.6736.1372 · Cited by 116
Breast Cancer Therapy: Exercising All Our Options
DeVita, V. T. (1989) · New England Journal of Medicine · doi:10.1056/nejm198902233200812 · Cited by 112
Underutilization of thrombolytic therapy in eligible women with acute myocardial infarction
Maynard, C. C., Althouse, R., Cerqueira, M. D., Olsufka, M., & Kennedy, J. W. (1991) · The American Journal of Cardiology · doi:10.1016/0002-9149(91)90791-i · Cited by 106
Are Randomized Controlled Trials Controlled? Patient Preferences and Unblind Trials
Mcpherson, K., Britton, A. R., & Wennberg, J. E. (1997) · Journal of the Royal Society of Medicine · doi:10.1177/014107689709001205 · Cited by 99
Self-citationA Bayesian Analysis of Institutional Effects in a Multicenter Cancer Clinical Trial
Gray, R. J. (1994) · Biometrics · doi:10.2307/2533216 · Cited by 95
The Recruitment of Research Participants: A Review
Holden, G. W., Rosenberg, G. A., Barker, K., Tuhrim, S., & Brenner, B. L. (1993) · Social Work in Health Care · doi:10.1300/j010v19n02_01 · Cited by 88
Montague, T. J., Ikuta, R. M., Wong, R., Bay, K. S., Teo, K. K., & Davies, N. J. (1991) · The American Journal of Cardiology · doi:10.1016/0002-9149(91)90397-4 · Cited by 84
Yeomans-Kinney, A., Vernon, S. W., Frankowski, R. F., Weber, D. M., Bitsura, J. M., & Vogel, V. G. (1995) · Cancer · doi:10.1002/1097-0142(19950701)76:1<46::aid-cncr2820760107>3.0.co;2-b · Cited by 69
Brand, R., & Kragt, H. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111605 · Cited by 64
Gorkin, L., Schron, E. B., Handshaw, K., Shea, S., Kinney, M. R., Branyon, M., Campion, J., Bigger, J. T., Sylvia, S. C., Duggan, J., et al. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00089-5 · Cited by 61
Cholesterol reduction and its impact on coronary artery disease and total mortality
Holme, I. (1995) · The American Journal of Cardiology · doi:10.1016/s0002-9149(99)80465-2 · Cited by 60
Cooperative groups and community hospitals. Measurement of impact in the community hospitals
Begg, C. B., Zelen, M., Carbone, P. P., Mcfadden, E. T., Brodovsky, H. S., Engstrom, P., Hatfield, A. K., Ingle, J. N., Schwartz, B., & Stolbach, L. L. (1983) · Cancer · doi:10.1002/1097-0142(19831101)52:9<1760::aid-cncr2820520934>3.0.co;2-r · Cited by 49
Recruitment and Retention in a Clinical Trial for Low Birth Weight, Premature Infants
CONSTANTINE, W. L., HAYNES, C. W., Spiker, D., Kendall-Tackett, K. A., & Constantine, N. A. (1993) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/00004703-199302000-00001 · Cited by 49
Bypass Angioplasty Revascularization Investigation (BARI): Baseline Clinical and Angiographic Data
ROGERS, W. (1995) · The American Journal of Cardiology · doi:10.1016/s0002-9149(99)80390-7 · Cited by 48
for the British Stomach Cancer Group*, Ward, L., Fielding, J., Dunn, J., & Kelly, K. (1992) · British Journal of Cancer · doi:10.1038/bjc.1992.390 · Cited by 39
Factors Affecting Sample Selection in a Randomized Trial of Balance Enhancement: The FICSIT Study
Pacala, J. T., Judge, J. O., & Boult, C. (1996) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.1996.tb06405.x · Cited by 37
Frequency of inclusion of patients with cardiogenic shock in trials of thrombolytic therapy
Col, N. F., Gurwitz, J. H., Alpert, J. S., & Goldberg, R. J. (1994) · The American Journal of Cardiology · doi:10.1016/0002-9149(94)90206-2 · Cited by 37
Stukenborg, G. J. (1997) · Archives of Neurology · doi:10.1001/archneur.1997.00550190018009 · Cited by 35
Generalizing from clinical trials
Davis, C. (1994) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)90023-x · Cited by 34
Chapter 14: Self-Help Group Participation Among People with Severe Mental Illness
Kaufmann, C. L., Schulberg, H. C., & Schooler, N. R. (1995) · Journal of Prevention & Intervention in the Community · doi:10.1300/j293v11n02_06 · Cited by 28
Davies, G., Pyke, S. D. M., & Kinmonth, A. L. (1994) · BMJ · doi:10.1136/bmj.309.6968.1553 · Cited by 26
Informed Consent: Influence on Patient Selection Among Critically III Premature Infants
Walterspiel, J. N. (1990) · PEDIATRICS · doi:10.1542/peds.85.1.119 · Cited by 24
The sodium-potassium blood pressure trial in children
Gómez-Marı́n, O., Prineas, R. J., & Sinaiko, A. R. (1991) · Controlled Clinical Trials · doi:10.1016/0197-2456(91)90020-m · Cited by 23
Ethical Implications of Rejecting Patients for Clinical Trials
Chalmers, T. C. (1990) · JAMA · doi:10.1001/jama.1990.03440060111043 · Cited by 19
Competing or Complementary?:Ethical Considerations and the Quality of Randomized Trials
Lumley, J., & Bastian, H. (1996) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300009600 · Cited by 18
Williford, W. O., Krol, W. F., & Buzby, G. P. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90170-6 · Cited by 16
Näslund, G. K., Fredrikson, M. G., Hellénius, M.-L., & Fairé, U. D. (1994) · Scandinavian Journal of Primary Health Care · doi:10.3109/02813439409029249 · Cited by 13
van Bergen, P. F. M. M., Jonker, J. J. C., Molhoek, G. P., van der Burgh, P. H., van Domburg, R. T., Deckers, J. W., & Albert Hofman, D. (1995) · International Journal of Cardiology · doi:10.1016/0167-5273(95)02298-b · Cited by 10
Selection of patients for randomised trials: a study based on the MACOP‐B vs CHOP in NHL study
Stone, J. M., Laidlaw, C. R., Page, F., & Cooper, I. A. (1994) · Australian and New Zealand Journal of Medicine · doi:10.1111/j.1445-5994.1994.tb01754.x · Cited by 10
Cited by
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
EFFECTIVE RECRUITMENT AND RETENTION OF MINORITY RESEARCH PARTICIPANTS
Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. (2006) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.27.021405.102113 · Cited by 1,196
Van Spall, H. G. C., Toren, A., Kiss, A., & Fowler, R. A. (2007) · JAMA · doi:10.1001/jama.297.11.1233 · Cited by 1,055
Alexander, K. P., Newby, L. K., Cannon, C. P., Armstrong, P. W., Gibler, W. B., Rich, M. W., Van de Werf, F., White, H. D., Weaver, W. D., Naylor, M. D., et al. (2007) · Circulation · doi:10.1161/circulationaha.107.182615 · Cited by 914
Comparative quantification of health risks conceptual framework and methodological issues
Murray, C. J., Ezzati, M., Lopez, A. D., Rodgers, A., & Vander Hoorn, S. (2003) · Population Health Metrics · doi:10.1186/1478-7954-1-1 · Cited by 887
Metadata
- Authors
- Britton, A., McKee, M., Black, N., McPherson, K., Sanderson, C., & Bain, C.
- Venue
- Journal of Health Services Research & Policy
- Publisher
- SAGE Publications
- Published
- 1999-04
- Volume
- 4
- Issue
- 2
- Pages
- 112-121
- ISSN
- 1355-8196 (ISSN-L)
- 1758-1060
- OpenAlex
- W1948961602
- Cited by
- 299
- References
- 52
- FWCI
- 10.46
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.