Are pilot trials useful for predicting randomisation and attrition rates in definitive studies: A review of publicly funded trials
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Background/aims: External pilot trials are recommended for testing the feasibility of main or confirmatory trials. However, there is little evidence that progress in external pilot trials actually predicts randomisation and attrition rates in the main trial. To assess the use of external pilot trials in trial design, we compared randomisation and attrition rates in publicly funded randomised controlled trials with rates in their pilots. Methods: Randomised controlled trials for which there was an external pilot trial were identified from reports published between 2004 and 2013 in the Health Technology Assessment Journal. Data were extracted from published papers, protocols and reports. Bland–Altman plots and descriptive statistics were used to investigate the agreement of randomisation and attrition rates between the full and external pilot trials. Results: Of 561 reports, 41 were randomised controlled trials with pilot trials and 16 met criteria for a pilot trial with sufficient data. Mean attrition and randomisation rates were 21.1% and 50.4%, respectively, in the pilot trials and 16.8% and 65.2% in the main. There was minimal bias in the pilot trial when predicting the main trial attrition and randomisation rate. However, the variation was large: the mean difference in the attrition rate between the pilot and main trial was −4.4% with limits of agreement of −37.1% to 28.2%. Limits of agreement for randomisation rates were −47.8% to 77.5%. Conclusion: Results from external pilot trials to estimate randomisation and attrition rates should be used with caution as comparison of the difference in the rates between pilots and their associated full trial demonstrates high variability. We suggest using internal pilot trials wherever appropriate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“More recently, research has begun to explore the differences between internal and external pilot studies and their contribution to main trials, and the appropriateness of pilot and feasibility studies for estimating the sample size.”
From Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study (2018) · cites this paper as [3]
Experimental — accuracy not yet measuredIn the citing paper
“Research by Cooper et al has also shown variability between external pilots and the prediction for randomisation and attrition rates.”
From Role of feasibility and pilot studies in randomised controlled trials: a cross-sectional study (2018) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384
CONSORT 2010 statement: extension to randomised pilot and feasibility trials
on behalf of the PAFS consensus group, Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., & Lancaster, G. A. (2016) · Pilot and Feasibility Studies · doi:10.1186/s40814-016-0105-8 · Cited by 2,843 · cited 1×
“Guidance on good reporting of pilot and feasibility trials has been published.(13) The National Institute for Health Research(14) defines a pilot study as a miniature version of the main study to determine whether the components of the main study can work together.”
BackgroundDesign and analysis of pilot studies: recommendations for good practice
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004) · Journal of Evaluation in Clinical Practice · doi:10.1111/j..2002.384.doc.x · Cited by 2,572 · cited 3×
“Lancaster et al.(15) have outlined the possible objectives that an external pilot study can achieve including testing the integrity of the study protocol, testing the randomisation procedure, finding the rates of recruitment and consent, assessing the acceptability of the intervention and identifying the most appropri…”
BackgroundResultInfluentialWhitehead, A. L., Julious, S. A., Cooper, C. L., & Campbell, M. J. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280215588241 · Cited by 1,936
Self-citationEldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150205 · Cited by 1,581 · cited 2×
“Eldridge et al.(12) propose that a pilot study should be considered to be a subtype of feasibility study with both feasibility and pilot studies aimed at answering whether and how something can be done, a pilot study being designed specifically as the full study but on a smaller scale.”
BackgroundWhat is a pilot or feasibility study? A review of current practice and editorial policy
Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-67 · Cited by 1,483 · cited 1×
“This definition has been recommended for use as pilot studies have been found to be poorly reported, in part due to variation in definitions and lack of distinct between pilot and feasibility studies.(17) To qualify the study for inclusion, in line with this definition, the pilot trial had to be a version of the main…”
MethodSelf-citationNational Institute for Health Research
Evans, R. (2014) · Nursing Standard · doi:10.7748/ns.28.49.32.s37 · Cited by 1,350
Billingham, S. A., Whitehead, A. L., & Julious, S. A. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-104 · Cited by 962 · cited 3×
“The sample size calculation is an essential step in the design of a RCT.(5,6) An important aspect of a sample size calculation is that it provides the researcher with an estimate of the number of evaluable participants required in the dataset at the end of the study in order to investigate the study hypothesis with th…”
BackgroundInfluentialSelf-citationMcDonald, A. M., Knight, R. C., Campbell, M. K., Entwistle, V. A., Grant, A. M., Cook, J. A., Elbourne, D. R., Francis, D., Garcia, J., Roberts, I., et al. (2006) · Trials · doi:10.1186/1745-6215-7-9 · Cited by 946
Avery, K. N. L., Williamson, P. R., Gamble, C., O'Connell Francischetto, E., Metcalfe, C., Davidson, P., Williams, H., & Blazeby, J. M. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013537 · Cited by 575 · cited 2×
“Avery suggests that rather than employing dichotomous stop/go criteria, investigators should consider a traffic light red/ amber/green system which identifies the level of risk to proceeding.(21) In addition, data from the internal pilot are used in the main analysis which is not the case for external pilots and is th…”
BackgroundWalters, S. J., Bonacho dos Anjos Henriques-Cadby, I., Bortolami, O., Flight, L., Hind, D., Jacques, R. M., Knox, C., Nadin, B., Rothwell, J., Surtees, M., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015276 · Cited by 560 · cited 1×
“A recent review of Health Technology Assessment–funded trials found that the average attrition rate was 11%, while the average proportion of eligible patients who are randomised was 70%.(3) Accurate estimates of both these rates are important a priori for the design of future studies.”
BackgroundSelf-citationThe statistical interpretation of pilot trials: should significance thresholds be reconsidered?
Lee, E. C., Whitehead, A. L., Jacques, R. M., & Julious, S. A. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-41 · Cited by 423
Self-citationSully, B. G. O., Julious, S. A., & Nicholl, J. (2013) · Trials · doi:10.1186/1745-6215-14-166 · Cited by 410 · cited 1×
“A review of a cohort of trials funded by the UK Medical Research Council and the Health Technology Assessment Programme between 2002 and 2008 demonstrated that of 73 funded studies, 55% (40/ 73) of the trials achieved their original patient recruitment target, 16/73 (22%) achieved \80% of their original target and 45%…”
BackgroundSelf-citationHandling drop‐out in longitudinal studies
Hogan, J. W., Roy, J., & Korkontzelou, C. (2004) · Statistics in Medicine · doi:10.1002/sim.1728 · Cited by 404
Pilot and feasibility studies come of age!
Lancaster, G. A. (2015) · Pilot and Feasibility Studies · doi:10.1186/2055-5784-1-1 · Cited by 260
Palmer, R., Enderby, P., Cooper, C., Latimer, N., Julious, S., Paterson, G., Dimairo, M., Dixon, S., Mortley, J., Hilton, R., et al. (2012) · Stroke · doi:10.1161/strokeaha.112.650671 · Cited by 150 · cited 1×
“When they are successful, they can provide sufficient confidence that a larger trial can be conducted.(7,8) Even when the results are not as expected, the experiences from the trial can still be useful.”
BackgroundSelf-citationHind, D., O’Cathain, A., Cooper, C. L., Parry, G. D., Isaac, C. L., Rose, A., Martin, L., & Sharrack, B. (2010) · Psychology & Health · doi:10.1080/08870440902842739 · Cited by 101
Self-citationAcceptance checklist for clinical effectiveness pilot trials: a systematic approach
Charlesworth, G., Burnell, K., Hoe, J., Orrell, M., & Russell, I. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-78 · Cited by 87 · cited 2×
“An internal pilot as the first part of the main trial has a number of advantages over an external pilot.(27) Undertaking an external pilot trial can delay the main study by 3–4 years by the time the pilot study has completed, reported and further funding has been applied for.”
BackgroundMethodHind, D., Parkin, J., Whitworth, V., Rex, S., Young, T. A., Hampson, L. V., Sheehan, J., Maguire, C., Cantrill, H., Scott, E., et al. (2017) · Health Technology Assessment · doi:10.3310/hta21270 · Cited by 65
Cooper, C. L., Hind, D., Parry, G. D., Isaac, C. L., Dimairo, M., O'Cathain, A., Rose, A., Freeman, J. V., Martin, L., Kaltenthaler, E. C., et al. (2011) · Trials · doi:10.1186/1745-6215-12-259 · Cited by 61
Self-citationPalmer, R., Cooper, C., Enderby, P., Brady, M., Julious, S., Bowen, A., & Latimer, N. (2015) · Trials · doi:10.1186/s13063-014-0527-7 · Cited by 45 · cited 1×
“When they are successful, they can provide sufficient confidence that a larger trial can be conducted.(7,8) Even when the results are not as expected, the experiences from the trial can still be useful.”
BackgroundSelf-citationHind, D., Parkin, J., Whitworth, V., Rex, S., Young, T. A., Hampson, L. V., Sheehan, J., Maguire, C., Cantrill, H., Scott, E., et al. (2017) · Pilot and Feasibility Studies · doi:10.1186/s40814-017-0132-0 · Cited by 30
Definition and reporting of pilot and feasibility studies
Eldridge, S., Bond, C., Campbell, M., Lancaster, G., Thabane, L., & Hopwell, S. (2013) · Trials · doi:10.1186/1745-6215-14-s1-o18 · Cited by 29
on behalf of the OPTION-DM group, Selvarajah, D., Petrie, J., White, D. A., Julious, S. A., Bortolami, O., Cooper, C. L., Bradburn, M. J., Loban, A., Bowler, H., et al. (2018) · Trials · doi:10.1186/s13063-018-2959-y · Cited by 16
Self-citationJackson, L., Cohen, J., Sully, B., & Julious, S. A. (2015) · Pilot and Feasibility Studies · doi:10.1186/s40814-015-0007-1 · Cited by 10 · cited 1×
“Even when the results are not as expected, the experiences from the trial can still be useful.(9,10) The sample size calculation is an essential step in the design of a RCT.”
Self-citationJackson, L., Sully, B., Cohen, J., & Julious, S. A. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-515 · Cited by 5 · cited 1×
“Even when the results are not as expected, the experiences from the trial can still be useful.(9,10) The sample size calculation is an essential step in the design of a RCT.”
Self-citation
Cited by
Totton, N., Lin, J., Julious, S. A., Chowdhury, M., & Brand, A. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01416-w · Cited by 168
“Whitehead et al. [7] criticise the rule of thumb approach stating that the size of the full RCT should be taken into account to minimise the sample size across both studies.”
BackgroundSelf-citationRole of feasibility and pilot studies in randomised controlled trials: a cross-sectional study
Blatch-Jones, A. J., Pek, W., Kirkpatrick, E., & Ashton-Key, M. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-022233 · Cited by 139
“However, how they are reported, and in what context, requires caution especially when interpreting the findings and extrapolating these to the delivery of a definitive trial.(3) 11 12 17”
BackgroundInfluentialPilot and feasibility studies: extending the conceptual framework
Bond, C. M., Lancaster, G., Campbell, M., Chan, C. L., Eddy, S., Hopewell, S., Mellor, K., Thabane, L., & Eldridge, S. M. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01233-1 · Cited by 149
“Indeed, the authors of a recently published systematic review of publicly funded trials have recommended that internal pilot studies should be used when evidence is needed for estimating recruitment, randomisation and attrition rates [28].”
BackgroundLong-Term Weight Outcomes after Bariatric Surgery: A Single Center Saudi Arabian Cohort Experience
Alfadda, A. A., Al-Naami, M. Y., Masood, A., Elawad, R., Isnani, A. C., Ahamed, S. S., & Alfadda, N. A. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10214922 · Cited by 57
“The yearly attrition rate (in %) was derived by dividing the number of withdrawn participants (calculated as the number of participants retained in the study subtracted from the total number of participants originally included in the study at pre-surgery) by the number of participants originally included in the study…”
MethodVolkmer, A., Walton, H., Swinburn, K., Spector, A. E., Warren, J. D., & Beeke, S. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01301-6 · Cited by 46
“Sample size In line with guidance on conducting pilot studies, one of the aims of the study was to predict recruitment and retention rates for a future full trial [24, 25].”
Background
Metadata
- Authors
- Cooper, C. L., Whitehead, A., Pottrill, E., Julious, S. A., & Walters, S. J.
- Affiliations
- University of Sheffield (GB)
- University of Southampton (GB)
- Corresponding author
- Cooper, Cindy L
- Venue
- Clinical Trials
- Publisher
- SAGE Publications
- Published
- 2018-01-23
- Volume
- 15
- Issue
- 2
- Pages
- 189-196
- ISSN
- 1740-7745 (ISSN-L)
- 1740-7753
- PMC
- PMC5894808
- OpenAlex
- W2787517844
- Semantic Scholar
- cedc52d480a02e640c72a5f0e22c599400e55e6f
- Cited by
- 59
- References
- 26
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 10 · 2023: 6 · 2022: 5 · 2021: 9 · 2020: 10 · 2019: 5 · 2018: 2
- FWCI
- 5.39
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Clinical Trials (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- ePrints Soton (University of Southampton) (acceptedVersion)
- Funders
- Wellcome Trust
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- attrition rates, trial design, feasibility assessment, limits of agreement
- MeSH terms
- Humans, Patient Dropouts, Pilot Projects, Randomized Controlled Trials as Topic, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.