How big should the pilot study for my cluster randomised trial be?
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
There is currently a lot of interest in pilot studies conducted in preparation for randomised controlled trials. This paper focuses on sample size requirements for external pilot studies for cluster randomised trials. We consider how large an external pilot study needs to be to assess key parameters for input to the main trial sample size calculation when the primary outcome is continuous, and to estimate rates, for example recruitment rates, with reasonable precision. We used simulation to provide the distribution of the expected number of clusters for the main trial under different assumptions about the natural cluster size, intra-cluster correlation, eventual cluster size in the main trial, and various decisions made at the piloting stage. We chose intra-cluster correlation values and pilot study size to reflect those commonly reported in the literature. Our results show that estimates of sample size required for the main trial are likely to be biased downwards and very imprecise unless the pilot study includes large numbers of clusters and individual participants. We conclude that pilot studies will usually be too small to estimate parameters required for estimating a sample size for a main cluster randomised trial (e.g. the intra-cluster correlation coefficient) with sufficient precision and too small to provide reliable estimates of rates for process measures such as recruitment or follow-up rates.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“investigators should also be aware that estimates of rates in pilot trials may be subject to considerable uncertainty”
From CONSORT 2010 statement: extension to randomised pilot and feasibility trials. (2016) · cites this paper as [41]
In the citing paper
“Furthermore, several sources emphasised the importance of testing implementation in a range of contexts [15, 29, 39, 41]—especially in less socioeconomically advantaged groups, to examine the risk of widening health inequalities [29, 39].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [41]
In the citing paper
“Some condemned statistically testing for effectiveness [7, 20, 29, 32, 41], as such studies are often underpowered, hence leading to imprecise and potentially misleading estimates of effect sizes [7, 20].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [41]
In the citing paper
“Others made the case for identifying evidence of potential effectiveness, including use of interim or surrogate endpoints [7, 41]”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [41]
In the citing paper
“Cluster RCTs, which are common in public health interventions, were specifically earmarked as unsuitable for estimating parameters for sample size calculations (e.g. intra-cluster correlation coefficients) as well as recruitment and follow-up rates without additional information from other resources, because a large number of clusters and individual participants would be required [41].”
From Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance (2018) · cites this paper as [41]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582 · cited 1×
“Furthermore, an additional complication arises because estimates of these variances depend on the number of clusters as well as the number of individual participants, and the rather small number of clusters typically involved in a pilot study can potentially cause downward bias in estimates of between-cluster variatio…”
MethodA tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384
Design and analysis of pilot studies: recommendations for good practice
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004) · Journal of Evaluation in Clinical Practice · doi:10.1111/j..2002.384.doc.x · Cited by 2,572 · cited 2×
“Objectives related to these five aspects are usually less well articulated in trial reports than an overarching feasibility objective, even though the importance of stating specific objectives has been stressed in several papers focusing on the methodology of pilot studies.(9,10) In addition, many studies labelled as…”
BackgroundResultSelf-citationSim, J., & Lewis, M. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.07.011 · Cited by 1,036 · cited 1×
“Previous research in relation to individually randomised trials has suggested inflating sample sizes estimated from pilot studies using upper confidence interval limits to counteract imprecision in these estimates.(29) However, a more recent study concludes that if individually randomised trials are designed using par…”
Gulliford, M. C., Ukoumunne, O. C., & Chinn, S. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009904 · Cited by 852
Teare, M. D., Dimairo, M., Shephard, N., Hayman, A., Whitehead, A., & Walters, S. J. (2014) · Trials · doi:10.1186/1745-6215-15-264 · Cited by 773 · cited 2×
“This has included investigations to assess how large such studies should be.(4) However, none of these investigations have focussed on cluster randomised trials.”
BackgroundResultA Practical Guide to Cluster Randomised Trials in Health Services Research
Eldridge, S., & Kerry, S. (2012) · Wiley eBooks · doi:10.1002/9781119966241 · Cited by 351 · cited 5×
“Patterns in ICCs have been explored by a number of authors and these are summarised in the book by Eldridge and Kerry, where the issue of how to combine estimates from different sources is also discussed.(3) Briefly, it has been shown that ICCs for outcomes that measure the process of care are generally higher than IC…”
BackgroundMethodResultInfluentialSelf-citationEldridge, S. M., Ashby, D., Feder, G. S., Rudnicka, A. R., & Ukoumunne, O. C. (2004) · Clinical Trials · doi:10.1191/1740774504cn006rr · Cited by 288 · cited 1×
“The interventions they evaluate can be complex, trial logistics complicated, and execution difficult.(1,2) Potentially, some of the reported problems in such trials could have been avoided if procedures and activities had been tested beforehand.”
BackgroundSelf-citationQuestions asked and answered in pilot and feasibility randomized controlled trials
Shanyinde, M., Pickering, R. M., & Weatherall, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-117 · Cited by 229 · cited 1×
“In addition, many studies labelled as pilot or feasibility randomised controlled trials continue to focus on assessing efficacy or effectiveness, rather than feasibility objectives.(11) In fact, objectives related to the different aspects of feasibility lead to different methods of determining how large such a study s…”
BackgroundBayesian methods of analysis for cluster randomized trials with binary outcome data
Turner, R. M., Omar, R. Z., & Thompson, S. G. (2001) · Statistics in Medicine · doi:10.1002/1097-0258(20010215)20:3<453::aid-sim803>3.0.co;2-l · Cited by 126
Self-citationAn Empirical Study of Cluster Randomization
DONNER, A. (1982) · International Journal of Epidemiology · doi:10.1093/ije/11.3.283 · Cited by 103
Bell, M. L., & McKenzie, J. E. (2013) · Psycho-Oncology · doi:10.1002/pon.3205 · Cited by 99
Ukoumunne, O. C. (2002) · Statistics in Medicine · doi:10.1002/sim.1330 · Cited by 96 · cited 1×
“Our findings also concur with previous research that indicates that the uncertainty around ICC estimates is considerable even for relatively large samples and that estimating ICCs only from pilot studies for cluster randomised trials is not optimal.(3,24) This is in spite of the fact that our brief review indicates th…”
ResultReeves, R., West, E., & Barron, D. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-259 · Cited by 96 · cited 1×
“ask the question in relation to producing feedback summaries (their intervention): ‘Could ward-level postal surveys be conducted successfully?’.(15) Presumably, if the response rate was very low the investigators would have considered their trial not feasible.”
BackgroundSample size re‐estimation in cluster randomization trials
Lake, S., Kammann, E., Klar, N., & Betensky, R. (2002) · Statistics in Medicine · doi:10.1002/sim.1121 · Cited by 92 · cited 1×
“Some research suggests that internal pilot studies may be good sources of information for ultimate sample size required for a cluster randomised trial, although this research used different methods and scenarios from the ones we used.(27,28) Previous research in relation to individually randomised trials has suggested…”
Farrin, A., Russell, I., Torgerson, D., & Underwood, M. (2005) · Clinical Trials · doi:10.1191/1740774505cn073oa · Cited by 79 · cited 1×
“The interventions they evaluate can be complex, trial logistics complicated, and execution difficult.(1,2) Potentially, some of the reported problems in such trials could have been avoided if procedures and activities had been tested beforehand.”
BackgroundLittenberg, B., & MacLean, C. D. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-20 · Cited by 73
CONNECT for Better Fall Prevention in Nursing Homes: Results from a Pilot Intervention Study
Colon‐Emeric, C. S., McConnell, E., Pinheiro, S. O., Corazzini, K., Porter, K., Earp, K. M., Landerman, L., Beales, J., Lipscomb, J., Hancock, K., et al. (2013) · Journal of the American Geriatrics Society · doi:10.1111/jgs.12550 · Cited by 66
Drahota, A., Ward, D., Udell, J. E., Soilemezi, D., Ogollah, R. O., Higgins, B., Dean, T., & Severs, M. (2013) · Age and Ageing · doi:10.1093/ageing/aft067 · Cited by 57
Diaz-Ordaz, K., Froud, R., Sheehan, B., & Eldridge, S. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-127 · Cited by 47
Self-citationLeBlanc, A., Légaré, F., Labrecque, M., Godin, G., Thivierge, R., Laurier, C., Côté, L., O'Connor, A. M., & Rousseau, M. (2011) · Implementation Science · doi:10.1186/1748-5908-6-5 · Cited by 37 · cited 4×
“Okwen et al.,8 Jairath et al.,16 and Leblanc et al.18 are examples of this.”
BackgroundInfluentialJairath, V., Kahan, B. C., Gray, A. J., Doré, C. J., Mora, A., Dyer, C., Stokes, E. A., LLEWELYN, C. A., Bailey, A. A., Dallal, H. J., et al. (2013) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2013.04.001 · Cited by 29
Self-citationEldridge, S., Cryer, C., Feder, G., & Underwood, M. (2001) · Statistics in Medicine · doi:10.1002/1097-0258(20010215)20:3<367::aid-sim798>3.0.co;2-r · Cited by 28
Self-citationHawk, M. (2013) · AIDS Education and Prevention · doi:10.1521/aeap.2013.25.6.519 · Cited by 27 · cited 1×
“The final three studies were shared apartment communities in Germany (eight sheltered housing units randomised into two groups of four)(14); nursing homes in the USA (four community nursing homes matched to four Veteran Affairs facilities randomised within pair to receive intervention or control)(17); and HIV communit…”
BackgroundRe‐estimating sample size in cluster randomised trials with active recruitment within clusters
van Schie, S., & Moerbeek, M. (2014) · Statistics in Medicine · doi:10.1002/sim.6172 · Cited by 27 · cited 1×
“Some research suggests that internal pilot studies may be good sources of information for ultimate sample size required for a cluster randomised trial, although this research used different methods and scenarios from the ones we used.(27,28) Previous research in relation to individually randomised trials has suggested…”
Kerr, B., Hawkins, T. L.-A., Herman, R. J., Barnes, S., Kaufmann, S., Fraser, K. L., & Ma, I. W. Y. (2013) · Medical Education Online · doi:10.3402/meo.v18i0.21312 · Cited by 26 · cited 1×
“The final example in a medical setting involved medical conference workshops in Canada (three workshops with one randomised to traditional case-based teaching and two to simulation-based computerised teaching).(7) The final three studies were shared apartment communities in Germany (eight sheltered housing units rando…”
BackgroundA pilot cluster randomized controlled trial of structured goal-setting following stroke
Taylor, W. J., Brown, M., Levack, W., McPherson, K. M., Reed, K., Dean, S. G., & Weatherall, M. (2011) · Clinical Rehabilitation · doi:10.1177/0269215511419384 · Cited by 26 · cited 3×
“decided not to proceed with a main trial partly on the basis of large ICC estimates from a small pilot trial with four clusters and 41 patients, even though they report confidence intervals for their ICCs that all include zero.(20) It has been suggested that the most reliable method of estimating ICCs is from patterns…”
BackgroundInfluentialTeut, M., Schnabel, K., Baur, R., Kerckhoff, Reese, Berger, F., Pilgram, Lüdtke, R., & Witt, C. M. (2013) · Clinical Interventions in Aging · doi:10.2147/cia.s45242 · Cited by 16 · cited 2×
“Drahota et al.5 and Teut et al.14 are examples of this.”
BackgroundPrimary eye care skills scores for health workers in routine and enhanced supervision settings
Okwen, M., Lewallen, S., & Courtright, P. (2013) · Public Health · doi:10.1016/j.puhe.2013.10.007 · Cited by 12
Porteous, T., Smith, S. E., Sheikh, A., Francis, J., Lee, A. J., Bond, C. M., Lowrie, R., Scotland, G., Thomas, M. D., Wyke, S., et al. (2013) · Clinical & Experimental Allergy · Cited by 1
Research works printed but not linked: 2
- Mass CJM, 2005, Methodology, 1, 7
- Porteous T, 2013, Clin Exp Allergy, 43, 1444
Cited by
CONSORT 2010 statement: extension to randomised pilot and feasibility trials.
Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., & Lancaster, G. A. (2016) · BMJ · doi:10.1136/bmj.i5239 · Cited by 3,404
“[51] In addition to the possibility of making changes to the trial protocol, investigators should also be aware that estimates of rates in pilot trials may be subject to considerable uncertainty so that it is best to be cautious about setting definitive thresholds that could be missed simply due to chance variation.[4…”
Self-citationCONSORT 2010 statement: extension to randomised pilot and feasibility trials
on behalf of the PAFS consensus group, Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., & Lancaster, G. A. (2016) · Pilot and Feasibility Studies · doi:10.1186/s40814-016-0105-8 · Cited by 2,843
Self-citationGuidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
How to design efficient cluster randomised trials
Hemming, K., Eldridge, S., Forbes, G., Weijer, C., & Taljaard, M. (2017) · BMJ · doi:10.1136/bmj.j3064 · Cited by 270
Self-citationHallingberg, B., Turley, R., Segrott, J., Wight, D., Craig, P., Moore, L., Murphy, S., Robling, M., Simpson, S. A., & Moore, G. (2018) · Pilot and Feasibility Studies · doi:10.1186/s40814-018-0290-8 · Cited by 248
“Some explicitly stated progression criteria should not be judged as strict thresholds but as guidelines using, for example, a traffic lights system with varying levels of acceptability [7, 41].”
BackgroundInfluential
Metadata
- Authors
- Eldridge, S. M., Costelloe, C. E., Kahan, B. C., Lancaster, G. A., & Kerry, S. M.
- Affiliations
- University of London (GB)
- Centre for Primary Care and Public Health, Queen Mary University of London, London, UK
- Lancaster University (GB)
- Corresponding author
- Eldridge, Sandra M
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2015-06-12
- Volume
- 25
- Issue
- 3
- Pages
- 1039-1056
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W1927287710
- Cited by
- 135
- References
- 29
- Citations by year
- 2026: 14 · 2025: 22 · 2024: 11 · 2023: 13 · 2022: 8 · 2021: 16 · 2020: 22 · 2019: 10 · 2018: 5 · 2017: 10 · 2016: 4
- FWCI
- 5.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Statistical Methods in Medical Research (publishedVersion)
- PubMed (publishedVersion)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Funders
- National Institute for Health and Care Research (CDF-2016-09-015)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Keywords
- pilot study, sample size determination, intra-cluster correlation, recruitment rate, follow-up rates, process indicators, sample size bias
- MeSH terms
- Humans, Pilot Projects, Cluster Analysis, Randomized Controlled Trials as Topic, Sample Size, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.