The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 28 of the 33 sources its claims cite were checked for support.
Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!
Open accessPublished version of a preprint
Abstract
Background: The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. Methods: Many pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant. Results: For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0–17 participants (0–50%) will fall into the RED zone and be statistically non-significant, 18–25 (51–74%) fall into AMBER and may or may not be significant and 26–34 (75–100%) fall into GREEN and will be significant indicating acceptable fidelity. Discussion: In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.
Overview
CoChat Summary
TL;DR
The paper proposes a formal hypothesis-testing framework for binary feasibility progression criteria in pragmatic pilot RCTs, using traffic-light RED/AMBER/GREEN zones and sample size chosen to reject the RED zone with high power when the GREEN zone is true. It provides look-up tables and a worked example showing how to derive sample sizes and significance cut-points for multi-criteria progression decisions.
Key findings
- For treatment fidelity with RED upper limit 50% and GREEN lower limit 75%, 90% power and one-sided 5% alpha requires around n = 34 in the intervention group; observed 0–17 (0–50%) falls in RED and is statistically non-significant, 18–25 (51–74%) falls in AMBER, and 26–34 (75–100%) falls in GREEN and is significant.
- In the worked example, recruitment uptake (RUL 20%, GLL 35%) required n = 78 screened; treatment fidelity (RUL 50%, GLL 75%) required n = 34 in the intervention arm; follow-up (RUL 65%, GLL 85%) required n = 44 randomised. The overall sample was dictated by the largest requirement: 68 randomised, inflated to 200 screened assuming 35% uptake, giving 99.9%, 90%, and 98.8% power for the three criteria and collective 88.8% power.
- Multi-criteria progression uses the worst-performing criterion: RED if any criterion is RED, AMBER if none are RED but at least one is AMBER, and GREEN only if all are GREEN. The extended 4-tier approach splits AMBER at the significance cut-point AC into AMBERR (major amend) and AMBERG (minor amend).
- For a difference in proportions of 0.15 to 0.25 with α = 0.05 and β = 0.1, the normal approximation with continuity correction gives sample sizes of 33 to 100 (median 56); the binomial exact method gives 33 to 98 (median 54).
Methods
- Methodological paper constructing a one-sample, one-tailed hypothesis test for binary feasibility outcomes: null hypothesis that the true feasibility outcome is not greater than the upper RED zone limit (RUL), alternative that it is greater than RUL; suggested α = 0.05 and β = 0.05, 0.1, or 0.2.
- Sample sizes are derived using the normal approximation with continuity correction and the binomial exact approach (G*Power version 3.1.9.7), with look-up grids across RUL/GLL combinations; significance cut-point AC is calculated from the one-sided upper 95% confidence limit for the null proportion.
- A motivating example uses three progression criteria—recruitment uptake, treatment fidelity, and follow-up—in a two-arm 1:1 pilot trial in children with cystic fibrosis; progression is assessed by the worst signal across criteria, and the 4-tier approach further splits AMBER by statistical significance.
Limitations
- Stated: the methodology focuses on external pilot/feasibility studies and does not directly apply to internal pilots, though many principles may extend; outcomes are proportions/percentages where high values are acceptable and low values are not, so a reverse scale is needed for outcomes such as adverse events.
- Stated: biased sample estimates can lead to wrong progression decisions; multi-criteria collective power is lower than individual power, so individual tests should be powered high (e.g., 90–95%) to achieve collective power of at least 70–80%, and if a marginal difference of ≤15% is tested, typical pilot sizes may fall short.
- Evident from the text: the approach is illustrated with worked examples and tables rather than empirically validated through simulation; the AMBER major/minor split is described as arbitrary based on p-value significance and shifts with sample size; no multiple testing correction is recommended because criteria yield separate conclusions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
17 Valid · 2 Not valid · 0 Not checked · 46 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“A common misuse of pilot and feasibility studies has been the application of hypothesis testing for clinical outcomes in small under-powered studies.”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Background
What the sources say
- Supportedcites [19] Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67. source
The Conclusions describe an inappropriate emphasis on hypothesis-testing in pilot studies. The Discussion says pilot studies usually lack the sample size needed for hypothesis testing, yet most pilot interventional studies reviewed reported such tests, including tests of intervention effectiveness. The Background also describes underpowered small studies being labeled as pilots. (LLM, conf=0.93)
- Supportedcites [20] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45(5):626–9. source
In “What Pilot Studies Cannot Do” (page unavailable), the paper says a pilot should not be treated as a preliminary test of an intervention hypothesis, citing its small sample size. Its Discussion warns that drawing conclusions from pilot outcomes risks false positive or false negative results. Together, these passages support the characterization of clinical-outcome hypothesis testing in small pilot studies as a misuse. (LLM, conf=0.88)
- Supportedcites [21] Horne E, Lancaster GA, Matson R, Cooper A, Ness A, Leary S. Pilot trials in physical activity journals: a review of reporting and editorial policy. Pilot Feasibility Stud. 2018;4:125. source
The Discussion reports that many trials with feasibility objectives placed inappropriate emphasis on hypothesis tests of the primary outcome intended for a definitive trial. It also identifies small sample sizes and hypothesis testing despite inadequate power as characteristics of trials inappropriately labelled as pilots. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Hypothesis testing in pilot studies, particularly in the context of effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations [2, 20, 21, 23].”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Discussion
What the sources say
- Supportedcites [2] Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64. source
Box 1 states that formal hypothesis testing for effectiveness or efficacy is not recommended because a pilot trial is not intended to assess it and will usually be underpowered. The Introduction likewise advises against an effectiveness-testing objective and says any underpowered tests should be treated as secondary and discussed with a caveat. (LLM, conf=0.92)
- Supportedcites [20] Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45(5):626–9. source
In “What Pilot Studies Cannot Do” (page not provided), the paper says a pilot study should not be treated as a preliminary test of an intervention hypothesis, citing its limited purpose and small sample size. “Sample Size Determination in Designing a Pilot Study” likewise says inferential tests should not be proposed; the discussion of efficacy data explains that small pilot samples yield uninformative estimates. (LLM, conf=0.91)
- Supportedcites [21] Horne E, Lancaster GA, Matson R, Cooper A, Ness A, Leary S. Pilot trials in physical activity journals: a review of reporting and editorial policy. Pilot Feasibility Stud. 2018;4:125. source
The Background describes multiple publications criticising small, underpowered studies that test efficacy or effectiveness while calling themselves pilot studies. The Discussion likewise calls hypothesis tests of outcomes intended for definitive trials an inappropriate emphasis and says most such trials were underpowered. Page numbers are not supplied in our copy. (LLM, conf=0.93)
- Supportedcites [23] Sim J. Should treatment effects be estimated in pilot and feasibility studies? Pilot Feasibility Stud. 2019;5:107. source
The Background (page unavailable) says pilot studies are intended to test whether trial components work together and notes that the CONSORT extension does not recommend formal hypothesis testing of treatment effects. The Abstract’s Methodological discussion (page unavailable) explains that typically small pilot studies yield imprecise effect estimates that cannot support robust decisions, matching the claim’s concern about limited statistical power. (LLM, conf=0.88)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the example in order to meet the individual ≥ 90% power requirement for all three criteria we would need: (i) for recruitment, the number to be screened to be 78; (ii) for treatment fidelity, the number in the intervention arm to be 34; and (iii) for follow up, the number randomised to be 44.”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to determine the overall sample size for the whole study, we base our decision on the criterion that requires the largest numbers, which is the treatment fidelity criterion which requires 68 to be randomised.”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Probability of 0.4 of it falling in the AMBERG zone”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims
Create a free account to see all 65 claims and every source verdict.
- Probably not a substantial claim. 21
- It may state two separate claims as one. 19
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Results
“57.8 to 70 (GREEN)”
§ Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! > Results
“Follow up [nr = 68 (intervention and control arms)]”
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
A tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384
Sample size of 12 per group rule of thumb for a pilot study
Julious, S. A. (2005) · Pharmaceutical Statistics · doi:10.1002/pst.185 · Cited by 3,068
CONSORT 2010 statement: extension to randomised pilot and feasibility trials
on behalf of the PAFS consensus group, Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., & Lancaster, G. A. (2016) · Pilot and Feasibility Studies · doi:10.1186/s40814-016-0105-8 · Cited by 2,843
Self-citationDesign and analysis of pilot studies: recommendations for good practice
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004) · Journal of Evaluation in Clinical Practice · doi:10.1111/j..2002.384.doc.x · Cited by 2,572
Self-citationConsiderations in determining sample size for pilot studies
Hertzog, M. A. (2008) · Research in Nursing & Health · doi:10.1002/nur.20247 · Cited by 2,337
Whitehead, A. L., Julious, S. A., Cooper, C. L., & Campbell, M. J. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280215588241 · Cited by 1,936
The role and interpretation of pilot studies in clinical research
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.10.008 · Cited by 1,917
What is a pilot or feasibility study? A review of current practice and editorial policy
Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-67 · Cited by 1,483
Self-citation
Cited by
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171
Mellor, K., Albury, C., Dutton, S. J., Eldridge, S., & Hopewell, S. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01291-5 · Cited by 145
Totton, N., Lin, J., Julious, S. A., Chowdhury, M., & Brand, A. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01416-w · Cited by 168
Pilot and feasibility studies: extending the conceptual framework
Bond, C. M., Lancaster, G., Campbell, M., Chan, C. L., Eddy, S., Hopewell, S., Mellor, K., Thabane, L., & Eldridge, S. M. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01233-1 · Cited by 140
Self-citationSample size justification in feasibility studies: moving beyond published guidance
Montgomery, R. N. (2025) · Pilot and Feasibility Studies · doi:10.1186/s40814-025-01675-9 · Cited by 88
Metadata
- Authors
- Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A.
- Affiliations
- Keele University (GB)
- University of Manchester (GB)
- Corresponding author
- Lewis, M.
- Venue
- Pilot and Feasibility Studies
- Published
- 2021-02-03
- Volume
- 7
- Issue
- 1
- Pages
- 40-40
- ISSN
- 2055-5784 (ISSN-L)
- PMC
- PMC7856754
- OpenAlex
- W3109269546
- Cited by
- 330
- References
- 33
- Citations by year
- 2026: 72 · 2025: 89 · 2024: 69 · 2023: 61 · 2022: 33 · 2021: 6
- FWCI
- 31.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Pilot and Feasibility Studies (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute for Health and Care Research
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- sample size determination, progression criteria, process outcomes, hypothesis testing, intervention fidelity, CONSORT guidelines, binary outcomes, statistical power, traffic light system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! https://app.cochat.ai/papers/10.1186/s40814-021-00770-x
- The paper
- Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A. (2021). Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! Pilot and Feasibility Studies. https://doi.org/10.1186/s40814-021-00770-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.