Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 28 of the 33 sources its claims cite were checked for support.

Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!

Open access
Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., Lancaster, G. A. · 2021 · Pilot and Feasibility Studies

Published version of a preprint

Abstract

Background: The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials. Methods: Many pilot trials adopt a ‘traffic light’ system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant. Results: For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0–17 participants (0–50%) will fall into the RED zone and be statistically non-significant, 18–25 (51–74%) fall into AMBER and may or may not be significant and 26–34 (75–100%) fall into GREEN and will be significant indicating acceptable fidelity. Discussion: In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.

Overview

CoChat Summary

TL;DR

The paper proposes a formal hypothesis-testing framework for binary feasibility progression criteria in pragmatic pilot RCTs, using traffic-light RED/AMBER/GREEN zones and sample size chosen to reject the RED zone with high power when the GREEN zone is true. It provides look-up tables and a worked example showing how to derive sample sizes and significance cut-points for multi-criteria progression decisions.

Key findings

  • For treatment fidelity with RED upper limit 50% and GREEN lower limit 75%, 90% power and one-sided 5% alpha requires around n = 34 in the intervention group; observed 0–17 (0–50%) falls in RED and is statistically non-significant, 18–25 (51–74%) falls in AMBER, and 26–34 (75–100%) falls in GREEN and is significant.
  • In the worked example, recruitment uptake (RUL 20%, GLL 35%) required n = 78 screened; treatment fidelity (RUL 50%, GLL 75%) required n = 34 in the intervention arm; follow-up (RUL 65%, GLL 85%) required n = 44 randomised. The overall sample was dictated by the largest requirement: 68 randomised, inflated to 200 screened assuming 35% uptake, giving 99.9%, 90%, and 98.8% power for the three criteria and collective 88.8% power.
  • Multi-criteria progression uses the worst-performing criterion: RED if any criterion is RED, AMBER if none are RED but at least one is AMBER, and GREEN only if all are GREEN. The extended 4-tier approach splits AMBER at the significance cut-point AC into AMBERR (major amend) and AMBERG (minor amend).
  • For a difference in proportions of 0.15 to 0.25 with α = 0.05 and β = 0.1, the normal approximation with continuity correction gives sample sizes of 33 to 100 (median 56); the binomial exact method gives 33 to 98 (median 54).

Methods

  • Methodological paper constructing a one-sample, one-tailed hypothesis test for binary feasibility outcomes: null hypothesis that the true feasibility outcome is not greater than the upper RED zone limit (RUL), alternative that it is greater than RUL; suggested α = 0.05 and β = 0.05, 0.1, or 0.2.
  • Sample sizes are derived using the normal approximation with continuity correction and the binomial exact approach (G*Power version 3.1.9.7), with look-up grids across RUL/GLL combinations; significance cut-point AC is calculated from the one-sided upper 95% confidence limit for the null proportion.
  • A motivating example uses three progression criteria—recruitment uptake, treatment fidelity, and follow-up—in a two-arm 1:1 pilot trial in children with cystic fibrosis; progression is assessed by the worst signal across criteria, and the 4-tier approach further splits AMBER by statistical significance.

Limitations

  • Stated: the methodology focuses on external pilot/feasibility studies and does not directly apply to internal pilots, though many principles may extend; outcomes are proportions/percentages where high values are acceptable and low values are not, so a reverse scale is needed for outcomes such as adverse events.
  • Stated: biased sample estimates can lead to wrong progression decisions; multi-criteria collective power is lower than individual power, so individual tests should be powered high (e.g., 90–95%) to achieve collective power of at least 70–80%, and if a marginal difference of ≤15% is tested, typical pilot sizes may fall short.
  • Evident from the text: the approach is illustrated with worked examples and tables rather than empirically validated through simulation; the AMBER major/minor split is described as arbitrary based on p-value significance and shifts with sample size; no multiple testing correction is recommended because criteria yield separate conclusions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s40814-021-00770-x

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Illustration of power using the 1-tailed hypothesis testing against the traffic light signalling approach to pilot progression. E, observed point estimate; RUL, upper limit of RED zone; GLL, lower limit of GREEN zone; Ac, cut-off for statistical significance (at the 1-sided 5% level); α, type I error; β, type II error
Fig. 1 Illustration of power using the 1-tailed hypothesis testing against the traffic light signalling approach to pilot progression. E, observed point estimate; RUL, upper limit of RED zone; GLL, lower limit of GREEN zone; Ac, cut-off for statistical significance (at the 1-sided 5% level); α, type I error; β, type II errorCC BY · PubMed Central
Fig. 2 Probability of traffic light given true underlying probability of an event using the example from Table 5 (i). Two plots are presented: a relating to normal approximation approach and b relating to binomial exact approach. Based on n = 200, RUL = 40 and GLL = 70
Fig. 2 Probability of traffic light given true underlying probability of an event using the example from Table 5 (i). Two plots are presented: a relating to normal approximation approach and b relating to binomial exact approach. Based on n = 200, RUL = 40 and GLL = 70CC BY · PubMed Central

Verification

Claims the paper makes

17 Valid · 2 Not valid · 0 Not checked · 46 Own results and interpretations

30 more claims

Create a free account to see all 65 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 21
  • It may state two separate claims as one. 19

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 33 references

Cited by

See all 330

Metadata

Authors
Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A.
Affiliations
  • Keele University (GB)
  • University of Manchester (GB)
Corresponding author
Lewis, M.
Venue
Pilot and Feasibility Studies
Published
2021-02-03
Volume
7
Issue
1
Pages
40-40
ISSN
2055-5784 (ISSN-L)
OpenAlex
W3109269546
Cited by
330
References
33
Citations by year
2026: 72 · 2025: 89 · 2024: 69 · 2023: 61 · 2022: 33 · 2021: 6
FWCI
31.42
Type
journal article
Language
en
Open access
Licence
Locations
  • Pilot and Feasibility Studies (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institute for Health and Care Research
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
sample size determination, progression criteria, process outcomes, hypothesis testing, intervention fidelity, CONSORT guidelines, binary outcomes, statistical power, traffic light system

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! https://app.cochat.ai/papers/10.1186/s40814-021-00770-x
The paper
Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A. (2021). Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! Pilot and Feasibility Studies. https://doi.org/10.1186/s40814-021-00770-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.