The DOI matches its Crossref record; 31 of the 32 references it cites were found in a registry or an index.
Should treatment effects be estimated in pilot and feasibility studies?
Open accessAbstract
BACKGROUND: Feasibility studies and external pilot studies are used increasingly to inform planning decisions related to a definitive randomized controlled trial. These studies can provide information on process measures, such as consent rates, treatment fidelity and compliance, and methods of outcome measurement. Additionally, they can provide initial parameter estimates for a sample size calculation, such as a standard deviation or the 'success' rate for a binary outcome in the control group. However, the issue of estimating treatment effects in pilot or feasibility studies is controversial. METHODOLOGICAL DISCUSSION: Between-group estimates of treatment effect from pilot studies are sometimes used to calculate the sample size for a main trial, alongside estimated standard deviations. However, whilst estimating a standard deviation is an empirical matter, a targeted treatment effect should be established in terms of clinical judgement, as a minimum important difference (MID), not through analysis of pilot data. Secondly, between-group effects measured in pilot studies are sometimes used to indicate the magnitude of an effect that might be obtained in a main trial, and a decision on progression made with reference to the associated confidence interval. Such estimates will be imprecise in typically small pilot studies and therefore do not allow a robust decision on a main trial; both a decision to proceed and a decision not to proceed may be made too readily. Thirdly, a within-group change might be estimated from a pilot or a feasibility study in a desire to assess the potential efficacy of a novel intervention prior to testing it in a main trial, but again such estimates are liable to be imprecise and do not allow sound causal inferences. CONCLUSION: Treatment effects calculated from pilot or feasibility studies should not be the basis of a sample size calculation for a main trial, as the MID to be detected should be based primarily on clinical judgement rather than statistics. Deciding on progression to a main trial based on these treatment effects is also misguided, as they will normally be imprecise, and may be biased if the pilot or feasibility study is unrepresentative of the main trial.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 Estimated treatment effects, with 95% confidence intervals, reported for four outcomes by Sheehan et al. [25]. A difference in percentage improvement of 40% (indicated by the dashed horizontal reference line) was taken to be the minimum important difference](/paper-figures/4433f5310c5e4f558ed4051c5e3ecc6c/should-treatment-effects-be-estimated-in-pilot-and-feasibility-studies-figure-1.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Hypothesis testing in pilot studies, particularly in the context of effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations [2, 20, 21, 23].”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384
CONSORT 2010 statement: extension to randomised pilot and feasibility trials
on behalf of the PAFS consensus group, Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., & Lancaster, G. A. (2016) · Pilot and Feasibility Studies · doi:10.1186/s40814-016-0105-8 · Cited by 2,843
Design and analysis of pilot studies: recommendations for good practice
Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004) · Journal of Evaluation in Clinical Practice · doi:10.1111/j..2002.384.doc.x · Cited by 2,572 · cited 1×
“Background Feasibility studies and external pilot studies are increasingly common precursors to a main, or definitive, randomized controlled trial [1, 2].”
BackgroundFundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237
Whitehead, A. L., Julious, S. A., Cooper, C. L., & Campbell, M. J. (2016) · Statistical Methods in Medical Research · doi:10.1177/0962280215588241 · Cited by 1,936 · cited 1×
“They may also function to estimate parameters required for a sample size calculation, such as the standard deviation of a continuous outcome measure [5, 6], or possibly the control group proportion for a binary”
BackgroundThe role and interpretation of pilot studies in clinical research
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.10.008 · Cited by 1,917
Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150205 · Cited by 1,581
What is a pilot or feasibility study? A review of current practice and editorial policy
Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-67 · Cited by 1,483
Sim, J., & Lewis, M. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.07.011 · Cited by 1,036
Self-citation. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702
Cited by
Sorted by citations
Lewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A. (2021) · Pilot and Feasibility Studies · doi:10.1186/s40814-021-00770-x · Cited by 330
Totton, N., Lin, J., Julious, S. A., Chowdhury, M., & Brand, A. (2023) · Pilot and Feasibility Studies · doi:10.1186/s40814-023-01416-w · Cited by 160
Incorporating feasibility protocols in intervention research
Gadke, D. L., Kratochwill, T. R., & Gettinger, M. (2020) · Journal of School Psychology · doi:10.1016/j.jsp.2020.11.004 · Cited by 158
Consolidated guidance for behavioral intervention pilot and feasibility studies
Pfledderer, C. D., von Klinggraeff, L., Burkart, S. A., Bandeira, A. D. S., Lubans, D. R., Jago, R., Okely, A. D., van Sluijs, E. M. F., Ioannidis, J. P. A., Thrasher, J. F., et al. (2024) · Pilot and Feasibility Studies · doi:10.1186/s40814-024-01485-5 · Cited by 69
Determining sample size for pilot trials: a tutorial
Ying, X., Freedland, K. E., Powell, L. H., Stuart, E. A., Ehrhardt, S., & Mayo‐Wilson, E. (2025) · BMJ · doi:10.1136/bmj-2024-083405 · Cited by 64
Metadata
- Authors
- Sim, J.
- Affiliations
- Keele University (GB)
- Corresponding author
- Sim, Julius
- ORCID
- Sim, Julius
- Venue
- Pilot and Feasibility Studies
- Published
- 2019-08-28
- Volume
- 5
- Issue
- 1
- Pages
- 107-107
- ISSN
- 2055-5784 (ISSN-L)
- PMC
- PMC6712606
- OpenAlex
- W2970940014
- Cited by
- 123
- References
- 31
- Citations by year
- 2026: 15 · 2025: 13 · 2024: 13 · 2023: 23 · 2022: 20 · 2021: 18 · 2020: 19 · 2019: 2
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Pilot and Feasibility Studies (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- pilot study, feasibility study, treatment effect estimation, sample size determination, minimal clinically important difference, clinical judgment, trial planning, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
