The role and interpretation of pilot studies in clinical research
Open accessAbstract
Pilot studies represent a fundamental phase of the research process. The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design. A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study. Investigators should be forthright in stating these objectives of a pilot study. Grant reviewers and other stakeholders should expect no more.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 5 cited sources checked · 0 supported · 0 flagged
- Premisecitation
When event rates are reported and no adverse event is observed for a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n.
“When event rates are reported and no adverse event is observed for a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Jovanovic & Levy, 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Jovanovic et al., 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
Basing a decision to proceed or terminate evaluation of a particular intervention on pilot data is perilous because there is a very good chance that the decision will be derived from false positive or false negative results.
“Basing a decision to proceed or terminate evaluation of a particular intervention on pilot data is perilous because there is a very good chance that the decision will be derived from false positive or false negative results.”
Quote in the paperSubstantial claim - Interpretationreasoning
Each aspect of feasibility could be quite different from an uncontrolled study when intervention assignment is randomized and blinded, particularly if placebo is a distinct possibility.
“Each aspect of feasibility could be quite different from an uncontrolled study when intervention assignment is randomized and blinded, particularly if placebo is a distinct possibility.”
Quote in the paperNot a substantial claim - Interpretationreasoning
A pilot study can better address its goals if a control group is part of the design.
“With these issues in mind, a pilot study can better address its goals if a control group is part of the design.”
Quote in the paperNot a substantial claim - Interpretationreasoning
A case series estimate tends to be steeped in optimism, particularly if an estimate of such magnitude is seldom, if ever, seen in full scale trials for psychiatric disorders.
“A case series estimate tends to be steeped in optimism, particularly if an estimate of such magnitude is seldom, if ever, seen in full scale trials for psychiatric disorders.”
Quote in the paperSubstantial claim
- Premisereasoning
It is the population effect size, not the sample effect size, that an RCT is designed to detect with sufficient power.
“(It is the population effect size, not the sample effect size, that an RCT is designed to detect with sufficient power.)”
Quote in the paperSubstantial claim - Interpretationcitation
The tradition of using pilot studies to estimate between group effect sizes for designing subsequent larger scale hypothesis testing studies is ill-founded.
“There has been a venerable tradition of using pilot studies to estimate between group effect sizes that, in turn, are used to inform the design of subsequent larger scale hypothesis testing studies. Despite it widespread use, it has been argued that the tradition is ill-founded. (Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006)”
Quote in the paperSubstantial claim0 of 2 sources checked - Interpretationreasoning
A pilot sample size is based on the pragmatics of recruitment and the necessities for examining feasibility.
“A pilot sample size is instead based on the pragmatics of recruitment and the necessities for examining feasibility.”
Quote in the paperSubstantial claim - Interpretationreasoning
With no inferential statistical tests, a pilot study will not provide p-values.
“With no inferential statistical tests, a pilot study will not provide p-values.”
Quote in the paperSubstantial claim - Interpretationreasoning
Combining pilot data with data from the subsequent larger scale study risks adding additional, perhaps unknown, source of variation because methods are likely to be modified after the pilot.
“Pilot study data generally should not be combined with data from the subsequent larger scale study. This is because it is quite likely that the methods will be modified after the pilot, even if minimally. Such changes in protocol risk adding additional, perhaps unknown, source of variation.”
Quote in the paperSubstantial claim- Rests on an assumption: Methods will likely be modified after a pilot study, even if minimally.
- Interpretationreasoning
A pilot study does not provide a meaningful effect size estimate for planning subsequent studies because data from small samples are imprecise.
“Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples.”
- Also stated in: § DISCUSSION · plain
“pilot studies do not provide useful information regarding the population effect size because the estimates are quite crude owing to the small sample sizes.”
Quote in the paperSubstantial claim - Also stated in: § DISCUSSION · plain
- Interpretationreasoning
The imprecision seen in the wide confidence interval underscores the limited value of safety data from a pilot study.
“This imprecision, seen in the wide confidence interval, underscores the limited value of safety data from a pilot study.”
Quote in the paperSubstantial claim - Interpretationreasoning
Inclusion of a control group in a pilot allows for a more realistic examination of recruitment, randomization, implementation of interventions, blinded assessment procedures, and retention in blinded interventions.
“Although not essential for many aspects of the study, inclusion of a control group allows for a more realistic examination of recruitment, randomization, implementation of interventions, blinded assessment procedures, and retention in blinded interventions.”
Quote in the paperSubstantial claim - Interpretationreasoning
The efficacy data from a pilot study of this size are uninformative.
“In other words, the efficacy data from a pilot study of this size are uninformative.”
Quote in the paperSubstantial claim - Interpretationreasoning
If a pilot study effect size is unduly small (false negative), subsequent development of the intervention could very well be terminated - even if the intervention eventually would have proven to be effective.
“If a pilot study effect size is unduly small (i.e., a false negative result), subsequent development of the intervention could very well be terminated – even if the intervention eventually would have proven to be effective.”
Quote in the paperSubstantial claim - Interpretationnone given
A study can only examine feasibility of the patient type included in the study.
“Further, a study can only examine feasibility of the patient type included in the study.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Feasibility results from a pilot study do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design.
“Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design.”
- Also stated in: § DISCUSSION · plain
“The feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot.”
Quote in the paperSubstantial claim - Also stated in: § DISCUSSION · plain
- Interpretationreasoning
Only in an extreme, unfortunate case, where a death occurs or repeated serious adverse events surface, do pilot studies inform the safety of testing an intervention due to the small sample size.
“Only in an extreme, unfortunate case, where a death occurs or repeated serious adverse events surface, do pilot studies inform the safety of testing an intervention due to the small sample size.”
Quote in the paperSubstantial claim - Premisecitation
Inclusion of a control or comparator group in a full scale trial accounts for the passage of time, increased attention, expectation of therapeutic intervention, and psychological consequences of legitimized sick role.
“The inclusion of a control or comparator group in a full scale trial accounts for the passage of time, the increased attention received in the study, the expectation of a therapeutic intervention, and the psychological consequences of legitimized sick role.(Klerman, 1986)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Klerman, 1986
- Interpretationreasoning
It is the feasibility results, not the efficacy or safety results, that inform subsequent trials.
“It is the feasibility results, not the efficacy or safety results, that inform subsequent trials.”
Quote in the paperSubstantial claim - Interpretationreasoning
A false negative result could preclude the opportunity to further examine its latent efficacy.
“Unfortunately, a false negative result could preclude the opportunity to further examine its latent efficacy.”
Quote in the paperSubstantial claim - Interpretationreasoning
Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples.
“Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples.”
Quote in the paperSubstantial claim - Premisereasoning
A pilot study is not a hypothesis testing study.
“A pilot study is not a hypothesis testing study.”
- Also stated in: § INTRODUCTION · plain
“A pilot study, however, is not used for hypothesis testing.”
- Also stated in: § SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY · plain
“A pilot study is not a hypothesis testing study.”
Quote in the paperSubstantial claim - Also stated in: § INTRODUCTION · plain
- Interpretationreasoning
If a well-specified adaptive design were explicated prior to the start of a pilot study, and the risk of elevated type I error appropriately controlled, it is conceivable that data from before and after protocol changes could be pooled.
“However, if a well-specified adaptive design were explicated prior to the start of a pilot study, and the risk of elevated type I error appropriately controlled, it is conceivable that data from before and after protocol changes could, in fact, be pooled.”
Quote in the paperSubstantial claim
- May combine two claims. 13
- Classified as probably not a substantial claim. 13
May combine two claims.
InterpretationreasoningPilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study.
“Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study.”
- Also stated in: § DISCUSSION · plain
“Pilot results inform feasibility, which in turn, is instructive in that it points to modifications needed in the planning and design of a larger efficacy trial.”
Quote in the paperSubstantial claim- Also stated in: § DISCUSSION · plain
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThe definition of clinically meaningful is not entirely empirically-based, but instead requires input from clinicians who treat the patient population of interest and perhaps from patients with the disorder.
“The definition of clinically meaningful is not entirely empirically-based, but instead requires input from clinicians who treat the patient population of interest and perhaps from patients with the disorder.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenGood clinical practices developed in a pilot include refinement of source documentation, informed consent procedures, data collection tools, regulatory reporting procedures, and monitoring/oversight procedures, especially when multiple sites and investigators are engaged.
“These good clinical practices include the refinement of source documentation, informed consent procedures, data collection tools, regulatory reporting procedures, and monitoring/oversight procedures, especially when multiple sites and investigators are engaged in the study.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningSafety, efficacy and effectiveness are not evaluated in a pilot study.
“Safety, efficacy and effectiveness are not evaluated in a pilot.”
- Also stated in: § DISCUSSION · plain
“a pilot study is not a hypothesis testing study and therefore safety and efficacy are not evaluated.”
Quote in the paperSubstantial claim- Also stated in: § DISCUSSION · plain
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
InterpretationreasoningPilot studies generate pilot data, and their design need not be guided with the support of prior pilot data.
“Pilot studies generate pilot data, their design need not be guided with the support of prior pilot data.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningIt is possible, but highly unlikely, that the between group effect size (d) from a pilot study sample will provide a reasonable estimate of the population effect size (Δ), but that cannot be known based on the pilot data.
“It is possible, but highly unlikely, that the between group effect size (d) from a pilot study sample will provide a reasonable estimate of the population effect size (Δ), but that cannot be known based on the pilot data.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningA pilot study can be critical in research staff training and provide experiences that strengthen and confirm competencies and skills required for the investigation to be conducted with accuracy and precision.
“A pilot study can be critical in research staff training and provide experiences that strengthen and confirm competencies and skills required for the investigation to be conducted with accuracy and precision.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningAn alternative approach is to base sample size estimates for a full scale RCT on what is deemed a clinically meaningful effect.
“An alternative approach is to base sample size estimates for a full scale RCT on what is deemed a clinically meaningful effect.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenA pilot study provides opportunity to develop consistent practices to enhance data integrity and protection of human subjects.
“A pilot study provides opportunity to develop consistent practices to enhance data integrity and the protection of human subjects.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningWe learn little if anything about the efficacy of an intervention with data from a small sample; yet, a great deal can be learned from a pilot study.
“We learn little if anything about the efficacy of an intervention with data from a small sample; yet, as discussed earlier, a great deal can be learned from a pilot study.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
InterpretationreasoningThis applies to effect sizes for any type of outcome, be it a severity rating, a response status, or survival status.
“This applies to effects sizes for any type of outcome, be it a severity rating, a response status, or survival status.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningWith imprecision comes a vast disparity in sample size estimates and, if sample size determination for a subsequent study is based on an imprecise estimate, there is an enormous risk of underpowered or overpowered design.
“Hence, with imprecision comes a vast disparity in sample size estimates and, if sample size determination for a subsequent study is based on an imprecise estimate, there is an enormous risk of underpowered or overpowered design.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningPilot studies are a necessary first step in exploring novel interventions and novel applications of interventions.
“In summary, pilot studies are a necessary first step in exploring novel interventions and novel applications of interventions – whether in a new patient population or with a novel delivery system (e.g., transdermal patch).”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningA pilot study is not, and should not be considered, a preliminary test of the intervention hypothesis.
“Although a pilot study will undoubtedly incorporate relevant outcome measures and can serve a vital role in treatment development, it is not, and should not, be considered a preliminary test of the intervention hypothesis.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisereasoningThe primary role of a pilot study is to examine the feasibility of a research endeavor.
“The primary role of a pilot study is to examine the feasibility of a research endeavor.”
- Also stated in: § INTRODUCTION · plain
“The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study.”
Quote in the paperSubstantial claim- Also stated in: § INTRODUCTION · plain
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenA theoretical rationale for the intervention or methodology being proposed for a pilot study is still needed.
“This does not preclude the need for a theoretical rationale for the intervention or the methodology being proposed for a pilot study.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningIf a pilot study effect size is unduly large (false positive), subsequent trials will be designed with an inadequate number of participants to provide the statistical power needed to detect clinically meaningful effects and that would lead to negative trials.
“If a pilot study effect size is unduly large (i.e., a false positive result), subsequent trials will be designed with an inadequate number of participants to provide the statistical power needed to detect clinically meaningful effects and that would lead to negative trials.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenPilot studies provide an opportunity to implement and examine the feasibility of the adverse event reporting system.
“However, pilot studies provide an opportunity to implement and examine the feasibility of the adverse event reporting system.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
InterpretationreasoningIt is reasonable and expected that a pilot study is proposed with no pilot or other preliminary data supporting the proposal and that its proposed sample size is based on pragmatics such as patient flow and budgetary constraints.
“It is quite reasonable and expected that a pilot study is proposed with no pilot or other preliminary data supporting the proposal and that its proposed sample size is based on pragmatics such as patient flow and budgetary constraints.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisenone givenA pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.”
- Also stated in: § INTRODUCTION · plain
“evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and/or implementation of the novel intervention.”
- Also stated in: § WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY · plain
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, and implementation of the novel intervention and each of these can be quantified”
Quote in the paperSubstantial claim- Also stated in: § INTRODUCTION · plain
May combine two claims.
Interpretationnone givenTwo fundamental reasons that hypothesis testing is not used in a pilot study are the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.
“There are two fundamental reasons that hypothesis testing is not used in a pilot study: the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningAs a consequence, pilot studies can reduce the proportion of failed trials and allow research funds to be spent on projects for which feasibility has been demonstrated and quantified.
“As a consequence, pilot studies can reduce the proportion of failed trials and allow research funds to be spent on projects for which feasibility has been demonstrated and quantified.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningWell-conceived and implemented pilot studies will reduce the risk of several problems commonly faced in clinical trials, including inability to recruit the proposed sample size and corresponding reduction in statistical power, excessive attrition due to intolerable procedures or interventions, and the need to modify a protocol midway through a trial.
“Well-conceived and implemented pilot studies will reduce the risk of several problems that are commonly faced in clinical trials. These include the inability to recruit the proposed sample size and a corresponding reduction in statistical power, excessive attrition due to intolerable procedures or interventions, and the need to modify a protocol midway through a trial.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenEstimates become more precise with larger sample sizes.
“Estimates become more precise with larger sample sizes.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
The true population effect size is unknown when a pilot study is designed.
“An essential challenge of therapeutic development is that the true population effect size is unknown at the time a pilot study is designed.”
The paper relies on this without support.
Methods will likely be modified after a pilot study, even if minimally.
“This is because it is quite likely that the methods will be modified after the pilot, even if minimally.”
The paper relies on this without support.
Pilot results presented to grant review committees tend to be selective, overly optimistic, and at times misrepresentational.
“Furthermore, pilot results that are presented to grant review committees tend to be selective, overly optimistic and, at times, misrepresentational.”
The paper relies on this without support.
Many pilot studies have far fewer than 18 participants per group.
“Many pilot studies have far fewer than 18 participants/group and therefore even greater imprecision.”
The paper relies on this without support.
It is not unusual for research grant applications and pilot study publications to convey optimism based on pilot data.
“Nevertheless, it is not unusual for research grant applications and pilot study publications to convey such optimism, particularly when based on pilot data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The planned sample size was 30 allowing for attrition.
“The planned sample size was 30 allowing for attrition. This study will inform the sample size for any future RCT (Leon et al., 2011).”
From A mixed methods pilot and feasibility open trial of internet-delivered cognitive behaviour therapy (iCanADAPT Advanced) for people with advanced cancer with depression and/or anxiety (2021) · cites this paper as Leon et al., 2011
The citation attaches to the statement that this pilot study will inform the sample size of a future RCT, not to the authors’ planned sample of 30. In “Pilot Study Effect Sizes and Sample Size Determination” (page unavailable), Leon et al. say pilot results should not be used to determine a subsequent trial’s sample size because small-sample effect estimates are imprecise. They do say pilots can assess recruitment and attrition, but do not endorse using pilot results to determine the RCT sample size. (LLM, conf=0.82)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThis pilot study did not have a power calculation.
“This was a pilot study which like other pilot studies of MI did not have a power calculation. 31,32”
From A Pilot Randomized Controlled Trial to Assess the Impact of Motivational Interviewing on Initiating Behavioral Therapy for Migraine (2020) · cites this paper as [31]
In “Sample Size Determination in Designing a Pilot Study,” the paper says power analyses should not be presented for pilot studies that do not propose inferential tests; sample size should instead reflect feasibility needs. It does not establish whether the citing authors’ particular study, or other MI pilot studies, actually omitted a power calculation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredData analyses in the context of a pilot study are not hypothesis-driven but serve to provide information regarding feasibility and acceptability of the intervention.
“Data analyses in the context of a pilot study are not hypothesis-driven but serve to provide information regarding feasibility and acceptability of the intervention [55].”
From Acceptance and Commitment Therapy Wellness Program for Latine Adults Who Smoke and Have Psychological Distress: Protocol for a Feasibility Study (2023) · cites this paper as [55]
The Abstract and Sample Size Determination section say pilot studies assess feasibility rather than test hypotheses, including the feasibility of implementing an intervention. The 'Rationale for a Control or Comparison Group' section mentions acceptability specifically for control-group aspects and procedures in psychotherapy or psychosocial interventions; it does not establish assessment of intervention acceptability as a general purpose of pilot-study analyses. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredA common misuse of pilot and feasibility studies has been the application of hypothesis testing for clinical outcomes in small under-powered studies.
“A common misuse of pilot and feasibility studies has been the application of hypothesis testing for clinical outcomes in small under-powered studies.”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
Hypothesis testing in pilot studies, particularly for effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations.
“Hypothesis testing in pilot studies, particularly in the context of effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations [2, 20, 21, 23].”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
No power calculation was required because the focus was on feasibility and acceptability and no hypothesis testing was performed.
“Since the focus was on feasibility and acceptability of the intervention and we did no hypothesis testing, no power calculation was required (Leon et al., 2011).”
From PandaMom – Feasibility and acceptability of an internet- and mobile-based intervention to enhance peripartum mental well-being and to prevent postpartum depression (2024) · cites this paper as Leon et al., 2011
The use of effect sizes was appropriate for this pilot trial.
“Our use of effect sizes was, however, deemed appropriate for this pilot trial (55, 63).”
From EMPOWER: A Multi-Site Pilot Trial to Reduce Distress in Surrogate Decision-Makers in the ICU (2024) · cites this paper as [63]
The section “Pilot Study Effect Sizes and Sample Size Determination” warns against using pilot effect sizes to plan a larger trial because the estimates are imprecise. It does not say whether calculating or reporting effect sizes in a pilot trial is appropriate, and the citing sentence does not specify how they were used. (LLM, conf=0.72)
Experimental — accuracy not yet measuredPrevious work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses.
“Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
A Bayesian approach might be more appropriate than Fisherian statistics as the Fisher approach is not appropriate with limited data.
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
Cites
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237 · see citing works
Porta, M. (2026) · doi:10.1093/acref/9780197663639.001.0001 · Cited by 2,175 · see citing works
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998) · doi:10.1007/978-1-4757-2915-3 · Cited by 873 · see citing works
. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702 · cited 1× · see citing works
“(Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006) Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples.”
Self-citationWolfstein (1905) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-190502000-00009 · Cited by 346 · see citing works
Attrition in randomized controlled clinical trials: methodological issues in psychopharmacology
Leon, A. C., Mallinckrodt, C. H., Chuang-Stein, C., Archibald, D. G., Archer, G. E., & Chartier, K. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.10.020 · Cited by 303 · cited 1× · see citing works
“(Leon et al., 2006; Leon & Davis, 2009).”
Self-citationSixteen <i>S</i> ‐squared over <i>D</i> ‐squared: A relation for crude sample size estimates
Lehr, R. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110811 · Cited by 253 · cited 1× · see citing works
“(N/group ≈ 16/d2; Lehr, 1992) The clinical interpretation of HAMD change of 1.2 to 3.0 would drive the choice among possible sample sizes in planning a study.”
BackgroundInfluentialJovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.1080/00031305.1997.10473947 · Cited by 179 · cited 1× · see citing works
“…a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n. (Jovanovic & Levy, 1997; Jovanovic et al., 1997) For example, consider a study with N=15 receiving medication and no suicidal ideation was reported…”
BackgroundHarvey, I. M. (2009) · Public Health · doi:10.1016/j.puhe.2009.05.009 · Cited by 173 · see citing works
Jovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.2307/2685405 · Cited by 50 · see citing works
Chlorpromazine and reserpine in the treatment of psychoses
Kinross‐Wright, V. (1955) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1955.tb42464.x · Cited by 46 · cited 2× · see citing works
“An enthusiastic investigator may well believe that a series of cases provides evidence of efficacy, but such data are observational and uncontrolled; realistically, they are seldom replicated in RCTs -- as seen years ago with reserpine (Kinross-Wright, 1955; Campden-Main, Wegielski, 1955; Goller 1960).”
BackgroundSafety Evaluation and Confidence Intervals When the Number of Observed Events Is Small or Zero
Jovanovic, B. D., & Zalenski, R. J. (1997) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(97)70165-3 · Cited by 44 · see citing works
(1986) · PubMed · Cited by 33 · see citing works
Enhancing clinical trial design of interventions for posttraumatic stress disorder
Leon, A. C., & Davis, L. L. (2009) · Journal of Traumatic Stress · doi:10.1002/jts.20466 · Cited by 26 · cited 1× · see citing works
“(Leon et al., 2006; Leon & Davis, 2009).”
Self-citationCampden‐Main, B. C., & Wegielski, Z. (1955) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1955.tb42458.x · Cited by 14 · see citing works
A controlled trial of reserpine in chronic schizophrenia
Goller, E. S. (1960) · Journal of Mental Science · doi:10.1192/bjp.106.445.1408 · Cited by 9 · cited 1× · see citing works
“…investigator may well believe that a series of cases provides evidence of efficacy, but such data are observational and uncontrolled; realistically, they are seldom replicated in RCTs -- as seen years ago with reserpine (Kinross-Wright, 1955; Campden-Main, Wegielski, 1955; Goller 1960).”
Background
Flagged first, then by citations
1 of the 17 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962 · see citing works
Senolytics in idiopathic pulmonary fibrosis: Results from a first-in-human, open-label, pilot study
Justice, J. N., Nambiar, A. M., Tchkonia, T., LeBrasseur, N. K., Pascual, R., Hashmi, S. K., Prata, L., Masternak, M. M., Kritchevsky, S. B., Musi, N., et al. (2019) · EBioMedicine · doi:10.1016/j.ebiom.2018.12.052 · Cited by 1,261 · see citing works
“However, the primary purpose of this pilot study was to evaluate study feasibility rather than drug efficacy [39,40].”
BackgroundSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136 · see citing works
“Furthermore, small sample sizes hamper the precision and accuracy of the statistical results and therefore limit our interpretations [45].”
BackgroundDo “brain-training” programs work?
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. L. (2016) · Psychological Science in the Public Interest · doi:10.1177/1529100616661983 · Cited by 1,183 · see citing works
Guidelines for designing and evaluating feasibility pilot studies
Teresi, J. A., Yu, X., Stewart, A. L., & Hays, R. D. (2022) · Medical Care · doi:10.1097/mlr.0000000000001664 · Cited by 734 · see citing works
Brickwood, K.-J., Watson, G., O'Brien, J., & Williams, A. D. (2019) · JMIR mHealth and uHealth · doi:10.2196/11819 · Cited by 690 · see citing works
“Potential reasons why other included studies did not find a significant intervention effect or observed a reduction in physical activity participation may be because of several of the studies being pilot studies [29,30,40,47,52], which often have insufficient power because of small sample sizes [93]; a moderate-to-hig…”
Jennings, P. A., Brown, J. L., Frank, J. L., Doyle, S., Oh, Y., Davis, R., Rasheed, D., DeWeese, A., DeMauro, A. A., Cham, H., et al. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000187 · Cited by 645 · see citing works
The Distinctive Features of a Feasibility Study
Orsmond, G., & Cohn, E. (2015) · OTJR: Occupation, Participation and Health · doi:10.1177/1539449215578649 · Cited by 616 · see citing works
“…focus on describing the variance in key outcomes and include the examination and presentation of confidence intervals (Thabane et al., 2010), examination of clinically meaningful effects (Dobkin, 2009; Leon et al., 2011), and computation of the reliable change index (Jacobson & Truax, 1991).”
BackgroundGuidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475 · see citing works
In, J. (2017) · Korean Journal of Anesthesiology · doi:10.4097/kjae.2017.70.6.601 · Cited by 547 · see citing works
“for the main trial needs to be performed with caution [2,14,16].”
BackgroundAarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015) · Implementation Science · doi:10.1186/s13012-014-0192-y · Cited by 435 · see citing works
“These items were developed by the research team specifically for this study to assess these three pilot study outcomes [49].”
MethodResultFava, M., Freeman, M. P., Flynn, M., Judge, H., Hoeppner, B. B., Cusin, C., Ionescu, D. F., Mathew, S. J., Chang, L. C., Iosifescu, D. V., et al. (2020) · Molecular Psychiatry · doi:10.1038/s41380-018-0256-5 · Cited by 433 · see citing works
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406 · see citing works
Nuts and Bolts of Conducting Feasibility Studies
Tickle‐Degnen, L. (2013) · American Journal of Occupational Therapy · doi:10.5014/ajot.2013.006270 · Cited by 397 · see citing works
Albers, C. J., & Lakens, D. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.09.004 · Cited by 366 · see citing works
“Pilot studies can be performed because they have other uses, such as determining the feasibility of performing the designed study in practice, which is especially useful when trying out new methods or procedures (Leon et al., 2011).”
BackgroundLewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A. (2021) · Pilot and Feasibility Studies · doi:10.1186/s40814-021-00770-x · Cited by 330 · see citing works
White, S. W., Ollendick, T. H., Albano, A. M., Oswald, D. P., Johnson, C. R., Southam‐Gerow, M. A., Kim, I., & Scahill, L. D. (2012) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-012-1577-x · Cited by 314 · see citing works
“The most likely explanation is that the sample sizes in these trials were small and estimates of effect size are therefore unstable (Leon et al. 2011).”
BackgroundResultInfluentialRapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise
Riley, W. T., Glasgow, R. E., Etheredge, L., & Abernethy, A. P. (2013) · Clinical and Translational Medicine · doi:10.1186/2001-1326-2-10 · Cited by 267 · see citing works
“This belabored research process typically begins with pilot trials that we posit have limited benefit, are used inappropriately to estimate effect size [14], and often prematurely concretize a less than optimal intervention.”
BackgroundPilot and feasibility studies come of age!
Lancaster, G. A. (2015) · Pilot and Feasibility Studies · doi:10.1186/2055-5784-1-1 · Cited by 260 · see citing works
“Whether or not hypothesis testing is to be used, what is clear is that the main objective of a pilot or feasibility study is not to test treatment effectiveness [10,15,19].”
BackgroundResultGames for Health for Children—Current Status and Needed Research
Baranowski, T., Blumberg, F. C., Buday, R., DeSmet, A., Fiellin, L. E., Green, C. S., Kato, P. M., Lu, A. S., Maloney, A. E., Mellecker, R. R., et al. (2015) · Games for Health Journal · doi:10.1089/g4h.2015.0026 · Cited by 245 · see citing works
“…who should be involved, as well as types of involvement that enhance game effectiveness, the optimal role of feasibility studies in game research, 108–112 the definition of feasibility in game interventions (i.e., when is the game considered feasible), sample sizes needed for feasibility studies…”
Almirall, D., Compton, S. N., Gunlicks‐Stoessel, M., Duan, N., & Murphy, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.4512 · Cited by 223 · see citing works
“Scientists preparing to execute a randomized trial should also refer to the literature on executing external pilot studies for pointers on more general uses of pilot studies [34–36].”
Whitehead, A. L., Sully, B., & Campbell, M. J. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.04.001 · Cited by 235 · see citing works
“[9] state that a pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods and implementation of the novel intervention.”
BackgroundAgile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · see citing works
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
BackgroundMethodWoo, S. L., Kung, T. A., Brown, D. L., Leonard, J. A., Kelly, B. M., & Cederna, P. S. (2016) · Plastic and Reconstructive Surgery - Global Open · doi:10.1097/gox.0000000000001038 · Cited by 230 · see citing works
Pachankis, J. E., McConocha, E. M., Clark, K. A., Wang, K., Behari, K., Fetzner, B. K., Brisbin, C. D., Scheer, J. R., & Lehavot, K. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000508 · Cited by 227 · see citing works
Mendoza, J. A., Baker, K. S., Moreno, M. A., Whitlock, K. B., Abbey‐Lambertz, M., Waite, A., Colburn, T. A., & Chow, E. J. (2017) · Pediatric Blood & Cancer · doi:10.1002/pbc.26660 · Cited by 222 · see citing works
Pediatric Delirium and Associated Risk Factors
Silver, G. H., Traube, C., Gerber, L. M., Sun, X., Kearney, J. A., Patel, A. K., & Greenwald, B. M. (2015) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000000356 · Cited by 191 · see citing works
“However, it is important not to overconclude based on these findings (34).”
BackgroundClough, B. A., March, S., Chan, R. J., Casey, L. M., Phillips, R., & Ireland, M. J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0526-3 · Cited by 186 · see citing works
“The majority (67%, n = 10) of these studies had less than 20 participants per condition, which limits any interpretations of population efficacy [51].”
BackgroundPossemato, K., Kuhn, E., Johnson, E., Hoffman, J. E., Owen, J. E., Kanuri, N., De Stefano, L., & Brooks, E. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.09.005 · Cited by 167 · see citing works
Applying mixed methods to pilot feasibility studies to inform intervention trials
Aschbrenner, K. A., Kruse, G. R., Gallo, J. J., & Clark, V. L. P. (2022) · Pilot and Feasibility Studies · doi:10.1186/s40814-022-01178-x · Cited by 188 · see citing works
“The primary aim of pilot feasibility studies is to examine the feasibility of interventions and the methods used to evaluate them to answer the overarching question “Can it work?” prior to examining “Does it work?” [6].”
BackgroundMethodMoberg, C., Niles, A., & Beermann, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12556 · Cited by 157 · see citing works
“Though the use of pilot studies for estimating effect sizes is questionable (cf [42]), this pilot study was the most relevant data available on which to base our power analysis.”
Powering population health research: Considerations for plausible and actionable effect sizes
Matthay, E. C., Hagan, E., Gottlieb, L. M., Tan, M. L., Vlahov, D., Adler, N. E., & Glymour, M. M. (2021) · SSM - Population Health · doi:10.1016/j.ssmph.2021.100789 · Cited by 182 · see citing works
“Likely effect sizes may be estimated based on pilot studies, theories of change, causal models, expert opinion, or scientific literature on similar interventions (Leon et al., 2011; Matthay, 2020; Thabane et al., 2010).”
BackgroundChristopher, M. S., Hunsinger, M., Goerling, L. R. J., Bowen, S., Rogers, B. S., Gross, C. R., Dapolonia, E., & Pruessner, J. C. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.03.059 · Cited by 173 · see citing works
“Although recent guidelines suggest preliminary e ffi cacy testing not be included in pilot studies (Leon et al., 2011), given the dearth of e ff ective interventions among LEOs (Patterson et al., 2014), we deemed it important to include these promising outcome data.”
Detecting pediatric delirium: development of a rapid observational assessment tool
Silver, G. H., Traube, C., Kearney, J. A., Kelly, D., Yoon, M. J., Moyal, W. N., Gangopadhyay, M., Shao, H., & Ward, M. J. (2012) · Intensive Care Medicine · doi:10.1007/s00134-012-2518-z · Cited by 177 · see citing works
“Although pilot studies, by their very nature, are imprecise [32], the 29% prevalence rate of delirium in our study group suggests that pediatric delirium is a common and under-recognized problem that requires the PICU community’s attention.”
BackgroundCollins, L. M., Kugler, K. C., & Gwadz, M. V. (2015) · AIDS and Behavior · doi:10.1007/s10461-015-1145-4 · Cited by 162 · see citing works
“[68]: ‘‘The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study.”
A Step-by-Step Process on Sample Size Determination for Medical Research
Bujang, M. A. (2021) · Malaysian Journal of Medical Sciences · doi:10.21315/mjms2021.28.2.2 · Cited by 177 · see citing works
Luby, J. L., Lenze, S. N., & Tillman, R. (2011) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2011.02483.x · Cited by 152 · see citing works
Monti, D., Zabrecky, G. P., Kremens, D. E., Liang, T., Wintering, N., Cai, J., Wei, X., Bazzan, A. J., Li, Z., Bowen, B., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157602 · Cited by 151 · see citing works
“Overall, this was a pilot study designed to generate hypothesis-testing data that could inform clinical trials adequately powered to assess safety and efficacy [61].”
MethodConner, C. M., White, S. W., Beck, K. B., Golt, J., Smith, I. C., & Mazefsky, C. A. (2019) · Autism · doi:10.1177/1362361318810709 · Cited by 150 · see citing works
“In accordance with guidelines regarding treatment development, the primary aim of this open trial was to gauge feasibility and acceptability of the new intervention (Leon et al., 2011).”
Kohrt, B. A., Jordans, M. J. D., Turner, E. L., Sikkema, K. J., Luitel, N. P., Rai, S., Singla, D. R., Lamichhane, J., Lund, C., & Patel, V. (2018) · Pilot and Feasibility Studies · doi:10.1186/s40814-018-0234-3 · Cited by 141 · see citing works
Beets, M. W., Weaver, R. G., Ioannidis, J. P. A., Geraci, M., Brazendale, K., Decker, L., Okely, A. D., Lubans, D. R., van Sluijs, E. M. F., Jago, R., et al. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-0918-y · Cited by 142 · see citing works
Keefe, R. S. E., Buchanan, R. W., Marder, S. R., Schooler, N. R., Dugar, A., Zivkov, M., & Stewart, M. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbr153 · Cited by 138 · see citing works
O'Mahen, H., Himle, J. A., Fedock, G., Henshaw, E., & Flynn, H. (2013) · Depression and Anxiety · doi:10.1002/da.22050 · Cited by 136 · see citing works
Lebares, C. C., Guvva, E. V., Olaru, M., Sugrue, L. P., Staffaroni, A. M., Delucchi, K. L., Kramer, J. H., Ascher, N. L., & Harris, H. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.4108 · Cited by 131 · see citing works
Update on the State of the Evidence for Stroke Family Caregiver and Dyad Interventions
Bakas, T., McCarthy, M. J., & Miller, E. (2017) · Stroke · doi:10.1161/strokeaha.117.016052 · Cited by 130 · see citing works
Should treatment effects be estimated in pilot and feasibility studies?
Sim, J. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0493-7 · Cited by 123 · see citing works
Sharp, W. G., Burrell, T. L., & Jaquess, D. L. (2013) · Autism · doi:10.1177/1362361313489190 · Cited by 123 · see citing works
“With this backdrop in place, the purpose of this article is to (a) describe a behaviorally based parent-training program—the Autism MEAL Plan—a curriculum specifically developed to assist caregivers to Manage Eating Aversions and Low intake among children with ASD; (b) evaluate the feasibility of the parent-training p…”
MethodConducting a pilot study: case study of a novice researcher
Doody, O., & Doody, C. M. (2015) · British Journal of Nursing · doi:10.12968/bjon.2015.24.21.1074 · Cited by 123 · see citing works
“…be made before the large-scale study is undertaken and answer methodological question(s), guide the development of the research plan to ensure the methods work in practice and assess the feasibility of the proposed research process (Hundley and van Teijlingen, 2002; Kim, 2011; Leon et al, 2011).”
InfluentialPerepletchikova, F., Nathanson, D., Axelrod, S., Merrill, C. R. S., Walker, A., Grossman, M., Rebeta, J., Scahill, L., Kaufman, J., Flye, B., et al. (2017) · Journal of the American Academy of Child and Adolescent Psychiatry · doi:10.1016/j.jaac.2017.07.789 · Cited by 111 · see citing works
“The 6 domains of feasibility included recruitment, randomization, retention, attendance, participants’ satisfaction, and therapist adherence.(22) The following benchmarks were set for DBT-C feasibility: at least 2 participants per month for recruitment; dropout rate lower than 30% for retention (defined as dropped out…”
BackgroundNaltrexone vs Placebo for the Treatment of Alcohol Dependence
Oslin, D. W., Leong, S. H., Lynch, K. G., Berrettini, W. H., O’Brien, C. P., Gordon, A. J., & Rukstalis, M. R. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.3053 · Cited by 118 · see citing works
Vulnerability to Depression in Youth: Advances From Affective Neuroscience
Kujawa, A., & Burkhouse, K. L. (2016) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.09.006 · Cited by 118 · see citing works
Rixon, L., Baron, J., McGale, N., Lorencatto, F., Francis, J., & Davies, A. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1904-6 · Cited by 117 · see citing works
“Design* RCT 16 (29.1) Quasi-experimental 3 (5.5) Cluster RCT 12 (38.2) Case study 1 (1.8) Pilot/feasibility 15 (27.3) Controlled 1 (1.8) Pre-post design 2 (3.6) Unclear 5 (9.1) Intervention recipients People with health conditions 17 (30.9) Employees/workers 5 (9.1) Family/informal carers 6 (10.9) Care home staff and…”
MethodInfluentialA Pilot Study: Vital Methodological Issues
Hazzi, O. A., & Maldaon, I. (2015) · Verslas teorija ir praktika · doi:10.3846/btp.2015.437 · Cited by 121 · see citing works
Wesson, J., Clemson, L. M., Brodaty, H., Lord, S. R., Taylor, M. E., Gitlin, L. N., & Close, J. (2013) · BMC Geriatrics · doi:10.1186/1471-2318-13-89 · Cited by 115 · see citing works
“Therefore, efficacy or intervention effects cannot be conclusively determined from our results, in keeping with the aims of a feasibility trial [32].”
ResultBehavioral activation for children and adolescents: a systematic review of progress and promise
Martin, F., & Oliver, T. (2019) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-018-1126-z · Cited by 111 · see citing works
“A detailed review is required not only in relation with the potential effectiveness of the intervention, but also for how the intervention can be delivered and implemented, which should be considered early in the intervention development process [28, 29].”
BackgroundPope, Z., Barr‐Anderson, D. J., Lewis, B. A., Pereira, M. A., & Gao, Z. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16193579 · Cited by 118 · see citing works
“Given the study’s pilot nature, our primary outcomes included intervention interest, use/acceptability, adherence, and retention, as recommended by the National Institutes of Health’s National Center for Complementary and Integrative Health (NCCIH) [36] and others [37] for the reporting of pilot trials.”
BackgroundMethodRice, E. R. W., Tulbert, E., Cederbaum, J. A., Adhikari, A. B., & Milburn, N. G. (2012) · Health Education Research · doi:10.1093/her/cyr113 · Cited by 107 · see citing works
“[50] suggest that pilot data should not be used to conduct hypothesis testing about outcomes but rather efforts should be focused on collecting data that can guide in the design and implementation of larger scale efficacy studies.”
BackgroundMethodMeasuring Treatment Fidelity in a Rehabilitation Intervention Study
Hildebrand, M. W., Host, H., Binder, E., Carpenter, B., Freedland, K., Morrow-Howell, N., Baum, C., Doré, P. M., & Lenze, E. (2012) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e31824ad462 · Cited by 96 · see citing works
Lee, S., & Ha‐Brookshire, J. (2017) · Sustainability · doi:10.3390/su9030465 · Cited by 107 · see citing works
“To test feasibility, equipment, and methods, an initial pilot study was conducted, which is a small-scale preliminary study of the larger research design [80].”
MethodDeal, J. A., Albert, M., Arnold, M. L., Bangdiwala, S. I., Chisolm, T. H., Thomas, S., Eddins, A. C., Glynn, N. W., Goman, A. M., Minotti, M., et al. (2017) · Alzheimer s & Dementia Translational Research & Clinical Interventions · doi:10.1016/j.trci.2017.06.003 · Cited by 106 · see citing works
“Given limitations of this pilot study, however, including small sample size, the magnitude of the effects reported here should not be interpreted as that would be estimated in a fully powered trial [21].”
BackgroundMcguire, J. F., Arnold, E. B., Park, J. M., Nadeau, J. M., Lewin, A. B., Murphy, T. K., & Storch, E. A. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.11.045 · Cited by 105 · see citing works
Fraser, J. H., Fahlman, D., Arscott, J., & Guillot, I. (2018) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v19i1.3326 · Cited by 114 · see citing works
“Nevertheless, as described above, properly conducted pilot studies can greatly strengthen outcomes of the main study regardless of the field (van Teijlingen & Hundley, 2002; Cohen, Manion, & Morrison, 2007; Gudmundsdottir & Brock-Utne, 2010; Leon et al., 2011; Doody & Doody, 2015).”
BackgroundResultInfluentialKooistra, L. C., Ruwaard, J., Wiersma, J. E., van Oppen, P., van der Vaart, R., van Gemert-Pijnen, J. E. W. C., & Riper, H. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.01.003 · Cited by 92 · see citing works
“In order to gain insight into the feasibility (Leon et al., 2011) of bCBT, we first examined the reach of the intervention in terms of the proportion of patients enrolled at the mental health care centre that was eligible for bCBT.”
BackgroundLearmonth, Y. C., Adamson, B., Kinnett-Hopkins, D., Bohri, M., & Motl, R. W. (2016) · Contemporary Clinical Trials · doi:10.1016/j.cct.2016.11.012 · Cited by 102 · see citing works
Systematic review of the evidence for stroke family caregiver and dyad interventions
Bakas, T., McCarthy, M. J., & Miller, E. L. (2022) · Stroke · doi:10.1161/strokeaha.121.034090 · Cited by 109 · see citing works
Neonatal pain: What′s age got to do with it?
Hatfield, L. (2014) · Surgical Neurology International · doi:10.4103/2152-7806.144630 · Cited by 102 · see citing works
“[29] During gestation, neuronal tissues acquire specific patterns of DNA methylation to regulate genome function.[96] Increased methylation inhibits gene expression, decreased methylation increases gene expression.”
BackgroundSharp, W. G., Stubbs, K. H., Adams, H., Wells, B. M., Lesack, R. S., Criado, K. K., Simon, E. L., McCracken, C. E., West, L. L., & Scahill, L. D. (2016) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000001043 · Cited by 102 · see citing works
Quintiliani, L. M., Mann, D. M., Puputti, M., Quinn, E. K., & Bowen, D. J. (2016) · JMIR Cancer · doi:10.2196/cancer.5305 · Cited by 100 · see citing works
“Although it is appropriate for sample sizes for pilot studies to be based on practical considerations based on recruitment and budgetary limitations [46], a larger sample size may have expanded our capacity to generate feasibility data.”
Wälti, P., Kool, J., & Luomajoki, H. A. (2015) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-015-0533-2 · Cited by 101 · see citing works
“The outcomes of this study have to be interpreted with caution because of the small sample size, as there is evidence that the risk of a type 1 error is larger in small sample sizes [39,41].”
Feasibility of "CopeSmart": A Telemental Health App for Adolescents
Kenny, R., Dooley, B., & Fitzgerald, A. (2015) · JMIR Mental Health · doi:10.2196/mental.4370 · Cited by 79 · see citing works
“Feasibility studies achieve this by exploring factors such as the acceptability of an intervention to participants, adherence and drop-out rates, and aspects of the intervention that may be altered to improve participants’ engagement [26], and are considered a crucial first step prior to the conduction of larger scale…”
Bjureberg, J., Sahlin, H., Hedman-Lagerlöf, E., Gratz, K. L., Tull, M. T., Jokinen, J., Hellner, C., & Ljótsson, B. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1885-6 · Cited by 96 · see citing works
Montag, C., Haase, L., Seidel, D., Bayerl, M., Gallinat, J., Herrmann, U., & Dannecker, K. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112348 · Cited by 97 · see citing works
“Results may therefore only give a very rough indication of the direction of the treatment effect but cannot serve to confirm or reject our hypotheses, and even results on feasibility might not generalise [78].”
Kiosses, D. N., Leon, A. C., & Areán, P. A. (2011) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2011.03.001 · Cited by 91 · see citing works
Self-citationRapaport, M. H., Schettler, P. J., & Bresee, C. J. (2012) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2011.0071 · Cited by 96 · see citing works
“Small sample sizes can lead to an overestimate of effect sizes and thus, the data must be considered within the context of this caveat.(31,34) A second concern in studies where one is attempting to measure a decrease in hormones normally associated with a stress response in unstressed, normal volunteers is that there…”
BackgroundKhan, M. N., Hamdani, S. U., Chiumento, A., Dawson, K., Bryant, R. A., Sijbrandij, M., Nazir, H., Akhtar, P., Masood, A., Wang, D., et al. (2019) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796017000336 · Cited by 82 · see citing works
“Since this was a feasibility trial, no power calculations were carried out (Leon et al. 2011).”
BackgroundFarrell, L. J., Waters, A. M., Boschen, M. J., Hattingh, L., McConnell, H. W., Oar, E. L., Collings, N., Zimmer‐Gembeck, M. J., Shelton, D., Ollendick, T. H., et al. (2013) · Depression and Anxiety · doi:10.1002/da.22132 · Cited by 92 · see citing works
“Furthermore, a pilot study of this nature is an essential step toward the advancement and refinement of new technologies and procedures.[46] Results of pilot studies, such as this are necessary to”
BackgroundSrinivasan, S. M., Eigsti, I., Gifford, T. D., & Bhat, A. N. (2016) · Research in Autism · doi:10.1016/j.rasd.2016.04.001 · Cited by 91 · see citing works
COGNITIVE-EMOTIONAL TRAINING AS AN INTERVENTION FOR MAJOR DEPRESSIVE DISORDER
Iacoviello, B. M., Wu, G., Alvarez, E., Huryk, K. M., Collins, K. A., Murrough, J. W., Iosifescu, D. V., & Charney, D. S. (2014) · Depression and Anxiety · doi:10.1002/da.22266 · Cited by 90 · see citing works
“Lastly, it has been shown that pilot studies with small sample sizes can yield inflated effect sizes (the confidence interval for the true effect size in a small pilot study can be very large and difficult to interpret[52]) and have demonstrated a history of not replicating the large effects that were initially observ…”
BackgroundOn the Unhappiness of Software Developers
Graziotin, D., Fagerholm, F., Wang, X., & Abrahamsson, P. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084242 · Cited by 96 · see citing works
“We discarded the pilots’ contact data and questionnaire data from the final survey as many guidelines recommend (e.g., [48]).”
MethodCorsini-Munt, S., Bergeron, S., Rosen, N. O., Mayrand, M.-H., & Delisle, I. (2014) · The Journal of Sexual Medicine · doi:10.1111/jsm.12646 · Cited by 87 · see citing works
Kohrt, B. A., Jordans, M. J. D., Turner, E. L., Rai, S., Gurung, D., Dhakal, M., Bhardwaj, A., Lamichhane, J., Singla, D. R., Lund, C., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.31475 · Cited by 90 · see citing works
Acceptance checklist for clinical effectiveness pilot trials: a systematic approach
Charlesworth, G., Burnell, K., Hoe, J., Orrell, M., & Russell, I. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-78 · Cited by 87 · see citing works
“For example: as a substitute for hypothesistesting studies rather than being preparation for a full trial; to make inadequately powered studies sound more attractive to publishers; and, to report on outcomes rather than design, methods and procedures [2,9,10].”
BackgroundHassan, N. M., Landorf, K. B., Shields, N., & Munteanu, S. E. (2018) · Journal of Intellectual Disability Research · doi:10.1111/jir.12562 · Cited by 87 · see citing works
Shomaker, L. B., Bruggink, S. M., Pivarunas, B., Skoranski, A. M., Foss, J., Chaffin, E., Dalager, S. L., Annameier, S. K., Quaglia, J. T., Brown, K. W., et al. (2017) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2017.04.003 · Cited by 88 · see citing works
Well-being and self-efficacy in a sample of undergraduate nurse students: A small survey study
Priesack, A., & Alcock, J. (2015) · Nurse Education Today · doi:10.1016/j.nedt.2015.01.022 · Cited by 83 · see citing works
“Despite thepositive results, it should be regarded as a pilot study (Leon et al., 2011) and the results considered with caution in relation to making generalisations.”
ResultBjureberg, J., Sahlin, H., Hellner, C., Hedman-Lagerlöf, E., Gratz, K. L., Bjärehed, J., Jokinen, J., Tull, M. T., & Ljótsson, B. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1527-4 · Cited by 84 · see citing works
“Following recommendations for early studies in clinical research [21, 22], the present pilot study examined the feasibility, acceptance, and utility of this ERITA in an open uncontrolled pilot study.”
MethodDevoe, D. J. A., Peterson, A. M., & Addington, J. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx139 · Cited by 79 · see citing works
Conner, C. M., & White, S. W. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-017-3312-0 · Cited by 78 · see citing works
“As the primary role of pilot studies is to determine the feasibility of the intervention (Leon et al. 2011), the aim of the current study was to analyze the feasibility and efficacy of a MABI targeting ER difficulties for adults with ASD.”
BackgroundA systematic review of interventions to provide genetics education for primary care
Paneque, M., Turchetti, D., Jackson, L. M., Lunt, P. W., Houwink, E. J. F., & Skirton, H. (2016) · BMC Family Practice · doi:10.1186/s12875-016-0483-2 · Cited by 77 · see citing works
“As the purpose of a pilot study [21] is to assess the”
BackgroundAbrantes, A. M., Blevins, C. E., Battle, C. L., Read, J. P., Gordon, A. L., & Stein, M. D. (2017) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2017.07.006 · Cited by 78 · see citing works
The significance filter, the winner's curse and the need to shrink
van Zwet, E. W., & Cator, E. (2020) · Statistica neerlandica (Print) · doi:10.1111/stan.12241 · Cited by 71 · see citing works
“are often used to inform the sample size calculation of a larger trial [13],[7].”
MethodField, T. S., Dizonno, V., Almekhlafi, M., Bala, F., Alhabli, I., Wong, H., Norena, M., Villaluna, M. K., King-Azote, P., Ratnaweera, N., et al. (2023) · Stroke · doi:10.1161/strokeaha.123.044113 · Cited by 80 · see citing works
Orchowski, L. M., Barnett, N. P., Berkowitz, A. D., Borsari, B. E., Oesterle, D. W., & Zlotnick, C. (2018) · Violence Against Women · doi:10.1177/1077801218787928 · Cited by 76 · see citing works
Li, F., Harmer, P., Fitzgerald, K. M., & Winters‐Stone, K. M. (2022) · BMC Geriatrics · doi:10.1186/s12877-021-02747-0 · Cited by 78 · see citing works
Shimshoni, Y., Silverman, W. K., & Lebowitz, E. R. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23341 · Cited by 79 · see citing works
King, S. H., & Green, H. J. (2015) · Frontiers in Oncology · doi:10.3389/fonc.2015.00072 · Cited by 74 · see citing works
“Research has suggested that effect sizes reported in pilot studies often differ from consequent RCT studies implementing the same interventions (114).”
ResultSylvia, L., Salcedo, S., Bernstein, E. E., Baek, J., Nierenberg, A., & Deckersbach, T. (2013) · International Journal of Bipolar Disorders · doi:10.1186/2194-7511-1-24 · Cited by 74 · see citing works
“…be premature to conduct inferential statistics at this time and instead, for each measure, we calculated the mean, standard deviation, 95% confidence interval (CI), and then used Cohen’s d effect sizes for repeated measures to determine the magnitude of the changes we observed (Leon et al. 2011).”
Srinivasan, S. M., Eigsti, I., Neelly, L. P., & Bhat, A. N. (2016) · Research in Autism · doi:10.1016/j.rasd.2016.01.004 · Cited by 76 · see citing works
Murphy, T. K., Fernandez, T. V., Coffey, B. J., Rahman, O., Gavaletz, A., Hanks, C. E., Tillberg, C. S., Ibanez-Gomez, L., Sukhodolsky, D. G., Katsovich, L., et al. (2017) · Journal of Child and Adolescent Psychopharmacology · doi:10.1089/cap.2017.0024 · Cited by 75 · see citing works
“Pilot studies cannot be expected to prove efficacy (Leon et al. 2011).”
- Authors
- Leon, A. C., Davis, L. L., & Kraemer, H. C.
- Affiliations
- Weill Cornell Medicine (US)
- University of Alabama (US)
- Stanford University (US)
- Corresponding author
- Leon, Andrew C.
- Venue
- Journal of Psychiatric Research
- Publisher
- Elsevier BV
- Published
- 2011-05
- Volume
- 45
- Issue
- 5
- Pages
- 626-629
- ISSN
- 0022-3956 (ISSN-L)
- PMC
- PMC3081994
- OpenAlex
- W2168434488
- Cited by
- 1,917
- References
- 17
- Citations by year
- 2026: 203 · 2025: 190 · 2024: 143 · 2023: 169 · 2022: 163 · 2021: 217 · 2020: 169 · 2019: 136 · 2018: 128 · 2017: 120 · 2016: 92 · 2015: 80 · 2014: 52 · 2013: 31 · 2012: 17
- FWCI
- 14.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Psychiatric Research (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (P30MH068638, R01MH060447)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- pilot study, feasibility assessment, clinical trials, research design, sample size determination, effect size estimation, generalizability, recruitment and retention, intervention implementation, research funding review
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Feasibility Studies, Humans, Pilot Projects, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-28.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
