The DOI matches its Crossref record; 16 of the 17 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.
The role and interpretation of pilot studies in clinical research
Open accessAbstract
Pilot studies represent a fundamental phase of the research process. The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design. A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study. Investigators should be forthright in stating these objectives of a pilot study. Grant reviewers and other stakeholders should expect no more.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own interpretation · 5 No basis given · 2 No full text of the sources · 1 Sources not found
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“When event rates are reported and no adverse event is observed for a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Tolerability and Preliminary Safety
What the sources say
- Not matchedcites Jovanovic & Levy, 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jovanovic et al., 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Basing a decision to proceed or terminate evaluation of a particular intervention on pilot data is perilous because there is a very good chance that the decision will be derived from false positive or false negative results.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each aspect of feasibility could be quite different from an uncontrolled study when intervention assignment is randomized and blinded, particularly if placebo is a distinct possibility.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Rationale for a Control or Comparison Group in a Pilot
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With these issues in mind, a pilot study can better address its goals if a control group is part of the design.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Rationale for a Control or Comparison Group in a Pilot
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A case series estimate tends to be steeped in optimism, particularly if an estimate of such magnitude is seldom, if ever, seen in full scale trials for psychiatric disorders.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(It is the population effect size, not the sample effect size, that an RCT is designed to detect with sufficient power.)”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“There has been a venerable tradition of using pilot studies to estimate between group effect sizes that, in turn, are used to inform the design of subsequent larger scale hypothesis testing studies. Despite it widespread use, it has been argued that the tradition is ill-founded. (Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006)”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
What the sources say
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot sample size is instead based on the pragmatics of recruitment and the necessities for examining feasibility.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With no inferential statistical tests, a pilot study will not provide p-values.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot study data generally should not be combined with data from the subsequent larger scale study. This is because it is quite likely that the methods will be modified after the pilot, even if minimally. Such changes in protocol risk adding additional, perhaps unknown, source of variation.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Are Pilot Data Included in the Larger Trial?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Methods will likely be modified after a pilot study, even if minimally.
- It may state two separate claims as one. 13
- Probably not a substantial claim. 13
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study.”
- Also stated in: § DISCUSSION · plain
“Pilot results inform feasibility, which in turn, is instructive in that it points to modifications needed in the planning and design of a larger efficacy trial.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DISCUSSION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The definition of clinically meaningful is not entirely empirically-based, but instead requires input from clinicians who treat the patient population of interest and perhaps from patients with the disorder.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These good clinical practices include the refinement of source documentation, informed consent procedures, data collection tools, regulatory reporting procedures, and monitoring/oversight procedures, especially when multiple sites and investigators are engaged in the study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Safety, efficacy and effectiveness are not evaluated in a pilot.”
- Also stated in: § DISCUSSION · plain
“a pilot study is not a hypothesis testing study and therefore safety and efficacy are not evaluated.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DISCUSSION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot studies generate pilot data, their design need not be guided with the support of prior pilot data.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is possible, but highly unlikely, that the between group effect size (d) from a pilot study sample will provide a reasonable estimate of the population effect size (Δ), but that cannot be known based on the pilot data.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot study can be critical in research staff training and provide experiences that strengthen and confirm competencies and skills required for the investigation to be conducted with accuracy and precision.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An alternative approach is to base sample size estimates for a full scale RCT on what is deemed a clinically meaningful effect.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > An Alternative to Using Pilot Data for Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot study provides opportunity to develop consistent practices to enhance data integrity and the protection of human subjects.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We learn little if anything about the efficacy of an intervention with data from a small sample; yet, as discussed earlier, a great deal can be learned from a pilot study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Hypothetical Example
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This applies to effects sizes for any type of outcome, be it a severity rating, a response status, or survival status.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, with imprecision comes a vast disparity in sample size estimates and, if sample size determination for a subsequent study is based on an imprecise estimate, there is an enormous risk of underpowered or overpowered design.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Hypothetical Example
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, pilot studies are a necessary first step in exploring novel interventions and novel applications of interventions – whether in a new patient population or with a novel delivery system (e.g., transdermal patch).”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although a pilot study will undoubtedly incorporate relevant outcome measures and can serve a vital role in treatment development, it is not, and should not, be considered a preliminary test of the intervention hypothesis.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The primary role of a pilot study is to examine the feasibility of a research endeavor.”
- Also stated in: § INTRODUCTION · plain
“The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This does not preclude the need for a theoretical rationale for the intervention or the methodology being proposed for a pilot study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a pilot study effect size is unduly large (i.e., a false positive result), subsequent trials will be designed with an inadequate number of participants to provide the statistical power needed to detect clinically meaningful effects and that would lead to negative trials.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, pilot studies provide an opportunity to implement and examine the feasibility of the adverse event reporting system.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Tolerability and Preliminary Safety
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is quite reasonable and expected that a pilot study is proposed with no pilot or other preliminary data supporting the proposal and that its proposed sample size is based on pragmatics such as patient flow and budgetary constraints.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.”
- Also stated in: § INTRODUCTION · plain
“evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and/or implementation of the novel intervention.”
- Also stated in: § WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY · plain
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, and implementation of the novel intervention and each of these can be quantified”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are two fundamental reasons that hypothesis testing is not used in a pilot study: the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a consequence, pilot studies can reduce the proportion of failed trials and allow research funds to be spent on projects for which feasibility has been demonstrated and quantified.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Well-conceived and implemented pilot studies will reduce the risk of several problems that are commonly faced in clinical trials. These include the inability to recruit the proposed sample size and a corresponding reduction in statistical power, excessive attrition due to intolerable procedures or interventions, and the need to modify a protocol midway through a trial.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimates become more precise with larger sample sizes.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
The true population effect size is unknown when a pilot study is designed.
“An essential challenge of therapeutic development is that the true population effect size is unknown at the time a pilot study is designed.”
The paper relies on this without support.
Methods will likely be modified after a pilot study, even if minimally.
“This is because it is quite likely that the methods will be modified after the pilot, even if minimally.”
The paper relies on this without support.
Pilot results presented to grant review committees tend to be selective, overly optimistic, and at times misrepresentational.
“Furthermore, pilot results that are presented to grant review committees tend to be selective, overly optimistic and, at times, misrepresentational.”
The paper relies on this without support.
Many pilot studies have far fewer than 18 participants per group.
“Many pilot studies have far fewer than 18 participants/group and therefore even greater imprecision.”
The paper relies on this without support.
It is not unusual for research grant applications and pilot study publications to convey optimism based on pilot data.
“Nevertheless, it is not unusual for research grant applications and pilot study publications to convey such optimism, particularly when based on pilot data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The planned sample size was 30 allowing for attrition. This study will inform the sample size for any future RCT (Leon et al., 2011).”
From A mixed methods pilot and feasibility open trial of internet-delivered cognitive behaviour therapy (iCanADAPT Advanced) for people with advanced cancer with depression and/or anxiety (2021) · cites this paper as Leon et al., 2011
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was a pilot study which like other pilot studies of MI did not have a power calculation. 31,32”
From A Pilot Randomized Controlled Trial to Assess the Impact of Motivational Interviewing on Initiating Behavioral Therapy for Migraine (2020) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Data analyses in the context of a pilot study are not hypothesis-driven but serve to provide information regarding feasibility and acceptability of the intervention [55].”
From Acceptance and Commitment Therapy Wellness Program for Latine Adults Who Smoke and Have Psychological Distress: Protocol for a Feasibility Study (2023) · cites this paper as [55]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A common misuse of pilot and feasibility studies has been the application of hypothesis testing for clinical outcomes in small under-powered studies.”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Hypothesis testing in pilot studies, particularly in the context of effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations [2, 20, 21, 23].”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Since the focus was on feasibility and acceptability of the intervention and we did no hypothesis testing, no power calculation was required (Leon et al., 2011).”
From PandaMom – Feasibility and acceptability of an internet- and mobile-based intervention to enhance peripartum mental well-being and to prevent postpartum depression (2024) · cites this paper as Leon et al., 2011
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our use of effect sizes was, however, deemed appropriate for this pilot trial (55, 63).”
From EMPOWER: A Multi-Site Pilot Trial to Reduce Distress in Surrogate Decision-Makers in the ICU (2024) · cites this paper as [63]
Experimental — accuracy not yet measuredIn the citing paper
“Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
In the citing paper
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
Citations
Cites
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237
Porta, M. (2026) · doi:10.1093/acref/9780197663639.001.0001 · Cited by 2,175
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998) · doi:10.1007/978-1-4757-2915-3 · Cited by 873
. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702 · cited 1×
“(Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006) Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples.”
Self-citationWolfstein (1905) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-190502000-00009 · Cited by 346
Attrition in randomized controlled clinical trials: methodological issues in psychopharmacology
Leon, A. C., Mallinckrodt, C. H., Chuang-Stein, C., Archibald, D. G., Archer, G. E., & Chartier, K. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.10.020 · Cited by 303 · cited 1×
“(Leon et al., 2006; Leon & Davis, 2009).”
Self-citationSixteen S ‐squared over D ‐squared: A relation for crude sample size estimates
Lehr, R. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110811 · Cited by 253 · cited 1×
“(N/group ≈ 16/d2; Lehr, 1992) The clinical interpretation of HAMD change of 1.2 to 3.0 would drive the choice among possible sample sizes in planning a study.”
BackgroundInfluentialJovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.1080/00031305.1997.10473947 · Cited by 179 · cited 1×
“…a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n. (Jovanovic & Levy, 1997; Jovanovic et al., 1997) For example, consider a study with N=15 receiving medication and no suicidal ideation was reported…”
BackgroundHarvey, I. M. (2009) · Public Health · doi:10.1016/j.puhe.2009.05.009 · Cited by 173
Jovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.2307/2685405 · Cited by 50
Cited by
Sorted by citations
Lakens, D. (2022) · Collabra Psychology · doi:10.1525/collabra.33267 · Cited by 1,962
Senolytics in idiopathic pulmonary fibrosis: Results from a first-in-human, open-label, pilot study
Justice, J. N., Nambiar, A. M., Tchkonia, T., LeBrasseur, N. K., Pascual, R., Hashmi, S. K., Prata, L., Masternak, M. M., Kritchevsky, S. B., Musi, N., et al. (2019) · EBioMedicine · doi:10.1016/j.ebiom.2018.12.052 · Cited by 1,261
“However, the primary purpose of this pilot study was to evaluate study feasibility rather than drug efficacy [39,40].”
BackgroundSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136
“Furthermore, small sample sizes hamper the precision and accuracy of the statistical results and therefore limit our interpretations [45].”
BackgroundDo “brain-training” programs work?
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. L. (2016) · Psychological Science in the Public Interest · doi:10.1177/1529100616661983 · Cited by 1,183
Guidelines for designing and evaluating feasibility pilot studies
Teresi, J. A., Yu, X., Stewart, A. L., & Hays, R. D. (2022) · Medical Care · doi:10.1097/mlr.0000000000001664 · Cited by 734
Brickwood, K.-J., Watson, G., O'Brien, J., & Williams, A. D. (2019) · JMIR mHealth and uHealth · doi:10.2196/11819 · Cited by 689
“Potential reasons why other included studies did not find a significant intervention effect or observed a reduction in physical activity participation may be because of several of the studies being pilot studies [29,30,40,47,52], which often have insufficient power because of small sample sizes [93]; a moderate-to-hig…”
Jennings, P. A., Brown, J. L., Frank, J. L., Doyle, S., Oh, Y., Davis, R., Rasheed, D., DeWeese, A., DeMauro, A. A., Cham, H., et al. (2017) · Journal of Educational Psychology · doi:10.1037/edu0000187 · Cited by 645
The Distinctive Features of a Feasibility Study
Orsmond, G. I., & Cohn, E. S. (2015) · OTJR Occupational Therapy Journal of Research · doi:10.1177/1539449215578649 · Cited by 649
“…focus on describing the variance in key outcomes and include the examination and presentation of confidence intervals (Thabane et al., 2010), examination of clinically meaningful effects (Dobkin, 2009; Leon et al., 2011), and computation of the reliable change index (Jacobson & Truax, 1991).”
BackgroundGuidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
In, J. (2017) · Korean Journal of Anesthesiology · doi:10.4097/kjae.2017.70.6.601 · Cited by 547
“for the main trial needs to be performed with caution [2,14,16].”
BackgroundFava, M., Freeman, M. P., Flynn, M., Judge, H., Hoeppner, B. B., Cusin, C., Ionescu, D. F., Mathew, S. J., Chang, L. C., Iosifescu, D. V., et al. (2020) · Molecular Psychiatry · doi:10.1038/s41380-018-0256-5 · Cited by 433
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015) · Implementation Science · doi:10.1186/s13012-014-0192-y · Cited by 435
“These items were developed by the research team specifically for this study to assess these three pilot study outcomes [49].”
MethodResultAnderson, S. F., Kelley, K., & Maxwell, S. E. (2017) · Psychological Science · doi:10.1177/0956797617723724 · Cited by 406
Nuts and Bolts of Conducting Feasibility Studies
Tickle‐Degnen, L. (2013) · American Journal of Occupational Therapy · doi:10.5014/ajot.2013.006270 · Cited by 457
Albers, C. J., & Lakens, D. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.09.004 · Cited by 366
“Pilot studies can be performed because they have other uses, such as determining the feasibility of performing the designed study in practice, which is especially useful when trying out new methods or procedures (Leon et al., 2011).”
BackgroundLewis, M., Bromley, K., Sutton, C. J., McCray, G., Myers, H. L., & Lancaster, G. A. (2021) · Pilot and Feasibility Studies · doi:10.1186/s40814-021-00770-x · Cited by 330
White, S. W., Ollendick, T. H., Albano, A. M., Oswald, D. P., Johnson, C. R., Southam‐Gerow, M. A., Kim, I., & Scahill, L. D. (2012) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-012-1577-x · Cited by 330
“The most likely explanation is that the sample sizes in these trials were small and estimates of effect size are therefore unstable (Leon et al. 2011).”
BackgroundResultInfluentialRapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise
Riley, W. T., Glasgow, R. E., Etheredge, L., & Abernethy, A. P. (2013) · Clinical and Translational Medicine · doi:10.1186/2001-1326-2-10 · Cited by 267
“This belabored research process typically begins with pilot trials that we posit have limited benefit, are used inappropriately to estimate effect size [14], and often prematurely concretize a less than optimal intervention.”
BackgroundPilot and feasibility studies come of age!
Lancaster, G. A. (2015) · Pilot and Feasibility Studies · doi:10.1186/2055-5784-1-1 · Cited by 260
“Whether or not hypothesis testing is to be used, what is clear is that the main objective of a pilot or feasibility study is not to test treatment effectiveness [10,15,19].”
BackgroundResultGames for Health for Children—Current Status and Needed Research
Baranowski, T., Blumberg, F. C., Buday, R., DeSmet, A., Fiellin, L. E., Green, C. S., Kato, P. M., Lu, A. S., Maloney, A. E., Mellecker, R. R., et al. (2015) · Games for Health Journal · doi:10.1089/g4h.2015.0026 · Cited by 260
“…who should be involved, as well as types of involvement that enhance game effectiveness, the optimal role of feasibility studies in game research, 108–112 the definition of feasibility in game interventions (i.e., when is the game considered feasible), sample sizes needed for feasibility studies…”
Almirall, D., Compton, S. N., Gunlicks‐Stoessel, M., Duan, N., & Murphy, S. A. (2012) · Statistics in Medicine · doi:10.1002/sim.4512 · Cited by 223
“Scientists preparing to execute a randomized trial should also refer to the literature on executing external pilot studies for pointers on more general uses of pilot studies [34–36].”
Delaney, M. C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0207261 · Cited by 292
Whitehead, A. L., Sully, B., & Campbell, M. J. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.04.001 · Cited by 270
“[9] state that a pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods and implementation of the novel intervention.”
BackgroundAgile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
BackgroundMethodWoo, S. L., Kung, T. A., Brown, D. L., Leonard, J. A., Kelly, B. M., & Cederna, P. S. (2016) · Plastic and Reconstructive Surgery - Global Open · doi:10.1097/gox.0000000000001038 · Cited by 230
Pachankis, J. E., McConocha, E. M., Clark, K. A., Wang, K., Behari, K., Fetzner, B. K., Brisbin, C. D., Scheer, J. R., & Lehavot, K. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000508 · Cited by 227
Mendoza, J. A., Baker, K. S., Moreno, M. A., Whitlock, K. B., Abbey‐Lambertz, M., Waite, A., Colburn, T. A., & Chow, E. J. (2017) · Pediatric Blood & Cancer · doi:10.1002/pbc.26660 · Cited by 236
Pediatric Delirium and Associated Risk Factors
Silver, G. H., Traube, C., Gerber, L. M., Sun, X., Kearney, J. A., Patel, A. K., & Greenwald, B. M. (2015) · Pediatric Critical Care Medicine · doi:10.1097/pcc.0000000000000356 · Cited by 191
“However, it is important not to overconclude based on these findings (34).”
BackgroundClough, B. A., March, S., Chan, R. J., Casey, L. M., Phillips, R., & Ireland, M. J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0526-3 · Cited by 186
“The majority (67%, n = 10) of these studies had less than 20 participants per condition, which limits any interpretations of population efficacy [51].”
BackgroundPossemato, K., Kuhn, E., Johnson, E., Hoffman, J. E., Owen, J. E., Kanuri, N., De Stefano, L., & Brooks, E. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.09.005 · Cited by 167
Applying mixed methods to pilot feasibility studies to inform intervention trials
Aschbrenner, K. A., Kruse, G. R., Gallo, J. J., & Clark, V. L. P. (2022) · Pilot and Feasibility Studies · doi:10.1186/s40814-022-01178-x · Cited by 188
“The primary aim of pilot feasibility studies is to examine the feasibility of interventions and the methods used to evaluate them to answer the overarching question “Can it work?” prior to examining “Does it work?” [6].”
BackgroundMethodMoberg, C., Niles, A., & Beermann, D. (2019) · Journal of Medical Internet Research · doi:10.2196/12556 · Cited by 157
“Though the use of pilot studies for estimating effect sizes is questionable (cf [42]), this pilot study was the most relevant data available on which to base our power analysis.”
Powering population health research: Considerations for plausible and actionable effect sizes
Matthay, E. C., Hagan, E., Gottlieb, L. M., Tan, M. L., Vlahov, D., Adler, N. E., & Glymour, M. M. (2021) · SSM - Population Health · doi:10.1016/j.ssmph.2021.100789 · Cited by 199
“Likely effect sizes may be estimated based on pilot studies, theories of change, causal models, expert opinion, or scientific literature on similar interventions (Leon et al., 2011; Matthay, 2020; Thabane et al., 2010).”
BackgroundChristopher, M. S., Hunsinger, M., Goerling, L. R. J., Bowen, S., Rogers, B. S., Gross, C. R., Dapolonia, E., & Pruessner, J. C. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.03.059 · Cited by 173
“Although recent guidelines suggest preliminary e ffi cacy testing not be included in pilot studies (Leon et al., 2011), given the dearth of e ff ective interventions among LEOs (Patterson et al., 2014), we deemed it important to include these promising outcome data.”
Detecting pediatric delirium: development of a rapid observational assessment tool
Silver, G. H., Traube, C., Kearney, J. A., Kelly, D., Yoon, M. J., Moyal, W. N., Gangopadhyay, M., Shao, H., & Ward, M. J. (2012) · Intensive Care Medicine · doi:10.1007/s00134-012-2518-z · Cited by 177
“Although pilot studies, by their very nature, are imprecise [32], the 29% prevalence rate of delirium in our study group suggests that pediatric delirium is a common and under-recognized problem that requires the PICU community’s attention.”
BackgroundCollins, L. M., Kugler, K. C., & Gwadz, M. V. (2015) · AIDS and Behavior · doi:10.1007/s10461-015-1145-4 · Cited by 166
“[68]: ‘‘The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study.”
A Step-by-Step Process on Sample Size Determination for Medical Research
Bujang, M. A. (2021) · Malaysian Journal of Medical Sciences · doi:10.21315/mjms2021.28.2.2 · Cited by 177
Luby, J. L., Lenze, S. N., & Tillman, R. (2011) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2011.02483.x · Cited by 152
Monti, D., Zabrecky, G. P., Kremens, D. E., Liang, T., Wintering, N., Cai, J., Wei, X., Bazzan, A. J., Li, Z., Bowen, B., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157602 · Cited by 151
“Overall, this was a pilot study designed to generate hypothesis-testing data that could inform clinical trials adequately powered to assess safety and efficacy [61].”
MethodConner, C. M., White, S. W., Beck, K. B., Golt, J., Smith, I. C., & Mazefsky, C. A. (2019) · Autism · doi:10.1177/1362361318810709 · Cited by 150
“In accordance with guidelines regarding treatment development, the primary aim of this open trial was to gauge feasibility and acceptability of the new intervention (Leon et al., 2011).”
Kohrt, B. A., Jordans, M. J. D., Turner, E. L., Sikkema, K. J., Luitel, N. P., Rai, S., Singla, D. R., Lamichhane, J., Lund, C., & Patel, V. (2018) · Pilot and Feasibility Studies · doi:10.1186/s40814-018-0234-3 · Cited by 141
Beets, M. W., Weaver, R. G., Ioannidis, J. P. A., Geraci, M., Brazendale, K., Decker, L., Okely, A. D., Lubans, D. R., van Sluijs, E. M. F., Jago, R., et al. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-0918-y · Cited by 145
Keefe, R. S. E., Buchanan, R. W., Marder, S. R., Schooler, N. R., Dugar, A., Zivkov, M., & Stewart, M. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbr153 · Cited by 138
O'Mahen, H., Himle, J. A., Fedock, G., Henshaw, E., & Flynn, H. (2013) · Depression and Anxiety · doi:10.1002/da.22050 · Cited by 136
Lebares, C. C., Guvva, E. V., Olaru, M., Sugrue, L. P., Staffaroni, A. M., Delucchi, K. L., Kramer, J. H., Ascher, N. L., & Harris, H. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.4108 · Cited by 131
Update on the State of the Evidence for Stroke Family Caregiver and Dyad Interventions
Bakas, T., McCarthy, M. J., & Miller, E. (2017) · Stroke · doi:10.1161/strokeaha.117.016052 · Cited by 130
Should treatment effects be estimated in pilot and feasibility studies?
Sim, J. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0493-7 · Cited by 123
Sharp, W. G., Burrell, T. L., & Jaquess, D. L. (2013) · Autism · doi:10.1177/1362361313489190 · Cited by 136
“With this backdrop in place, the purpose of this article is to (a) describe a behaviorally based parent-training program—the Autism MEAL Plan—a curriculum specifically developed to assist caregivers to Manage Eating Aversions and Low intake among children with ASD; (b) evaluate the feasibility of the parent-training p…”
MethodConducting a pilot study: case study of a novice researcher
Doody, O., & Doody, C. M. (2015) · British Journal of Nursing · doi:10.12968/bjon.2015.24.21.1074 · Cited by 214
“…be made before the large-scale study is undertaken and answer methodological question(s), guide the development of the research plan to ensure the methods work in practice and assess the feasibility of the proposed research process (Hundley and van Teijlingen, 2002; Kim, 2011; Leon et al, 2011).”
InfluentialPerepletchikova, F., NATHANSON, D. L., Axelrod, S. R., Merrill, C. R. S., Walker, A., Grossman, M., Rebeta, J. L., Scahill, L. D., Kaufman, J., Flye, B. L., et al. (2017) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2017.07.789 · Cited by 120
“The 6 domains of feasibility included recruitment, randomization, retention, attendance, participants’ satisfaction, and therapist adherence.(22) The following benchmarks were set for DBT-C feasibility: at least 2 participants per month for recruitment; dropout rate lower than 30% for retention (defined as dropped out…”
BackgroundNaltrexone vs Placebo for the Treatment of Alcohol Dependence
Oslin, D. W., Leong, S. H., Lynch, K. G., Berrettini, W. H., O’Brien, C. P., Gordon, A. J., & Rukstalis, M. R. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.3053 · Cited by 121
Vulnerability to Depression in Youth: Advances From Affective Neuroscience
Kujawa, A., & Burkhouse, K. L. (2016) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2016.09.006 · Cited by 128
Rixon, L., Baron, J., McGale, N., Lorencatto, F., Francis, J., & Davies, A. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1904-6 · Cited by 117
“Design* RCT 16 (29.1) Quasi-experimental 3 (5.5) Cluster RCT 12 (38.2) Case study 1 (1.8) Pilot/feasibility 15 (27.3) Controlled 1 (1.8) Pre-post design 2 (3.6) Unclear 5 (9.1) Intervention recipients People with health conditions 17 (30.9) Employees/workers 5 (9.1) Family/informal carers 6 (10.9) Care home staff and…”
MethodInfluentialA Pilot Study: Vital Methodological Issues
Hazzi, O. A., & Maldaon, I. (2015) · Verslas teorija ir praktika · doi:10.3846/btp.2015.437 · Cited by 121
Wesson, J., Clemson, L. M., Brodaty, H., Lord, S. R., Taylor, M. E., Gitlin, L. N., & Close, J. (2013) · BMC Geriatrics · doi:10.1186/1471-2318-13-89 · Cited by 115
“Therefore, efficacy or intervention effects cannot be conclusively determined from our results, in keeping with the aims of a feasibility trial [32].”
ResultBehavioral activation for children and adolescents: a systematic review of progress and promise
Martin, F., & Oliver, T. (2019) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-018-1126-z · Cited by 111
“A detailed review is required not only in relation with the potential effectiveness of the intervention, but also for how the intervention can be delivered and implemented, which should be considered early in the intervention development process [28, 29].”
BackgroundPope, Z., Barr‐Anderson, D. J., Lewis, B. A., Pereira, M. A., & Gao, Z. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16193579 · Cited by 119
“Given the study’s pilot nature, our primary outcomes included intervention interest, use/acceptability, adherence, and retention, as recommended by the National Institutes of Health’s National Center for Complementary and Integrative Health (NCCIH) [36] and others [37] for the reporting of pilot trials.”
BackgroundMethodRice, E. R. W., Tulbert, E., Cederbaum, J. A., Adhikari, A. B., & Milburn, N. G. (2012) · Health Education Research · doi:10.1093/her/cyr113 · Cited by 107
“[50] suggest that pilot data should not be used to conduct hypothesis testing about outcomes but rather efforts should be focused on collecting data that can guide in the design and implementation of larger scale efficacy studies.”
BackgroundMethodMeasuring Treatment Fidelity in a Rehabilitation Intervention Study
Hildebrand, M. W., Host, H. H., Binder, E. F., Carpenter, B. D., Freedland, K. E., Morrow-Howell, N. L., Baum, C. M., Doré, P., & Lenze, E. J. (2012) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e31824ad462 · Cited by 114
Lee, S., & Ha‐Brookshire, J. (2017) · Sustainability · doi:10.3390/su9030465 · Cited by 123
“To test feasibility, equipment, and methods, an initial pilot study was conducted, which is a small-scale preliminary study of the larger research design [80].”
MethodDeal, J. A., Albert, M., Arnold, M. L., Bangdiwala, S. I., Chisolm, T. H., Thomas, S., Eddins, A. C., Glynn, N. W., Goman, A. M., Minotti, M., et al. (2017) · Alzheimer s & Dementia Translational Research & Clinical Interventions · doi:10.1016/j.trci.2017.06.003 · Cited by 106
“Given limitations of this pilot study, however, including small sample size, the magnitude of the effects reported here should not be interpreted as that would be estimated in a fully powered trial [21].”
BackgroundMcguire, J. F., Arnold, E. B., Park, J. M., Nadeau, J. M., Lewin, A. B., Murphy, T. K., & Storch, E. A. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.11.045 · Cited by 105
Fraser, J. H., Fahlman, D., Arscott, J., & Guillot, I. (2018) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v19i1.3326 · Cited by 146
“Nevertheless, as described above, properly conducted pilot studies can greatly strengthen outcomes of the main study regardless of the field (van Teijlingen & Hundley, 2002; Cohen, Manion, & Morrison, 2007; Gudmundsdottir & Brock-Utne, 2010; Leon et al., 2011; Doody & Doody, 2015).”
BackgroundResultInfluentialKooistra, L. C., Ruwaard, J., Wiersma, J. E., van Oppen, P., van der Vaart, R., van Gemert-Pijnen, J. E. W. C., & Riper, H. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.01.003 · Cited by 92
“In order to gain insight into the feasibility (Leon et al., 2011) of bCBT, we first examined the reach of the intervention in terms of the proportion of patients enrolled at the mental health care centre that was eligible for bCBT.”
BackgroundLearmonth, Y. C., Adamson, B., Kinnett-Hopkins, D., Bohri, M., & Motl, R. W. (2016) · Contemporary Clinical Trials · doi:10.1016/j.cct.2016.11.012 · Cited by 102
Systematic review of the evidence for stroke family caregiver and dyad interventions
Bakas, T., McCarthy, M. J., & Miller, E. L. (2022) · Stroke · doi:10.1161/strokeaha.121.034090 · Cited by 109
Neonatal pain: What′s age got to do with it?
Hatfield, L. (2014) · Surgical Neurology International · doi:10.4103/2152-7806.144630 · Cited by 102
“[29] During gestation, neuronal tissues acquire specific patterns of DNA methylation to regulate genome function.[96] Increased methylation inhibits gene expression, decreased methylation increases gene expression.”
BackgroundSharp, W. G., Stubbs, K. H., Adams, H., Wells, B. M., Lesack, R. S., Criado, K. K., Simon, E. L., McCracken, C. E., West, L. L., & Scahill, L. D. (2016) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000001043 · Cited by 102
Quintiliani, L. M., Mann, D. M., Puputti, M., Quinn, E. K., & Bowen, D. J. (2016) · JMIR Cancer · doi:10.2196/cancer.5305 · Cited by 100
“Although it is appropriate for sample sizes for pilot studies to be based on practical considerations based on recruitment and budgetary limitations [46], a larger sample size may have expanded our capacity to generate feasibility data.”
Wälti, P., Kool, J., & Luomajoki, H. A. (2015) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-015-0533-2 · Cited by 101
“The outcomes of this study have to be interpreted with caution because of the small sample size, as there is evidence that the risk of a type 1 error is larger in small sample sizes [39,41].”
Feasibility of "CopeSmart": A Telemental Health App for Adolescents
Kenny, R., Dooley, B., & Fitzgerald, A. (2015) · JMIR Mental Health · doi:10.2196/mental.4370 · Cited by 79
“Feasibility studies achieve this by exploring factors such as the acceptability of an intervention to participants, adherence and drop-out rates, and aspects of the intervention that may be altered to improve participants’ engagement [26], and are considered a crucial first step prior to the conduction of larger scale…”
Montag, C., Haase, L., Seidel, D., Bayerl, M., Gallinat, J., Herrmann, U., & Dannecker, K. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112348 · Cited by 97
“Results may therefore only give a very rough indication of the direction of the treatment effect but cannot serve to confirm or reject our hypotheses, and even results on feasibility might not generalise [78].”
Bjureberg, J., Sahlin, H., Hedman-Lagerlöf, E., Gratz, K. L., Tull, M. T., Jokinen, J., Hellner, C., & Ljótsson, B. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1885-6 · Cited by 96
Kiosses, D. N., Leon, A. C., & Areán, P. A. (2011) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2011.03.001 · Cited by 91
Self-citationRapaport, M. H., Schettler, P. J., & Bresee, C. J. (2012) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2011.0071 · Cited by 96
“Small sample sizes can lead to an overestimate of effect sizes and thus, the data must be considered within the context of this caveat.(31,34) A second concern in studies where one is attempting to measure a decrease in hormones normally associated with a stress response in unstressed, normal volunteers is that there…”
BackgroundKhan, M. N., Hamdani, S. U., Chiumento, A., Dawson, K., Bryant, R. A., Sijbrandij, M., Nazir, H., Akhtar, P., Masood, A., Wang, D., et al. (2019) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796017000336 · Cited by 82
“Since this was a feasibility trial, no power calculations were carried out (Leon et al. 2011).”
BackgroundFarrell, L. J., Waters, A. M., Boschen, M. J., Hattingh, L., McConnell, H. W., Oar, E. L., Collings, N., Zimmer‐Gembeck, M. J., Shelton, D., Ollendick, T. H., et al. (2013) · Depression and Anxiety · doi:10.1002/da.22132 · Cited by 98
“Furthermore, a pilot study of this nature is an essential step toward the advancement and refinement of new technologies and procedures.[46] Results of pilot studies, such as this are necessary to”
BackgroundSrinivasan, S. M., Eigsti, I., Gifford, T. D., & Bhat, A. N. (2016) · Research in Autism · doi:10.1016/j.rasd.2016.04.001 · Cited by 105
COGNITIVE-EMOTIONAL TRAINING AS AN INTERVENTION FOR MAJOR DEPRESSIVE DISORDER
Iacoviello, B. M., Wu, G., Alvarez, E., Huryk, K. M., Collins, K. A., Murrough, J. W., Iosifescu, D. V., & Charney, D. S. (2014) · Depression and Anxiety · doi:10.1002/da.22266 · Cited by 98
“Lastly, it has been shown that pilot studies with small sample sizes can yield inflated effect sizes (the confidence interval for the true effect size in a small pilot study can be very large and difficult to interpret[52]) and have demonstrated a history of not replicating the large effects that were initially observ…”
BackgroundOn the Unhappiness of Software Developers
Graziotin, D., Fagerholm, F., Wang, X., & Abrahamsson, P. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084242 · Cited by 105
“We discarded the pilots’ contact data and questionnaire data from the final survey as many guidelines recommend (e.g., [48]).”
MethodCorsini-Munt, S., Bergeron, S., Rosen, N. O., Mayrand, M.-H., & Delisle, I. (2014) · The Journal of Sexual Medicine · doi:10.1111/jsm.12646 · Cited by 87
Kohrt, B. A., Jordans, M. J. D., Turner, E. L., Rai, S., Gurung, D., Dhakal, M., Bhardwaj, A., Lamichhane, J., Singla, D. R., Lund, C., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.31475 · Cited by 90
Acceptance checklist for clinical effectiveness pilot trials: a systematic approach
Charlesworth, G., Burnell, K., Hoe, J., Orrell, M., & Russell, I. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-78 · Cited by 87
“For example: as a substitute for hypothesistesting studies rather than being preparation for a full trial; to make inadequately powered studies sound more attractive to publishers; and, to report on outcomes rather than design, methods and procedures [2,9,10].”
BackgroundHassan, N. M., Landorf, K. B., Shields, N., & Munteanu, S. E. (2018) · Journal of Intellectual Disability Research · doi:10.1111/jir.12562 · Cited by 96
Shomaker, L. B., Bruggink, S. M., Pivarunas, B., Skoranski, A. M., Foss, J., Chaffin, E., Dalager, S. L., Annameier, S. K., Quaglia, J. T., Brown, K. W., et al. (2017) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2017.04.003 · Cited by 95
Well-being and self-efficacy in a sample of undergraduate nurse students: A small survey study
Priesack, A., & Alcock, J. (2015) · Nurse Education Today · doi:10.1016/j.nedt.2015.01.022 · Cited by 91
“Despite thepositive results, it should be regarded as a pilot study (Leon et al., 2011) and the results considered with caution in relation to making generalisations.”
ResultBjureberg, J., Sahlin, H., Hellner, C., Hedman-Lagerlöf, E., Gratz, K. L., Bjärehed, J., Jokinen, J., Tull, M. T., & Ljótsson, B. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1527-4 · Cited by 84
“Following recommendations for early studies in clinical research [21, 22], the present pilot study examined the feasibility, acceptance, and utility of this ERITA in an open uncontrolled pilot study.”
MethodDevoe, D. J. A., Peterson, A. M., & Addington, J. (2017) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx139 · Cited by 82
Conner, C. M., & White, S. W. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-017-3312-0 · Cited by 78
“As the primary role of pilot studies is to determine the feasibility of the intervention (Leon et al. 2011), the aim of the current study was to analyze the feasibility and efficacy of a MABI targeting ER difficulties for adults with ASD.”
BackgroundA systematic review of interventions to provide genetics education for primary care
Paneque, M., Turchetti, D., Jackson, L. M., Lunt, P. W., Houwink, E. J. F., & Skirton, H. (2016) · BMC Family Practice · doi:10.1186/s12875-016-0483-2 · Cited by 77
“As the purpose of a pilot study [21] is to assess the”
BackgroundAbrantes, A. M., Blevins, C. E., Battle, C. L., Read, J. P., Gordon, A. L., & Stein, M. D. (2017) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2017.07.006 · Cited by 80
The significance filter, the winner's curse and the need to shrink
van Zwet, E. W., & Cator, E. (2021) · Statistica Neerlandica · doi:10.1111/stan.12241 · Cited by 84
“are often used to inform the sample size calculation of a larger trial [13],[7].”
MethodField, T. S., Dizonno, V., Almekhlafi, M., Bala, F., Alhabli, I., Wong, H., Norena, M., Villaluna, M. K., King-Azote, P., Ratnaweera, N., et al. (2023) · Stroke · doi:10.1161/strokeaha.123.044113 · Cited by 80
Orchowski, L. M., Barnett, N. P., Berkowitz, A. D., Borsari, B. E., Oesterle, D. W., & Zlotnick, C. (2018) · Violence Against Women · doi:10.1177/1077801218787928 · Cited by 76
Li, F., Harmer, P., Fitzgerald, K. M., & Winters‐Stone, K. M. (2022) · BMC Geriatrics · doi:10.1186/s12877-021-02747-0 · Cited by 78
Shimshoni, Y., Silverman, W. K., & Lebowitz, E. R. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23341 · Cited by 84
King, S. H., & Green, H. J. (2015) · Frontiers in Oncology · doi:10.3389/fonc.2015.00072 · Cited by 77
“Research has suggested that effect sizes reported in pilot studies often differ from consequent RCT studies implementing the same interventions (114).”
ResultSylvia, L., Salcedo, S., Bernstein, E. E., Baek, J., Nierenberg, A., & Deckersbach, T. (2013) · International Journal of Bipolar Disorders · doi:10.1186/2194-7511-1-24 · Cited by 74
“…be premature to conduct inferential statistics at this time and instead, for each measure, we calculated the mean, standard deviation, 95% confidence interval (CI), and then used Cohen’s d effect sizes for repeated measures to determine the magnitude of the changes we observed (Leon et al. 2011).”
Srinivasan, S. M., Eigsti, I., Neelly, L. P., & Bhat, A. N. (2016) · Research in Autism · doi:10.1016/j.rasd.2016.01.004 · Cited by 86
Murphy, T. K., Fernandez, T. V., Coffey, B. J., Rahman, O., Gavaletz, A., Hanks, C. E., Tillberg, C. S., Ibanez-Gomez, L., Sukhodolsky, D. G., Katsovich, L., et al. (2017) · Journal of Child and Adolescent Psychopharmacology · doi:10.1089/cap.2017.0024 · Cited by 78
“Pilot studies cannot be expected to prove efficacy (Leon et al. 2011).”
Metadata
- Authors
- Leon, A. C., Davis, L. L., & Kraemer, H. C.
- Affiliations
- Weill Cornell Medicine (US)
- University of Alabama (US)
- Stanford University (US)
- Corresponding author
- Leon, Andrew C.
- Venue
- Journal of Psychiatric Research
- Publisher
- Elsevier BV
- Published
- 2011-05
- Volume
- 45
- Issue
- 5
- Pages
- 626-629
- ISSN
- 0022-3956 (ISSN-L)
- PMC
- PMC3081994
- OpenAlex
- W2168434488
- Cited by
- 1,917
- References
- 17
- Citations by year
- 2026: 203 · 2025: 190 · 2024: 143 · 2023: 169 · 2022: 163 · 2021: 217 · 2020: 169 · 2019: 136 · 2018: 128 · 2017: 120 · 2016: 92 · 2015: 80 · 2014: 52 · 2013: 31 · 2012: 17
- FWCI
- 14.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Psychiatric Research (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (P30MH068638, R01MH060447)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- pilot study, feasibility assessment, clinical trials, research design, sample size determination, effect size estimation, generalizability, recruitment and retention, intervention implementation, research funding review
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Feasibility Studies, Humans, Pilot Projects, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
