The DOI matches its Crossref record; 16 of the 17 references it cites were found in a registry or an index; none of the 5 sources its claims cite is checked for support yet.
The role and interpretation of pilot studies in clinical research
Open accessAbstract
Pilot studies represent a fundamental phase of the research process. The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design. A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study. Investigators should be forthright in stating these objectives of a pilot study. Grant reviewers and other stakeholders should expect no more.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
40 Own interpretation · 5 No basis given · 2 No full text of the sources · 1 Sources not found
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“When event rates are reported and no adverse event is observed for a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Tolerability and Preliminary Safety
What the sources say
- Not matchedcites Jovanovic & Levy, 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jovanovic et al., 1997
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Basing a decision to proceed or terminate evaluation of a particular intervention on pilot data is perilous because there is a very good chance that the decision will be derived from false positive or false negative results.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each aspect of feasibility could be quite different from an uncontrolled study when intervention assignment is randomized and blinded, particularly if placebo is a distinct possibility.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Rationale for a Control or Comparison Group in a Pilot
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With these issues in mind, a pilot study can better address its goals if a control group is part of the design.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Rationale for a Control or Comparison Group in a Pilot
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A case series estimate tends to be steeped in optimism, particularly if an estimate of such magnitude is seldom, if ever, seen in full scale trials for psychiatric disorders.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(It is the population effect size, not the sample effect size, that an RCT is designed to detect with sufficient power.)”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“There has been a venerable tradition of using pilot studies to estimate between group effect sizes that, in turn, are used to inform the design of subsequent larger scale hypothesis testing studies. Despite it widespread use, it has been argued that the tradition is ill-founded. (Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006)”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
What the sources say
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot sample size is instead based on the pragmatics of recruitment and the necessities for examining feasibility.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With no inferential statistical tests, a pilot study will not provide p-values.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot study data generally should not be combined with data from the subsequent larger scale study. This is because it is quite likely that the methods will be modified after the pilot, even if minimally. Such changes in protocol risk adding additional, perhaps unknown, source of variation.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Are Pilot Data Included in the Larger Trial?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Methods will likely be modified after a pilot study, even if minimally.
- It may state two separate claims as one. 13
- Probably not a substantial claim. 13
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study.”
- Also stated in: § DISCUSSION · plain
“Pilot results inform feasibility, which in turn, is instructive in that it points to modifications needed in the planning and design of a larger efficacy trial.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DISCUSSION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The definition of clinically meaningful is not entirely empirically-based, but instead requires input from clinicians who treat the patient population of interest and perhaps from patients with the disorder.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These good clinical practices include the refinement of source documentation, informed consent procedures, data collection tools, regulatory reporting procedures, and monitoring/oversight procedures, especially when multiple sites and investigators are engaged in the study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Safety, efficacy and effectiveness are not evaluated in a pilot.”
- Also stated in: § DISCUSSION · plain
“a pilot study is not a hypothesis testing study and therefore safety and efficacy are not evaluated.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DISCUSSION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pilot studies generate pilot data, their design need not be guided with the support of prior pilot data.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is possible, but highly unlikely, that the between group effect size (d) from a pilot study sample will provide a reasonable estimate of the population effect size (Δ), but that cannot be known based on the pilot data.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot study can be critical in research staff training and provide experiences that strengthen and confirm competencies and skills required for the investigation to be conducted with accuracy and precision.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“An alternative approach is to base sample size estimates for a full scale RCT on what is deemed a clinically meaningful effect.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > An Alternative to Using Pilot Data for Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A pilot study provides opportunity to develop consistent practices to enhance data integrity and the protection of human subjects.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY > Good Clinical Practices
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We learn little if anything about the efficacy of an intervention with data from a small sample; yet, as discussed earlier, a great deal can be learned from a pilot study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Hypothetical Example
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This applies to effects sizes for any type of outcome, be it a severity rating, a response status, or survival status.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, with imprecision comes a vast disparity in sample size estimates and, if sample size determination for a subsequent study is based on an imprecise estimate, there is an enormous risk of underpowered or overpowered design.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Hypothetical Example
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, pilot studies are a necessary first step in exploring novel interventions and novel applications of interventions – whether in a new patient population or with a novel delivery system (e.g., transdermal patch).”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although a pilot study will undoubtedly incorporate relevant outcome measures and can serve a vital role in treatment development, it is not, and should not, be considered a preliminary test of the intervention hypothesis.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The primary role of a pilot study is to examine the feasibility of a research endeavor.”
- Also stated in: § INTRODUCTION · plain
“The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This does not preclude the need for a theoretical rationale for the intervention or the methodology being proposed for a pilot study.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a pilot study effect size is unduly large (i.e., a false positive result), subsequent trials will be designed with an inadequate number of participants to provide the statistical power needed to detect clinically meaningful effects and that would lead to negative trials.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, pilot studies provide an opportunity to implement and examine the feasibility of the adverse event reporting system.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Tolerability and Preliminary Safety
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is quite reasonable and expected that a pilot study is proposed with no pilot or other preliminary data supporting the proposal and that its proposed sample size is based on pragmatics such as patient flow and budgetary constraints.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY > Pilot Data for a Pilot Study
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.”
- Also stated in: § INTRODUCTION · plain
“evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and/or implementation of the novel intervention.”
- Also stated in: § WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY · plain
“A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, and implementation of the novel intervention and each of these can be quantified”
§ The Role and Interpretation of Pilot Studies in Clinical Research > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are two fundamental reasons that hypothesis testing is not used in a pilot study: the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a consequence, pilot studies can reduce the proportion of failed trials and allow research funds to be spent on projects for which feasibility has been demonstrated and quantified.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Well-conceived and implemented pilot studies will reduce the risk of several problems that are commonly faced in clinical trials. These include the inability to recruit the proposed sample size and a corresponding reduction in statistical power, excessive attrition due to intolerable procedures or interventions, and the need to modify a protocol midway through a trial.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimates become more precise with larger sample sizes.”
§ The Role and Interpretation of Pilot Studies in Clinical Research > EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO > Pilot Study Effect Sizes and Sample Size Determination
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
The true population effect size is unknown when a pilot study is designed.
“An essential challenge of therapeutic development is that the true population effect size is unknown at the time a pilot study is designed.”
The paper relies on this without support.
Methods will likely be modified after a pilot study, even if minimally.
“This is because it is quite likely that the methods will be modified after the pilot, even if minimally.”
The paper relies on this without support.
Pilot results presented to grant review committees tend to be selective, overly optimistic, and at times misrepresentational.
“Furthermore, pilot results that are presented to grant review committees tend to be selective, overly optimistic and, at times, misrepresentational.”
The paper relies on this without support.
Many pilot studies have far fewer than 18 participants per group.
“Many pilot studies have far fewer than 18 participants/group and therefore even greater imprecision.”
The paper relies on this without support.
It is not unusual for research grant applications and pilot study publications to convey optimism based on pilot data.
“Nevertheless, it is not unusual for research grant applications and pilot study publications to convey such optimism, particularly when based on pilot data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The planned sample size was 30 allowing for attrition. This study will inform the sample size for any future RCT (Leon et al., 2011).”
From A mixed methods pilot and feasibility open trial of internet-delivered cognitive behaviour therapy (iCanADAPT Advanced) for people with advanced cancer with depression and/or anxiety (2021) · cites this paper as Leon et al., 2011
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This was a pilot study which like other pilot studies of MI did not have a power calculation. 31,32”
From A Pilot Randomized Controlled Trial to Assess the Impact of Motivational Interviewing on Initiating Behavioral Therapy for Migraine (2020) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Data analyses in the context of a pilot study are not hypothesis-driven but serve to provide information regarding feasibility and acceptability of the intervention [55].”
From Acceptance and Commitment Therapy Wellness Program for Latine Adults Who Smoke and Have Psychological Distress: Protocol for a Feasibility Study (2023) · cites this paper as [55]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A common misuse of pilot and feasibility studies has been the application of hypothesis testing for clinical outcomes in small under-powered studies.”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Hypothesis testing in pilot studies, particularly in the context of effectiveness/efficacy of clinical outcomes, has been widely criticised due to the improper purpose and lack of statistical power of such evaluations [2, 20, 21, 23].”
From Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back! (2021) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Since the focus was on feasibility and acceptability of the intervention and we did no hypothesis testing, no power calculation was required (Leon et al., 2011).”
From PandaMom – Feasibility and acceptability of an internet- and mobile-based intervention to enhance peripartum mental well-being and to prevent postpartum depression (2024) · cites this paper as Leon et al., 2011
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our use of effect sizes was, however, deemed appropriate for this pilot trial (55, 63).”
From EMPOWER: A Multi-Site Pilot Trial to Reduce Distress in Surrogate Decision-Makers in the ICU (2024) · cites this paper as [63]
Experimental — accuracy not yet measuredIn the citing paper
“Previous work suggests early studies often test feasibility of methods (e.g., recruitment, randomization, retention, and assessment), not hypotheses [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
In the citing paper
“In terms of statistics, a Bayesian approach might be more appropriate than Fisherian statistics [56] as the Fisher approach is not appropriate with limited data [48].”
From Agile science: creating useful products for behavior change in the real world (2016) · cites this paper as [48]
Citations
Cites
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015) · doi:10.1007/978-3-319-18539-2 · Cited by 2,237
Porta, M. (2026) · doi:10.1093/acref/9780197663639.001.0001 · Cited by 2,175
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998) · doi:10.1007/978-1-4757-2915-3 · Cited by 873
. Caution regarding the use of pilot studies to guide power calculations for study proposals.
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.5.484 · Cited by 702 · cited 1×
“(Friedman, Furberg and DeMets, 1998; Kraemer et al., 2006) Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples.”
Self-citationWolfstein (1905) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-190502000-00009 · Cited by 346
Attrition in randomized controlled clinical trials: methodological issues in psychopharmacology
Leon, A. C., Mallinckrodt, C. H., Chuang-Stein, C., Archibald, D. G., Archer, G. E., & Chartier, K. (2006) · Biological Psychiatry · doi:10.1016/j.biopsych.2005.10.020 · Cited by 303 · cited 1×
“(Leon et al., 2006; Leon & Davis, 2009).”
Self-citationSixteen S ‐squared over D ‐squared: A relation for crude sample size estimates
Lehr, R. (1992) · Statistics in Medicine · doi:10.1002/sim.4780110811 · Cited by 253 · cited 1×
“(N/group ≈ 16/d2; Lehr, 1992) The clinical interpretation of HAMD change of 1.2 to 3.0 would drive the choice among possible sample sizes in planning a study.”
BackgroundInfluentialJovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.1080/00031305.1997.10473947 · Cited by 179 · cited 1×
“…a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n. (Jovanovic & Levy, 1997; Jovanovic et al., 1997) For example, consider a study with N=15 receiving medication and no suicidal ideation was reported…”
BackgroundHarvey, I. M. (2009) · Public Health · doi:10.1016/j.puhe.2009.05.009 · Cited by 173
Jovanovic, B. D., & Levy, P. S. (1997) · The American Statistician · doi:10.2307/2685405 · Cited by 50
Cited by
Sorted by citations
Items 101–200 of 1,903
Propofol for Treatment-Resistant Depression: A Pilot Study
Mickey, B. J., White, A. T., Arp, A. M., Leonardi, K., Torres, M. M., Larson, A. L., Odell, D. H., Whittingham, S. A., Beck, M. M., Jessop, J. E., et al. (2018) · International Journal of Neuropsychopharmacology · doi:10.1093/ijnp/pyy085 · Cited by 73
“This pilot study was not designed to test the hypothesis that propofol has antidepressant effects, so we report descriptive statistics only, as recommended (Leon et al., 2011).”
BackgroundPugliese, C. E., & White, S. W. (2013) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-013-1914-8 · Cited by 88
“Consistent with recommendations from clinical trial methodologists (e.g., Leon et al. 2011) and experts in the field of ASD treatment research (Smith 2007) to establish feasibility of a new intervention prior to determining efficacy, the primary aim of this project was to develop an adapted PST…”
MethodMartinez, B., Coyote, E., Hall‐Clifford, R., Juarez, M., Miller, A. C., Francis, A., Valderrama, C. E., Stroux, L., Clifford, G. D., & Rohloff, P. J. (2018) · Reproductive Health · doi:10.1186/s12978-018-0554-z · Cited by 74
A Guideline for Randomized Controlled Trials of Acupuncture
Chen, H., Yang, M., Ning, Z., Lam, W. L., Zhao, Y. K., Yeung, W., Ng, B. F. L., Ziea, E. T.-C., & Lao, L. (2019) · The American Journal of Chinese Medicine · doi:10.1142/s0192415x19500010 · Cited by 73
Efstathiou, N., Walker, W. M., Metcalfe, A., & Vanderspank‐Wright, B. (2018) · Journal of Critical Care · doi:10.1016/j.jcrc.2018.11.026 · Cited by 78
Aschbrenner, K. A., Naslund, J. A., Shevenell, M. D., Mueser, K. T., & Bartels, S. J. (2015) · Psychiatric Quarterly · doi:10.1007/s11126-015-9395-x · Cited by 79
“First, the sample was small, involving only volunteers that were self-selected, and the study was implemented with an uncontrolled design consistent with the intent of a pilot feasibility study [49].”
Schmitz, J. M., Green, C. E., Hasan, K. M., Vincent, J. N., Suchting, R., Weaver, M. F., Moeller, F. G., Narayana, P. A., Cunningham, K. A., Dineley, K. T., et al. (2017) · Addiction · doi:10.1111/add.13868 · Cited by 72
“[28], sample size for this pilot trial was based primarily on the pragmatics of recruitment and the necessities for examining feasibility.”
Bannon, S., Lester, E. G., Gates, M. V., McCurley, J., Lin, A., Rosand, J., & Vranceanu, A.-M. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00615-z · Cited by 70
Dew, M. A., DiMartini, A. F., Dabbs, A. D., Zuckoff, A., Tan, H. P., McNulty, M. L., Switzer, G. E., Fox, K. R., Greenhouse, J. B., & Humar, A. (2013) · American Journal of Transplantation · doi:10.1111/ajt.12393 · Cited by 69
“However, as per RCT reporting guidelines (65), we note that postdonation assessments were not conducted unless subjects underwent donation (see Figure 2) because it is not reasonable to ask subjects about outcomes from a procedure they did not undergo.”
BackgroundResultInfluentialKim, T.-H., Kong, S., Park, J., et al. (2018) · Journal of the Korean Gastric Cancer Association · doi:10.5230/jgc.2018.18.e19 · Cited by 69
“First, this was an investigational study evaluating the feasibility of the NIR-ICG technique in various partial gastrectomy settings [12].”
BackgroundAllin, L. J., Brolinson, P. G., Beach, B. M., Kim, S., Nussbaum, M. A., Roberto, K. A., & Madigan, M. L. (2020) · BMC Geriatrics · doi:10.1186/s12877-020-01605-9 · Cited by 66
Devlin, H., Ashley, M., Williams, T., et al. (2021) · BDJ · doi:10.1038/s41415-021-3526-6 · Cited by 68
Willging, C. E., Green, A. E., Gunderson, L., Chaffin, M., & Aarons, G. A. (2015) · Child Maltreatment · doi:10.1177/1077559514547384 · Cited by 66
Sajid, S., Dale, W., Mustian, K. M., Kotwal, A., Heckler, C. E., Porto, M., Fung, C., & Mohile, S. G. (2016) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2016.02.002 · Cited by 66
Trinh, L., Arbour‐Nicitopoulos, K. P., Sabiston, C. M., Berry, S. R., Loblaw, A., Alibhai, S. M. H., Jones, J. M., & Faulkner, G. (2018) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-018-0686-0 · Cited by 67
“An a priori sample size calculation was not conducted due to the feasibility nature of this study [22].”
BackgroundBell, A. N., Moss, D., & Kallmeyer, R. J. (2019) · NeuroRegulation · doi:10.15540/nr.6.2.54 · Cited by 65
Making ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 67
Kujawa, A., Kessel, E. M., Carroll, A., Arfer, K. B., & Klein, D. N. (2017) · Biological Psychology · doi:10.1016/j.biopsycho.2017.07.001 · Cited by 64
Lyon, A. R., Whitaker, K. M., Locke, J. J., Cook, C. R., King, K. M., Duong, M. T., Davis, C. J., Weist, M. D., Ehrhart, M. G., & Aarons, G. A. (2018) · Implementation Science · doi:10.1186/s13012-018-0721-1 · Cited by 66
“therefore potentially less susceptible to issues associated with calculating effect sizes in small samples [93, 94].”
BackgroundAschbrenner, K. A., Carpenter-Song, E. A., Mueser, K. T., Kinney, A., Pratt, S. I., & Bartels, S. J. (2012) · Community Mental Health Journal · doi:10.1007/s10597-012-9552-8 · Cited by 64
“However, the overall sample size (n = 30) is consistent with both qualitative methods (Miles and Huberman 1994) and the intent and scope of a pilot study aimed at determining feasibility and achieving preliminary descriptive findings (Leon et al. 2011).”
Ward, L., Stebbings, S. M., Athens, J., Cherkin, D. C., & Baxter, D. G. (2017) · Musculoskeletal Care · doi:10.1002/msc.1201 · Cited by 67
Carolan, S., Harris, P. R., Greenwood, K., & Cavanagh, K. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.08.001 · Cited by 72
“This pilot study has enabled us to make a more confident but still tentative prediction of effect size for our primary outcome of engagement, we recognise however the limitations of using this effect size to determine sample size for a full trial (Leon et al., 2011).”
MethodMuehlschlegel, S., Goostrey, K. J., Flahive, J., Zhang, Q., Pach, J. J., & Hwang, D. Y. (2022) · Neurology · doi:10.1212/wnl.0000000000200937 · Cited by 60
“Sample Size Calculation and Statistical Analyses Per recommendations for sample size calculations for pilot studies, we based our sample size of 40 patients and their surrogates on the necessities for examining feasibility.(25,26) We analyzed our primary feasibility outcomes using standard descriptive statistics.”
MethodFeasibility study design and methods for Project GEMS: Guidelines for Exercise in Multiple Sclerosis
Adamson, B., Learmonth, Y. C., Kinnett-Hopkins, D., Bohri, M., & Motl, R. W. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.12.002 · Cited by 64
“The clarity and strength of the proposed studywill come from the focus upon the elements of feasibility and secondarily upon a focus on the results of the RCT [12,22–25].”
BackgroundBergersen, K., Halvorsen, J. Ø., Tryti, E. A., Taylor, S. I., & Olsen, A. (2017) · Brain Injury · doi:10.1080/02699052.2016.1255779 · Cited by 62
“Next, pilot studies should be conducted in order to provide information on the feasibility and acceptability of the study design to be used in subsequent large-scale randomized controlled trials [49,50].”
MethodVranceanu, A., Jacobs, C. A., Lin, A. C., Greenberg, J., Funes, C. J., Harris, M. B., Heng, M. M., Macklin, E. A., & Ring, D. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0416-7 · Cited by 66
Uebelacker, L. A., Battle, C. L., Sutton, K. A., Magee, S. R., & Miller, I. W. (2015) · Archives of Women s Mental Health · doi:10.1007/s00737-015-0571-7 · Cited by 62
“Consistent with recommendations for aims of pilot studies (Leon et al. 2011), we examined intervention feasibility, safety, and adherence, and change in depression severity.”
MethodSchneider, K. L., Panza, E. A., Handschin, B., Ma, Y., Busch, A. M., Waring, M. E., Appelhans, B. M., Whited, M. C., Keeney, J., Kern, D., et al. (2015) · Behavior Therapy · doi:10.1016/j.beth.2015.10.005 · Cited by 64
Yardley, M. M., & Ray, L. A. (2016) · Addiction Biology · doi:10.1111/adb.12349 · Cited by 62
“…and extremely valuable opportunity to assess the feasibility of an approach prior to initiating larger scale clinical trials such as identifying a specific population more likely to respond to the medication and issues concerning retention, analyses, assessments, and so on (Leon et al., 2011).”
BackgroundmHealth Improved Fruit and Vegetable Accessibility and Intake in Young Children
Bakırcı-Taylor, A. L., Reed, D. B., McCool, B., & Dawson, J. (2019) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2018.11.008 · Cited by 62
Minen, M. T., Adhikari, S., Padikkala, J., Tasneem, S., Bagheri, A., Goldberg, E., Powers, S. W., & Lipton, R. B. (2020) · Headache The Journal of Head and Face Pain · doi:10.1111/head.14010 · Cited by 62
“4 (3-69), 15 [12-26], n = 137 Average number of Headache days/month 10.”
Keiser, J., Silué, K. D., Adiossan, L. G., et al. (2014) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0002975 · Cited by 60
“Thus, the choice of an adequate sample size for a pilot study is mainly based on practical considerations of the pilot trial rather than on statistical sample size calculations [25].”
BackgroundWhite, S. W., Richey, J. A., Gracanin, D., Coffman, M., Elias, R., Laconte, S., & Ollendick, T. H. (2016) · Education and Training in Autism and Developmental Disabilities · doi:10.1177/215416471605100308 · Cited by 59
Lee, S., & Ha‐Brookshire, J. (2018) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.1510 · Cited by 59
Healy, S., & Marchand, G. (2019) · International Journal of Disability Development and Education · doi:10.1080/1034912x.2019.1597968 · Cited by 60
Coughlin, L. N., Nahum-Shani, I. B., Philyaw-Kotov, M. L., Bonar, E. E., Rabbi, M., Klasnja, P., Murphy, S. A., & Walton, M. A. (2021) · JMIR mhealth and uhealth · doi:10.2196/24424 · Cited by 62
“Next, consistent with the best practices for pilot studies [76], we report (1) the association between baseline characteristics and self-reported app engagement based on zero-order correlations and a linear regression model and (2) change in preliminary outcomes from baseline to follow-up and the…”
BackgroundMethodNurses' practice in preventing postoperative wound infections: an observational study
Ding, S., Lin, F. F., Marshall, A. P., & Gillespie, B. M. (2017) · Journal of Wound Care · doi:10.12968/jowc.2017.26.1.28 · Cited by 60
MYmind: a Concurrent Group-Based Mindfulness Intervention for Youth with Autism and Their Parents
Salem-Guirgis, S., Albaum, C. S., Modica, P. T., Riosa, P. B., Nicholas, D. B., Drmic, I., & Weiss, J. A. (2019) · Mindfulness · doi:10.1007/s12671-019-01107-9 · Cited by 70
Greenberg, J., Popok, P., Lin, A. C., Kulich, R. J., James, P., Macklin, E. A., Millstein, R. A., Edwards, R. R., & Vranceanu, A. (2020) · JMIR Formative Research · doi:10.2196/18703 · Cited by 59
“As this is not an efficacy trial, we limited analyses to effect sizes for within-group changes between pretest and posttest and refrained from between-group comparisons [61].”
MethodMbuagbaw, L., Kosa, S. D., Lawson, D. O., Stalteri, R., Olaiya, O. R., Alotaibi, A., & Thabane, L. (2019) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0500-z · Cited by 57
Consolidated guidance for behavioral intervention pilot and feasibility studies
Pfledderer, C. D., von Klinggraeff, L., Burkart, S. A., Bandeira, A. D. S., Lubans, D. R., Jago, R., Okely, A. D., van Sluijs, E. M. F., Ioannidis, J. P. A., Thrasher, J. F., et al. (2024) · Pilot and Feasibility Studies · doi:10.1186/s40814-024-01485-5 · Cited by 69
“Definition Preliminary impact is the ability of an intervention, during a PFS to produce a desired or intended result [194].”
BackgroundA brief peer gatekeeper suicide prevention training: Results of an open pilot trial
Rallis, B. A., Esposito-Smythers, C. L., Disabato, D. J., Mehlenbeck, R., Kaplan, S. A., Geer, L., Adams, R., & Meehan, B. T. (2018) · Journal of Clinical Psychology · doi:10.1002/jclp.22590 · Cited by 55
Web‐based LGBT cultural competency training intervention for oncologists: Pilot study results
Seay, J., Hicks, A., Markham, M. J., et al. (2019) · Cancer · doi:10.1002/cncr.32491 · Cited by 56
“We also note there was variability in pre-/posttraining differences in measure scores, and the effect size of COLORS will need to be determined in a future larger trial, as effect size cannot be accurately estimated from pilot data.(46)”
BackgroundHirt, J., Ballhausen, N., Hering, A., Kliegel, M., Beer, T., & Meyer, G. (2020) · Journal of Alzheimer s Disease · doi:10.3233/jad-200347 · Cited by 63
“Rather, they intend to test an intervention and its effect(s) in order to plan the main randomized controlled trial with suffficient sample size and welldetermined outcomes [59].”
BackgroundCallisaya, M. L., Daly, R. M., Sharman, J. E., Bruce, D. G., Davis, T. M. E., Greenaway, T. M., Nolan, M. T., Beare, R. J., Schultz, M. G., Phan, T. G., et al. (2017) · BMC Geriatrics · doi:10.1186/s12877-017-0635-9 · Cited by 54
“However, 95% confidence interval were provided to give information about the magnitude and direction of an effect [23, 41].”
BackgroundMethodMaharjan, S. M., Poudyal, A., van Heerden, A. C., Byanjankar, P., Thapa, A., Islam, C., Kohrt, B. A., & Hagaman, A. K. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01473-2 · Cited by 56
“A sample size calculation was not conducted because this is a pilot study and the recruitment was done based on feasibility of using these devices by the participants [37, 38].”
MethodAminoff, V., Sellén, M., Sörliden, E., Ludvigsson, M., Berg, M., & Andersson, G. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.684540 · Cited by 54
Jansen-Kosterink, S. M., Veld, R. M. H. A. H. I. ’., Schönauer, C., Kaufmann, H., Hermens, H. J., & Vollenbroek-Hutten, M. M. R. (2013) · Games for Health Journal · doi:10.1089/g4h.2013.0043 · Cited by 58
“36 The framework for telemedicine evaluation proposed by DeChant et al. 17 was used, and in line with this framework, the presented pilot study was a stage 1–2 evaluation.”
Greenberg, J., Lin, A. C., Zale, E. L., Kulich, R. J., James, P., Millstein, R. A., Shapiro, H., Schatman, M. E., Edwards, R. R., & Vranceanu, A. (2019) · Journal of Pain Research · doi:10.2147/jpr.s222448 · Cited by 55
“Here, we report on a 2-phased study informed by the National Institute of Health (NIH) stage model for behavioral intervention development,(25) National Institute of Complementary and Integrative Health (NCCIH) model for developing and testing mind-body intervention,(26) and recent recommendations for early feasibilit…”
MethodCzyz, E. K., King, C. A., Prouty, D., Micol, V. J., Walton, M., & Nahum‐Shani, I. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13383 · Cited by 54
Brown, J. S. L., Blackshaw, E., Stahl, D., Fennelly, L., McKeague, L., Sclare, I., & Michelson, D. (2019) · Journal of Adolescence · doi:10.1016/j.adolescence.2018.11.009 · Cited by 52
Developing a cognitive behavioral therapy for hypersomnia using telehealth: a feasibility study
Ong, J. C., Dawson, S. C., Mundt, J. M., & Moore, C. (2020) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.8750 · Cited by 52
“Based on recommendations for pilot studies, 38 we focused on evaluating the feasibility and acceptability of the intervention and telehealth model for delivery and data collection to inform the design of a subsequent ef fi cacy study.”
Possemato, K., Johnson, E. M., Emery, J. B., Wade, M. J., Acosta, M. C., Marsch, L. A., Rosenblum, A., & Maisto, S. A. (2018) · Psychiatric Rehabilitation Journal · doi:10.1037/prj0000334 · Cited by 51
“Feasibility was assessed with several metrics recommended by Leon, Davis, and Kraemer (2011) using descriptive statistics.”
MethodConceptual Introduction to the Multiphase Optimization Strategy (MOST)
Collins, L. M. (2018) · Statistics for Social and Behavioral Sciences · doi:10.1007/978-3-319-72206-1_1 · Cited by 51
Voss, A. T., Jorgensen, M. K., & Murphy, J. G. (2021) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000451 · Cited by 51
Lenze, E. J., Farber, N. B., Kharasch, E. D., Schweiger, J. A., Yingling, M. D., Olney, J. W., & Newcomer, J. W. (2016) · The World Journal of Biological Psychiatry · doi:10.3109/15622975.2016.1142607 · Cited by 50
“We demonstrated feasibility in terms of adequate recruitment and retention of participants and adherence to the protocol (Leon et al. 2011), as well as good safety and tolerability for both ketamine and clonidine.”
Shomaker, L. B., Berman, Z., Burke, M., Annameier, S. K., Pivarunas, B., Sánchez, N., Smith, A. D., Hendrich, S. G., Riggs, N. R., Legget, K. T., et al. (2019) · Appetite · doi:10.1016/j.appet.2019.05.022 · Cited by 54
Narzisi, K., & Simons, J. (2020) · Journal of Child Health Care · doi:10.1177/1367493520917863 · Cited by 49
“The majority of the studies were undertaken in the United States (17) and Australia (6), with 1 each in the United Kingdom, Belgium, The Netherlands, Sweden, Canada, New Zealand and China.”
BackgroundMethodInfluentialPowell, B. J., Haley, A. D., Patel, S. V., Amaya-Jackson, L. M., Glienke, B., Blythe, M., Lengnick‐Hall, R., McCrary, S., Beidas, R. S., Lewis, C. C., et al. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00009-5 · Cited by 52
“larger implementation effectiveness trial [102, 103].”
Burnette, C. B., & Mazzeo, S. E. (2020) · International Journal of Eating Disorders · doi:10.1002/eat.23319 · Cited by 53
Mendoza, J. A., Watson, K., Chen, T. A., Baranowski, T., Nicklas, T. A., Uscanga, D. K., & Hanfling, M. J. (2012) · Health & Place · doi:10.1016/j.healthplace.2011.07.004 · Cited by 53
Kaur, N., Figueiredo, S. M., Bouchard, V., Moriello, C., & Mayo, N. E. (2017) · Clinical Rehabilitation · doi:10.1177/0269215517692129 · Cited by 57
Creatine as a Novel Treatment for Depression in Females Using Methamphetamine: A Pilot Study
Hellem, T. L., Sung, Y.-H., Shi, X.-F., Pett, M. A., Latendresse, G., Morgan, J., Huber, R. S., Kuykendall, D., Lundberg, K. J., & Renshaw, P. F. (2015) · Journal of Dual Diagnosis · doi:10.1080/15504263.2015.1100471 · Cited by 49
Digital Cognitive Behavioral Therapy for Insomnia in Women With Chronic Migraines
Crawford, M., Luik, A. I., Espie, C. A., Taylor, H. L., Burgess, H. J., Jones, A. L., & Ong, J. C. (2020) · Headache The Journal of Head and Face Pain · doi:10.1111/head.13777 · Cited by 51
Lived experience of infertility among Hong Kong Chinese women
Tiu, M. M. H., Hong, J. Y., Cheng, V. S., Kam, C. Y., & Ng, B. T. Y. (2018) · International Journal of Qualitative Studies on Health and Well-Being · doi:10.1080/17482631.2018.1554023 · Cited by 49
the Treat-CCM Investigators, Lanfranconi, S., Scola, E., Bertani, G., Zarino, B., Pallini, R., d’Alessandris, G., Mazzon, E., Marino, S., Carriero, M. R., et al. (2020) · Trials · doi:10.1186/s13063-020-4202-x · Cited by 47
Ewais, T., Begun, J., Kenny, M. A., Hay, K. E., Houldin, E., Chuang, K.-H., Tefay, M., & Kisely, S. R. (2021) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2021.110594 · Cited by 48
Replication initiatives will not salvage the trustworthiness of psychology
Coyne, J. C. (2016) · BMC Psychology · doi:10.1186/s40359-016-0134-3 · Cited by 46
“We need to consider the likelihood that original studies are often underpowered, and so we should discourage the common practice of relying on what are essentially pilot studies for generating estimated effect sizes, rather than simply demonstrating feasibility or acceptability [23].”
BackgroundTexting to Increase Adolescent Physical Activity: Feasibility Assessment
Thompson, D. I., Cantu, D., Ramirez, B., Cullen, K. W., Baranowski, T., Mendoza, J. A., Anderson, B. J., Jago, R., Rodgers, W. M., & Liu, Y. (2016) · American Journal of Health Behavior · doi:10.5993/ajhb.40.4.9 · Cited by 50
Sosnowski, K., Mitchell, M. L., White, H. T., Morrison, L., Sutton, J., Sharratt, J., & Lin, F. F. (2018) · Pilot and Feasibility Studies · doi:10.1186/s40814-017-0224-x · Cited by 54
“This is due to the imprecision inherent in data from small sample sizes [46].”
BackgroundDeJong, H. M., Abbott, S., Zelesco, M., Kennedy, B. F., Ziman, M. R., & Wood, F. M. (2017) · PubMed Central · Cited by 49
Strauss, C., Lea, L., Hayward, M., Forrester, E., Leeuwerik, T., Bibby-Jones, A.-M., & Rosten, C. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.04.007 · Cited by 50
Lang, A. J., Malaktaris, A. L., Casmar, P. V., Baca, S. A., Golshan, S., Harrison, T., & Negi, L. T. (2019) · Journal of Traumatic Stress · doi:10.1002/jts.22397 · Cited by 54
Lock, J., Couturier, J. L., Matheson, B. E., Datta, N., Citron, K. E., Sami, S., Welch, H. A., Webb, C., Doxtdator, K., & Carson, N. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23611 · Cited by 47
Sundberg, K. K., Langius‐Eklöf, A., Blomberg, K., Isaksson, A., & Wengström, Y. (2015) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2015.02.013 · Cited by 48
Bradizza, C. M., Stasiewicz, P. R., Zhuo, Y., Ruszczyk, M., Maisto, S. A., Lucke, J. F., Brandon, T. H., Eiden, R. D., Slosman, K. S., & Giarratano, P. (2017) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntw398 · Cited by 44
Drug development challenges and strategies to address emerging and resistant fungal pathogens
McCarthy, M. W., & Walsh, T. J. (2017) · Expert Review of Anti-infective Therapy · doi:10.1080/14787210.2017.1328279 · Cited by 46
“ential statistics but still demonstrates efficacy and safety (trials of this nature typically have wide confidence intervals) [55].”
BackgroundNkyekyer, J., Meyer, D., Pipingas, A., et al. (2019) · Clinical Interventions in Aging · doi:10.2147/cia.s183905 · Cited by 46
“Given the limitations of this study, including the small sample size, the magnitude of the effects reported here should not be interpreted as would be the case for a fully powered trial.(36) Our baseline results have provided the motivation needed to proceed with a full-scale, randomized hearing loss intervention and…”
Asante, E., Bam, V., Diji, A. K.-A., Lomotey, A. Y., Owusu Boateng, A., Sarfo-Kantanka, O., Oparebea Ansah, E., & Adjei, D. (2020) · The Diabetes Educator · doi:10.1177/0145721720954070 · Cited by 46
“The objective of the feasibility assessment was to evaluate study process measures such as participant recruitment, retention, program participation, and satisfaction.(25) The study authors targeted feasibility as achieved when at least 80% of the intervention participants completed at least 60% (≥10 calls) of the nur…”
A systematic review of school-based positive psychology interventions to foster teacher wellbeing
Vo, D. T., & Allen, K. (2022) · Teachers and Teaching · doi:10.1080/13540602.2022.2137138 · Cited by 64
Prenatal vitamin D and enamel hypoplasia in human primary maxillary central incisors: A pilot study
Reed, S. G., Voronca, D., Wingate, J. S., et al. (2016) · Pediatric Dental Journal · doi:10.1016/j.pdj.2016.08.001 · Cited by 45
“Though results of maternal vitamin D exposure and child dental outcome were not the primary aim of that study (33), we did find that the presence of EH in the primary maxillary central incisors of the 2-5 year olds of this pilot study tended to correspond with lower maternal serum circulating 25(OH)D concentrations du…”
BackgroundMethodResultInfluentialA step-by-step guide on preregistration and effective data sharing for psychopathology research.
Krypotos, A., Klugkist, I. G., Mertens, G., & Engelhard, I. M. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000424 · Cited by 43
The coping long term with active suicide program: Description and pilot data
Miller, I. W., Gaudiano, B. A., & Weinstock, L. M. (2016) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12247 · Cited by 44
Harrison, A., Al-Khairulla, H., & Kikoler, M. (2016) · The Journal of Positive Psychology · doi:10.1080/17439760.2015.1117125 · Cited by 12
“Thus, this longitudinal pilot study aims to describe the development and implementation of a five-session positive psychological intervention group for use in an adolescent ED inpatient setting and explore its feasibility and acceptability based on guidance from Leon, Davis, and Kraemer (2011).”
BackgroundMethodInfluentialAntonio, M. G., Williamson, A. K., Kameswaran, V., Bhogal, A. N.-B., Ankrah, E. A., Goulet, S., Wang, Y., Macias, G., James-Gist, J., Brown, L. K., et al. (2022) · Journal of Medical Internet Research · doi:10.2196/42586 · Cited by 44
“Pilot studies are exploratory to inform the development of larger-scale randomized studies and do not strive for a large sample size that can be used for inferential statistics [41].”
BackgroundMethodInfluentialMeyns, P., Pans, L., Plasmans, K., Heyrman, L., Desloovere, K., & Molenaers, G. (2017) · Games for Health Journal · doi:10.1089/g4h.2016.0069 · Cited by 44
“No inferential statistical tests (providing P values) should be used in pilot studies.(25) Results Participants Twelve participants were included.”
Flickinger, T., DeBolt, C. L., Xie, A., Kosmacki, A., Waselewski, M. E., Waldman, A. L., Reynolds, G., Conaway, M. R., Cohn, W. F., Ingersoll, K. S., et al. (2018) · AIDS and Behavior · doi:10.1007/s10461-018-2174-6 · Cited by 44
“Pilot studies can provide preliminary information on feasibility and acceptability but should not be used to draw conclusions for hypothesis testing or effectiveness [73, 74].”
BackgroundChiumento, A., Mukherjee, I., Chandna, J., Dutton, C., Rahman, A., & Bristow, K. (2018) · BMC Public Health · doi:10.1186/s12889-018-5661-9 · Cited by 54
Ghahramanlou‐Holloway, M., LaCroix, J. M., Perera, K. U., Neely, L. L., Grammer, G. G., Weaver, J. J., Novak, L. A., Colborn, V. A., & Lee‐Tauler, S. Y. (2018) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2018.11.006 · Cited by 44
Lakens, D., & Albers, C. J. (2017) · doi:10.31234/osf.io/b7z4q · Cited by 48
Feasibility of Automated Training for Facial Emotion Expression and Recognition in Autism
White, S. W., Abbott, L., Wieckowski, A. T., Capriola-Hall, N. N., Aly, S. F., & Youssef, A. E. (2017) · Behavior Therapy · doi:10.1016/j.beth.2017.12.010 · Cited by 43
Goals-of-care decision aid for critically ill patients with TBI
Muehlschlegel, S., Hwang, D. Y., Flahive, J., Quinn, T. V., Lee, C., Moskowitz, J., Goostrey, K. J., Jones, K., Pach, J. J., Knies, A. K., et al. (2020) · Neurology · doi:10.1212/wnl.0000000000009770 · Cited by 43
Stewart, A. L., Nápoles, A. M., Piawah, S., Santoyo‐Olsson, J., & Teresi, J. A. (2020) · Ethnicity & Disease · doi:10.18865/ed.30.s2.745 · Cited by 41
Sleep Well! An adapted behavioral sleep intervention implemented in urban primary care
Williamson, A. A., Okoroji, C., Cicalese, O., Evans, B. C., Ayala, A., Harvey, B. W., Honore, R., Kratchman, A. L., Beidas, R. S., Fiks, A. G., et al. (2021) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.9822 · Cited by 42
“The sample size was based on prior feasibility study research 32,33 and guidelines for qualitative data saturation (12 – 15 interviews).”
Sosnoff, J. J., Moon, Y., Wajda, D. A., Finlayson, M. L., McAuley, E., Peterson, E. W., Morrison, S., & Motl, R. W. (2014) · Clinical Rehabilitation · doi:10.1177/0269215514564899 · Cited by 42
The importance of pilot studies, how to write them and what they mean
Vogel, S. K., & Draper‐Rodi, J. (2017) · International journal of osteopathic medicine · doi:10.1016/j.ijosm.2017.02.001 · Cited by 56
Individualized Repetitive Transcranial Magnetic Stimulation Treatment in Chronic Tinnitus?
Kreuzer, P. M., Poeppl, T. B., Rupprecht, R., Vielsmeier, V., Lehner, A., Langguth, B., & Schecklmann, M. (2017) · Frontiers in Neurology · doi:10.3389/fneur.2017.00126 · Cited by 42
“Due to the pilot character of the study, we focus on the feasibility and identification of modifications for future studies (62, 63).”
MethodMerlin, J. S., Westfall, A. O., Long, D. M., Davies, S. L., Saag, M. S., Demonte, W., Young, S., Kerns, R. D., Bair, M. J., Kertesz, S. G., et al. (2018) · AIDS and Behavior · doi:10.1007/s10461-018-2028-2 · Cited by 43
“The goal total sample size was 40, with 20 participants per arm, a sample size generally sufficient to investigate feasibility/acceptability [22].”
Overcoming the “Valley of Death” in Medications Development for Alcohol Use Disorder
Ray, L. A., Bujarski, S., Roche, D. J. O., & Magill, M. (2018) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.13829 · Cited by 42
“Recent discussion about this issue has suggested that estimating effect sizes from pilot studies is unsupported from a methodological viewpoint (Kraemer et al., 2006; Leon et al., 2011) and that stable estimation of effect size requires hundreds of participants to be enrolled (Sch€ onbrodt and Perugini, 2013).”
Beani, E., Menici, V., Ferrari, A., Cioni, G., & Sgandurra, G. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.00016 · Cited by 43
“To test both these feasibility aspects, in addition to the ad hoc questionnaire and data about the wearing time of Actigraphs, a series of previously set and validated measures (21) based on relevant literature (19, 20) have been used.”
Method
Metadata
- Authors
- Leon, A. C., Davis, L. L., & Kraemer, H. C.
- Affiliations
- Weill Cornell Medicine (US)
- University of Alabama (US)
- Stanford University (US)
- Corresponding author
- Leon, Andrew C.
- Venue
- Journal of Psychiatric Research
- Publisher
- Elsevier BV
- Published
- 2011-05
- Volume
- 45
- Issue
- 5
- Pages
- 626-629
- ISSN
- 0022-3956 (ISSN-L)
- PMC
- PMC3081994
- OpenAlex
- W2168434488
- Cited by
- 1,917
- References
- 17
- Citations by year
- 2026: 203 · 2025: 190 · 2024: 143 · 2023: 169 · 2022: 163 · 2021: 217 · 2020: 169 · 2019: 136 · 2018: 128 · 2017: 120 · 2016: 92 · 2015: 80 · 2014: 52 · 2013: 31 · 2012: 17
- FWCI
- 14.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Psychiatric Research (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Mental Health (P30MH068638, R01MH060447)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- pilot study, feasibility assessment, clinical trials, research design, sample size determination, effect size estimation, generalizability, recruitment and retention, intervention implementation, research funding review
- MeSH terms
- Clinical Trials as Topic, Data Interpretation, Statistical, Feasibility Studies, Humans, Pilot Projects, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
