The DOI matches its Crossref record; 44 of the 45 references it cites were found in a registry or an index; none of the 10 sources its claims cite is checked for support yet.
The role of formative evaluation in implementation research and the QUERI experience
Open accessOverview
AI snapshot
Written from the paper’s full text.
TL;DR
This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.
Key findings
- Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
- The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
- QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
- Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.
Methods
- Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
- Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
- Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.
Limitations
- Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
- Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
- Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own interpretation · 20 Own result · 8 Not checked yet · 7 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interview responses indicated that planning, including identification of resources and potential barriers and facilitators, was a critical and universally underutilized step in the intervention process.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Implementation-Focused Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the depression project, FE has generated information about factors influencing caseload capacity for nurse ''care managers'' who maintain regular contact with depressed patients to monitor treatment adherence, assess symptomatic improvement, provide patient education, and facilitate communication among patients, primary care clinicians, and mental health specialists.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: the weekly teleconferencing and problem resolutions changed the perceptions of clinical staff that this research was not related to day-to-day activities.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: the analysis identified barriers to successful implementation that related primarily to the intervention process and secondarily to characteristics of the intervention context.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The research team of CHF QUERI was able to negotiate a policy change, so access to a scale was no longer a barrier to self-monitoring fluid retention when patients were discharged.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Implementation-Focused Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Data from the interviews suggested tools and guidelines to improve planning and implementation skills.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One barrier to accomplishing the weight management goal was uncovered during weekly calls.The Prosthetic Department at 1 site did not interpret current policy in a manner that allowed provision of scales for CHF patients.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 13
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Formative evaluation allowed quick identification of stagnation of progress that suggested the need for increased facilitation or introduction of new tools to sustain enthusiasm for the intervention.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Seventeen prioritized short-and long-term recommendations were generated to improve the usefulness and usability of the 9 reminders.59”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
What the sources say
- Not checked yetcites [59] Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“IQS FE was also used to refine the intervention, to some extent, during the project, and to a major extent for the next ''nationwide rollout'' phase.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prototype toolkits for implementing QI and monitoring clinic practices and patient outcomes were distributed to 8 pilot clinics.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: while the FE provided some suggestions about how to modify the intervention and tools, the data were also useful to accurately ''tell the story'' of the individual clinics and to illustrate differences between clinics that were successful with implementation versus those that were not.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, we were able to see that lower performing IQS sites were slow to begin applying key concepts and had not completed an entire PDSA cycle well into an action period.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Value: i.The original toolkit was modified to improve its usefulness.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The calls also provided a model of problem-solving behaviors for the clinical staff (e.g.,Plan, Do, Study, Act) as well as information for the research team that enabled ''user friendly'' modifications of original project tools.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the project moved forward, it was clear that some sites were very successful in applying the treatment plan while others stagnated.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We identified 6 barriers to the effective use of the CR intervention, including workload, lack of time to follow-up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation provides techniques for obtaining such information and for overcoming limitations identified in early implementation/QI studies.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > THE VETERAN HEALTH ADMINISTRATION'S QUERI PROGRAM
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“As Stead et al. 10,11 suggest, traditional intervention research can fail to ''capture the detail and complexity of intervention inputs and tactics'' (10, p. 354), thereby missing the true nature of interventions as well as significant organizational factors important for replication.10,11”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > NEED FOR FE IN IMPLEMENTATION/QI RESEARCH
What the sources say
- Not checked yetcites Stead et al. 10,11
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementation researchers need to answer critical questions about the feasibility of implementation strategies, degree of real-time implementation, status and potential influence of contextual factors, response of project participants, and any adaptations necessary to achieve optimal change.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > THE VETERAN HEALTH ADMINISTRATION'S QUERI PROGRAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation can save time and frustration as data highlight factors that impede the ability of clinicians to implement best practices.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it can make the realities and black box nature of implementation more transparent to decision makers;”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interpretive FE uses the results of all other FE stages.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementation-focused formative data also are used to keep the strategies on track and as a result optimize the likelihood of affecting change by resolving actionable barriers, enhancing identified levers of change, and refining components of the implemen-tation interventions.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, FE data collected at this stage offer several advantages.They can (a) highlight actual versus planned interventions, (b) enable implementation through identification of modifiable barriers, (c) facilitate any needed refinements in the original implementation intervention, (d) enhance interpretation of project results, and (e) identify critical details and guidance necessary for re”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such developmental data enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Developmental Evaluation
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, interpretive information can be collected at the end of the project about key stakeholder experiences.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proactive nature of FE is emphasized, as progress data become feedback about the degree of movement toward desired outcomes.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, as indicated in some of the examples, various evaluative activities can serve multiple stages, which then merge in practice.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > FORMATIVE EVALUATION WITHIN QUERI
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation helps to meet the many challenges to effective implementation and its scientific study, thereby facilitating integration of research findings into practice and improvement of patient care.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation is a study approach that is often key to the success, interpretation, and replication of the results of implementation/QI projects.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In QUERI, this action focus differentiates FE from ''process'' evaluations where data are not intended for concurrent use.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > FORMATIVE EVALUATION WITHIN QUERI
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, FE data provide alternative explanations for results, help to clarify the meaning of success in implementation, and enhance understanding of an implementation strategy's impact or ''worth.''”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Implementation work tends to be disorderly and complicated.
“As implementation efforts can be a relatively messy and complex process”
The paper relies on this without support.
Factors identified in a general way are likely to differ from one implementation site to another.
“Even if available, a diagnostic analysis is suggested given the likelihood that generically identified factors will vary across implementation sites.”
The paper relies on this without support.
In well-controlled randomized controlled trials, researchers do not typically modify an experimental intervention once it has been approved.
“In the well-controlled RCT, researchers do not typically modify an experimental intervention once approved.”
The paper relies on this without support.
Formative evaluation is a new concept when applied to health services research.
“Formative evaluation is a new concept as applied to health services research and as such presents multiple challenges.”
The paper relies on this without support.
Formative evaluation is a time-consuming activity.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Some researchers need help understanding how formative evaluation can be incorporated into a study design.
“Some researchers may need help in understanding how FE can be incorporated into a study design.”
The paper relies on this without support.
Project leaders may need to be convinced of formative evaluation's utility before committing study resources.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Implementation and quality improvement researchers may encounter the erroneous view that formative evaluation involves only qualitative research or is not rigorous.
“Implementation and QI researchers may encounter the erroneous view that FE involves only qualitative research or that it is not rigorous, e.g., that it consists of ''just talking to a few people''.”
The paper relies on this without support.
Inferential statistical significance does not necessarily equate with clinical significance.
“However, as inferential statistical significance does not necessarily equate with clinical significance, it is useful to obtain perceptions of stakeholders relative to the ''meaning'' of statistical findings.”
The paper relies on this without support.
Research evidence indicates that people with negative attitudes about influenza vaccine safety, effectiveness, or utility are less likely to seek it annually.
“Implementation goal: research evidence indicates that persons who have negative attitudes about influenza vaccine (safety, effectiveness, utility) are less likely to seek it out annually.”
The paper relies on this without support.
Formative evaluation, as a descriptive assessment activity, does not by itself test hypotheses.
“Formative evaluation, as a descriptive assessment activity, does not per se test hypotheses.”
The paper relies on this without support.
Formative evaluation does not lack rigor.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
The paper relies on this without support.
Formative evaluation is not simply a matter of qualitative research or a specific qualitative methodology.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In addition, it has been suggested that current conceptions of research designs fail to “maximize clinical utility for practicing clinicians and other decision makers”18; for example, through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of “effective” practices.4,21,22”
From Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact (2012) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [14–16, 19, 20].”
From Guidance for conducting feasibility and pilot studies for implementation trials (2020) · cites this paper as [20]
In the citing paper
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
From The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework (2013) · cites this paper as [10]
In the citing paper
“Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [71]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“Mental health (MH) QUERI”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“NH, mental health”
Citations
Cites
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,639
Qualitative research & evaluation methods
Patton, M. Q. (2002) · Sage eBooks · Cited by 15,530
(1963) · Medical Entomology and Zoology · Cited by 15,085
(1979) · Medical Entomology and Zoology · Cited by 11,252
Gluesing, J. C. (2012) · Sage eBooks · doi:10.4135/9781412994231.n14 · Cited by 3,264
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836
Utilization-focused evaluation : the new century text
Patton, M. Q. (1997) · Cited by 1,469
Complex interventions: how “out of control” can a randomised controlled trial be?
Hawe, P., Shiell, A., & Riley, T. (2004) · BMJ · doi:10.1136/bmj.328.7455.1561 · Cited by 1,244
Evaluation: A Systematic Approach.
Patton, M. Q., Rossi, P. H., Freeman, H. E., & Wright, S. R. (1981) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2066947 · Cited by 933
Getting evidence into practice: the meaning of `context'
McCormack, B., Kitson, A., Harvey, G., Rycroft‐Malone, J., Titchen, A., & Seers, K. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02150.x · Cited by 765
Research works printed but not linked: 12
- Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
- Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.
- Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.
- Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.
- Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.
- Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
- Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.
- Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.
- Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
- LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.
- La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
- Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.
Cited by
Sorted by citations
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
“[3] as diagnostic analysis, implementationand progress-focused evaluations, and interpretive evaluation).”
BackgroundMethodCurran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058
Self-citationAn introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L., Hagedorn, H., Smith, J., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 1,611
“These studies can involve one or more of three broad types of evaluation, process evaluation, formative evaluation, and summative evaluation [13, 31].”
BackgroundSelf-citationProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894
“can address the fundamental issue of public health impact (Stetler et al. 2006).”
BackgroundImplementation science: What is it and why should I care?
Bauer, M. S., & Kirchner, J. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2019.04.025 · Cited by 1,167
Feldstein, A. C., & Glasgow, R. E. (2008) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(08)34030-6 · Cited by 987
“that begin with a barrier and needs assessment are most effective.(5,31) The Re-Fracture Intervention Trial (REFIT; Table 2) Phase 1 RCT revealed that patient-specific clinical guideline advice to the primary care provider (PCP)”
BackgroundKontos, P. C., & Poland, B. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-1 · Cited by 760
“It is an approach to implementation evaluation (also referred to as formative evaluation) [63] that, when combined with outcomes evaluation, creates a powerful 'hybrid style approach for implementation research' [63] which provides a clearer direction for action because the decision maker not only has knowledge of the…”
BackgroundTen recommendations for using implementation frameworks in research and practice
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Albers, B., Nilsen, P., Broder-Fingert, S., Mukasa, B., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00023-7 · Cited by 488
Guidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475
Stetler, C. B., Legro, M. W., Rycroft‐Malone, J., Bowman, C., Curran, G. M., Guihan, M., Hagedorn, H., Piñeros, S. L., & Wallace, C. M. (2006) · Implementation Science · doi:10.1186/1748-5908-1-23 · Cited by 429
“definitions within the implementation field, as well as a failure at times to differentiate planned versus actual operationalization of those interventions [ 11 ,12].”
BackgroundInfluentialSelf-citationA guiding framework and approach for implementation research in substance use disorders treatment.
Damschroder, L. J., & Hagedorn, H. J. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022284 · Cited by 427
Self-citationA Guide for applying a revised version of the PARIHS framework for implementation
Stetler, C. B., Damschroder, L. J., Helfrich, C. D., & Hagedorn, H. J. (2011) · Implementation Science · doi:10.1186/1748-5908-6-99 · Cited by 402
“• Receptivity to the targeted innovation/change • Develop and organize a retrospective interpretive evaluation [20] to explore the perceived influence of Contextual features on implementation of the targeted EBP.”
BackgroundInfluentialSelf-citationKoorts, H., Eakin, E., Estabrooks, P. A., Timperio, A. F., Salmon, J., & Bauman, A. E. (2018) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-018-0678-0 · Cited by 337
“Formative and process evaluation approaches [36, 37], including intervention testing methodologies [12, 31, 34], inform strategies to address potential barriers to effective implementation in Step 4.”
MethodImplementation of the CALM intervention for anxiety disorders: a qualitative study
Curran, G. M., Sullivan, G., Mendel, P. J., Craske, M. G., Sherbourne, C. D., Stein, M. B., McDaniel, A., & Roy‐Byrne, P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-14 · Cited by 352
“[28] refer to as “developmental formative evaluation”, that is, context-specific “diagnostic” evaluation of barriers and facilitators to implementation, which leverages the experiences and lessons learned in effectiveness trials to enhance the development of wider implementation and dissemination interventions.”
BackgroundRycroft-Malone, J., Seers, K., Chandler, J., Hawkes, C. A., Crichton, N., Allen, C., Bullock, I., & Strunin, L. (2013) · Implementation Science · doi:10.1186/1748-5908-8-28 · Cited by 303
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
BackgroundGetting a clinical innovation into practice: An introduction to implementation strategies
Kirchner, J. E., Smith, J. L., Powell, B. J., Waltz, T. J., & Proctor, E. K. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.06.042 · Cited by 280
“Formative evaluation (FE) is a rigorous assessment process typically involving collection of qualitative and quantitative data to identify the determinants of current practice, barriers, and facilitators for a practice change or implementation of a clinical innovation (Stetler et al., 2006).”
BackgroundInfluentialSelf-citationWoodward, E. N., Singh, R. S., Ndebele‐Ngwenya, P., Castillo, A. M., Dickson, K. S., & Kirchner, J. E. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00146-5 · Cited by 284
Cohen, D. J., Crabtree, B. F., Etz, R., Balasubramanian, B. A., Donahue, K. E., Leviton, L. C., Clark, E. C., Isaacson, N. F., Stange, K. C., & Green, L. W. (2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2008.08.005 · Cited by 248
Wierenga, D., Engbers, L. H., van Empelen, P., Duijts, S. F. A., Hildebrandt, V. H., & van Mechelen, W. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-1190 · Cited by 231
“Hence, effect evaluations should be accompanied by systematic and real-time process evaluations [20,21].”
BackgroundGreen, C. A., Duan, N., Gibbons, R. D., Hoagwood, K. E., Palinkas, L. A., & Wisdom, J. P. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0552-6 · Cited by 205
“Formative, process, and evaluative questions are all fair game (Stetler et al. 2006), and mixed methods designs capitalize on the strengths of each method used while attempting to reduce each method’s weaknesses.”
Metzler, C. W., Sanders, M. R., Rusby, J. C., & Crowley, R. N. (2011) · Behavior Therapy · doi:10.1016/j.beth.2011.05.004 · Cited by 197
“Gathering feedback from potential consumers is central to a formative development and evaluation process that seeks out and incorporates feedback from potential users and accounts for factors that will facilitate or interfere with program usage (Stetler et al., 2006).”
May, C. R., Mair, F. S., Dowrick, C. F., & Finch, T. L. (2007) · BMC Family Practice · doi:10.1186/1471-2296-8-42 · Cited by 183
“Similar methodological or procedural approaches are encountered with other frameworks that explore, for example, empirically identified facilitators and barriers to the successful implementation of complex interventions [10-12].”
Forman, J. H., Creswell, J. W., Damschroder, L. J., Kowalski, C. P., & Krein, S. L. (2008) · American Journal of Infection Control · doi:10.1016/j.ajic.2008.03.010 · Cited by 197
“This knowledge can help us design interventions, improve our understanding of how these interventions work and why they are effective or ineffective, and evaluate them and thereby better translate research into practice.(25) For example, in the area of hand hygiene, researchers can observe hospital staff to determine…”
BackgroundHawk, K. F., D’Onofrio, G., Chawarski, M. C., O’Connor, P. G., Cowan, E. A., Lyons, M. S., Richardson, L. D., Rothman, R. E., Whiteside, L., Owens, P. H., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.4561 · Cited by 182
Stetler, C. B., Mittman, B. S., & Francis, J. (2008) · Implementation Science · doi:10.1186/1748-5908-3-8 · Cited by 176
Self-citationDevelopment and application of the RE-AIM QuEST mixed methods framework for program evaluation
Forman, J. H., Heisler, M., Damschroder, L. J., Kaselitz, E. B., & Kerr, E. A. (2017) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2017.04.002 · Cited by 155
“Retrospective interpretive evaluation (Stetler et al., 2006).”
MethodHarnessing Implementation Science to Increase the Impact of Health Equity Research
Chinman, M., Woodward, E. N., Curran, G. M., & Hausmann, L. R. M. (2017) · Medical Care · doi:10.1097/mlr.0000000000000769 · Cited by 169
“The Hybrid Type I design is a traditional effectiveness trial of a clinical intervention accompanied by a formative evaluation that tracks various aspects of that clinical intervention’s implementation (eg, integrity, dose) which could affect its adoption.(39) As shown in Figure 1, a Hybrid Type I design may be approp…”
Elwy, A. R., Wasan, A. D., Gillman, A. G., Johnston, K. L., Dodds, N. E., McFarland, C. E., & Greco, C. M. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112532 · Cited by 154
Saint, S. K., Olmsted, R. N., Fakih, M. G., Kowalski, C. P., Watson, S. R., Sales, A. E., & Krein, S. L. (2009) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(09)35062-x · Cited by 146
Improving Community-Based Mental Health Care for Children: Translating Knowledge into Action
Garland, A. F., Haine‐Schlagel, R., Brookman-Frazee, L. I., Baker-Ericzén, M. J., Trask, E. V., & Fawley-King, K. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0450-8 · Cited by 154
“…models that identify effective methods to implement EB interventions into UC highlight the need for contextual data on UC services to inform implementation processes and intervention ‘‘fit’’ (Aarons et al. 2011; Damschroder et al. 2009; Mendel et al. 2008; Powell et al. 2012; Stetler et al. 2006).”
Stetler, C. B., McQueen, L., Demakis, J. G., & Mittman, B. S. (2008) · Implementation Science · doi:10.1186/1748-5908-3-30 · Cited by 146
“▪ Selection of implementation designs and tools that would address the complexities of producing credible output while accounting for the realities of a real-time, dynamic health care setting [3,31];”
InfluentialSelf-citationRitchie, M. J., Parker, L. E., Edlund, C. N., & Kirchner, J. E. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2217-0 · Cited by 112
“Facilitators used formative evaluation techniques [47] to assess clinics’ current practice patterns and anticipate barriers and facilitators to PC-MHI implementation.”
MethodThe role of organizational research in implementing evidence-based practice: QUERI Series
Yano, E. M. (2008) · Implementation Science · doi:10.1186/1748-5908-3-29 · Cited by 112
Designing monitoring arrangements for collaborative learning about adaptation pathways
Hermans, L. M., Haasnoot, M., ter Maat, J., & Kwakkel, J. H. (2016) · Environmental Science & Policy · doi:10.1016/j.envsci.2016.12.005 · Cited by 115
Training Research: Practical Recommendations for Maximum Impact
Beidas, R. S., Koerner, K., Weingardt, K. R., & Kendall, P. C. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0338-z · Cited by 115
“For example, formative evaluation, a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts (Stetler et al. 2006), is a further research tool to augment hybrid efficacy-effectiveness models.”
BackgroundA process for developing an implementation intervention: QUERI Series
Curran, G. M., Mukherjee, S., Allee, E., & Owen, R. R. (2008) · Implementation Science · doi:10.1186/1748-5908-3-17 · Cited by 110
Quantitative approaches for the evaluation of implementation research studies
Smith, J. D., & Hasan, M. M. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112521 · Cited by 146
Self-citationEvaluation of Pragmatic Telehealth Physical Therapy Implementation During the COVID-19 Pandemic
Miller, M. J., Pak, S. S., Keller, D. R., & Barnes, D. E. (2021) · Physical Therapy · doi:10.1093/ptj/pzaa193 · Cited by 106
“002d Non-Hispanic White 1639 (52) 826 (55) Black/African American 190 (6) 91 (6) Asian 666 (21) 240 (16) Hispanic/Latino 298 (9) 152 (10) Unknown 376 (12) 192 (13) Comorbidity burden <.”
Why do interventions work in some places and not others: A breastfeeding support group trial
Hoddinott, P., Britten, J., & Pill, R. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.10.067 · Cited by 103
Dietary Management of Heart Failure: DASH Diet and Precision Nutrition Perspectives
Wickman, B. E., Enkhmaa, B., Ridberg, R. A., Romero, E., Cadeiras, M., Meyers, F., & Steinberg, F. M. (2021) · Nutrients · doi:10.3390/nu13124424 · Cited by 120
“Implementation science approaches are used to translate research effectively and swiftly into patient care to support the inclusion of new knowledge into systems of care to optimize health care and public health [114,115].”
BackgroundConnell, L., McMahon, N. E., Harris, J. E., Watkins, C. L., & Eng, J. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0090-3 · Cited by 99
“Stetler et al. [11] have highlighted the role of formative evaluation in implementation research, defining it as ‘ a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts.”
Measuring persistence of implementation: QUERI Series
Initiative, T. H. Q. E. R., Bowman, C. C., Sobo, E. J., Asch, S. M., & Gifford, A. L. (2008) · Implementation Science · doi:10.1186/1748-5908-3-21 · Cited by 100
Self-citationChan, C. L., & Ackermann, B. J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00706 · Cited by 120
“…implementation stages of developing an occupation-specific prevention and management program may increase the likelihood of intervention outcomes being successful (Viera and Kumar, 2004; Stetler et al., 2006; Boocock et al., 2007; Jurg et al., 2008; Bell and Burnett, 2009; Harding et al., 2009).”
BackgroundMethodInfluentialHamilton, A. B., Brunner, J., Cain, C., Chuang, E., Luger, T. M., Canelo, I. A., Rubenstein, L. V., & Yano, E. M. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-017-0501-5 · Cited by 78
“velop “QI roadmaps”; (2) formative evaluation [28],”
Model for the evaluation of implementation programs and professional pharmacy services
Moullin, J. C., Sabater‐Hernández, D., & Benrimoj, S. I. (2015) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2015.08.003 · Cited by 88
Soi, C., Gimbel, S., Chilundo, B., Muchanga, V., Matsinhe, L., & Sherr, K. (2018) · Implementation Science · doi:10.1186/s13012-018-0846-2 · Cited by 74
“A post-implementation interpretive evaluation [25], which sought to explain implementation success or failure through the triangulation of project implementation stakeholders’ experiences, was carried out.”
MethodHarden, S. M., Johnson, S. B., Almeida, F. A., & Estabrooks, P. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0380-6 · Cited by 75
“However, there has been little research to determine if they result in practical interventions that are more likely to be adopted and implemented when compared to interventions developed through a pipeline model [20, 21].”
BackgroundFulop, N., Ramsay, A. I. G., Perry, C., Boaden, R., McKevitt, C., Rudd, A. G., Turner, S., Tyrrell, P. J., Wolfe, C. D. A., & Stephen Morris, A. (2015) · Implementation Science · doi:10.1186/s13012-016-0445-z · Cited by 71
“Understanding how evidence-based practice is implemented in complex settings such as healthcare is enhanced when its various components are considered (decision to change, intervention selection, planning and implementation of change, and outcomes) [2, 5, 6, 10].”
BackgroundBroyles, L. M., Rodriguez, K. L., Kraemer, K. L., Sevick, M. A., Price, P. A., & Gordon, A. J. (2012) · Addiction Science & Clinical Practice · doi:10.1186/1940-0640-7-7 · Cited by 73
“Consistent with these assumptions and previously identified barriers to the uptake and utilization of alcohol screening, BI, and RT [28,30,31], we developed our semi-structured interview guide to (1) assess current practices in alcohol-related care, and (2) prospectively identify the potential barriers and facilitator…”
MethodInformation Technology to Support Improved Care For Chronic Illness
Young, A. S., Chaney, E. F., Shoai, R., Bonner, L. M., Cohen, A. N., Doebbeling, B. N., Dorr, D. A., Goldstein, M. K., Kerr, E. A., Nichol, P., et al. (2007) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0303-4 · Cited by 74
“also go beyond it to evaluate the role of information systems in clinical and policy efforts to improve care.(39) In this project, the expert panel process resulted in the development of use cases for depression, schizophrenia, and diabetes with comorbidity.”
BackgroundLessons Learned about Implementing Research Evidence into Clinical Practice.
Hagedorn, H., Hogan, M. M., Smith, J. L., Bowman, C., Curran, G. M., Espadas, D., Kimmel, B., Kochevar, L. K., Legro, M. W., & Sales, A. E. (2006) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2006.00358.x · Cited by 76
Self-citationEvaluation of a Pilot Implementation to Integrate Alcohol-Related Care within Primary Care
Bobb, J. F., Lee, A., Lapham, G. T., Oliver, M. M., Ludman, E. J., Achtmeyer, C. E., Parrish, R. S., Caldeiro, R. M., Lozano, P., Richards, J. A., et al. (2017) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph14091030 · Cited by 69
“The two practice coaches interacted with staff in the three pilot PC sites regularly regarding elements of planned implementation that were challenging, and brought information on barriers and facilitators to a weekly implementation-focused FE meeting that included the principal investigator, a co-investigator, and a…”
BackgroundMazzucca, S., Tabak, R. G., Pilar, M. R., Ramsey, A. T., Baumann, A. A., Kryzer, E., Lewis, E. M., Padek, M. M., Powell, B. J., & Brownson, R. C. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00032 · Cited by 65
“org February 2018 | Volume 6 | Article 32 to the choice of design, and assessment of the designs currently being used in D&I research is needed so that future implementation efforts may better account for these differences as well as the contextual factors and multiple levels involved in this field of study (25, 26).”
BackgroundA synthesis of implementation science frameworks and application to global health gaps
Dintrans, P. V., Bossert, T. J., Sherry, J., & Kruk, M. E. (2019) · Global Health Research and Policy · doi:10.1186/s41256-019-0115-1 · Cited by 73
Poland, F. M., Spalding, N. J., Gregory, S., McCulloch, J., Sargen, K., & Vicary, P. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013498 · Cited by 68
Partnering with health system operations leadership to develop a controlled implementation trial
Bauer, M. S., Miller, C., Kim, B., Lew, R., Weaver, K. R., Coldwell, C. M., Henderson, K., Holmes, S. K., Seibert, M. N., Stolzmann, K. L., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0385-7 · Cited by 62
“For the interpretive formative evaluation [45], we will code interviews using our grounded thematic analysis [57] coding, paying particular attention to factors that might be barriers or facilitators to future implementation efforts [62].”
BackgroundMethodLuck, J., Hagigi, F., Parker, L. E., Yano, E. M., Rubenstein, L. V., & Kirchner, J. E. (2009) · Implementation Science · doi:10.1186/1748-5908-4-64 · Cited by 66
“Formative evaluation activities [21], such as a systematic program of stakeholder interviews and measurement of program penetration and utilization, and spread to additional sites are ongoing.”
Developing a national dissemination plan for collaborative care for depression: QUERI Series
Smith, J. L., Williams, J. W., Owen, R. R., Rubenstein, L. V., & Chaney, E. F. (2008) · Implementation Science · doi:10.1186/1748-5908-3-59 · Cited by 66
“Formative evaluation of DSG activities includes implementation-focused, progress-focused, and interpretive components [47].”
BackgroundMethodInfluentialSelf-citationSelf-compassion in sport: a scoping review
Cormier, D. L., Kowalski, K. C., Ferguson, L. J., Mosewich, A. D., McHugh, T.-L., & Röthlin, P. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2022.2161064 · Cited by 71
“We highly recommend that researchers follow best practices for intervention development and implementation, including formative evaluation (Stetler et al., 2006), process evaluation (Saunders et al., 2005), and adherence to the TiDieR checklist (Template for Intervention Description and…”
Evaluations of Implementation at Early-Adopting Lung Cancer Screening Programs
Gesthalter, Y. B., Koppelman, E. A., Bolton, R. E., Slatore, C. G., Yoon, S. H., Cain, H. C., Tanner, N. T., Au, D. H., Clark, J. A., & Wiener, R. S. (2017) · CHEST Journal · doi:10.1016/j.chest.2017.02.012 · Cited by 56
Jani, N., Mathur, S., Kahabuka, C., Makyao, N., & Pilgrim, N. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246717 · Cited by 59
“It has been established that formative evaluations that are designed to address anticipated facilitators and barriers can help policy makers and health care providers increase the possibility of successful implementation and uptake of interventions [26–28].”
BackgroundCampbell, C. I., Saxon, A. J., Boudreau, D. M., Wartko, P. D., Bobb, J. F., Lee, A. K., Matthews, A. G., McCormack, J., Liu, D. S., Addis, M., et al. (2021) · Addiction Science & Clinical Practice · doi:10.1186/s13722-021-00218-w · Cited by 59
Menec, V. H., Novek, S., VESELYUK, D. M., et al. (2013) · Journal of Aging & Social Policy · doi:10.1080/08959420.2014.854606 · Cited by 59
“Formative evaluation focuses on an assessment of the process of implementing a program (Dehar, Casswell, & Duignan, 1993; Schoster, Altpeter, Meier, & Callahan, 2012; Stetler et al., 2006).”
BackgroundMethodLeveraging implementation science to advance antibiotic stewardship practice and research
Livorsi, D. J., Drainoni, M., Reisinger, H. S., Nanda, N., McGregor, J. C., Barlam, T. F., Morris, A. M., & Szymczak, J. E. (2021) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2021.480 · Cited by 50
Richards, J. A., Bobb, J. F., Lee, A. K., Lapham, G. T., Williams, E. C., Glass, J. E., Ludman, E. J., Achtmeyer, C. E., Caldeiro, R. M., Oliver, M. M., et al. (2019) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2019.04.015 · Cited by 52
“other systems integrating drug screening may benefit from lessons learned during the pilot, we described barriers and facilitators identified in formative evaluation (Stetler et al., 2006) and adaptations made to the implementation strategies.”
BackgroundHagedorn, H., Kenny, M. E., Gordon, A. J., Ackland, P. E., Noorbaloochi, S., Yu, W., & Harris, A. H. S. (2018) · Addiction Science & Clinical Practice · doi:10.1186/s13722-018-0127-z · Cited by 54
Self-citationDefining and Addressing Gaps in Care for Obsessive-Compulsive Disorder in the United States
Senter, M. S., Patel, S. R., Dixon, L. B., Myers, R. W., & Simpson, H. B. (2021) · Psychiatric Services · doi:10.1176/appi.ps.202000296 · Cited by 54
The Effect of Care Pathways for Hip Fractures: A Systematic Review
Leigheb, F., Vanhaecht, K., Sermeus, W., Lodewijckx, C., Deneckere, S., Boonen, S., Boto, P., Mendes, R. V., & Panella, M. (2012) · Calcified Tissue International · doi:10.1007/s00223-012-9589-2 · Cited by 52
Brown, A. H., Cohen, A. N., Chinman, M. J., Kessler, C., & Young, A. S. (2008) · Implementation Science · doi:10.1186/1748-5908-3-9 · Cited by 53
“At the conclusion of the project, we will conduct semistructured interviews with the clinicians and clinic managers regarding the usefulness of the EQUIP-2 strategy, their satisfaction with the implementation process, barriers to and facilitators of implementation, and recommendations for future refinements [2].”
BackgroundMethodInfluentialRobben, S. H., Perry, M., Huisjes, M., van Nieuwenhuijzen, L., Schers, H., van Weel, C., Rikkert, M. G. M. O., van Achterberg, T., Heinen, M. M., & Melis, R. J. F. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-251 · Cited by 52
“First, because information concerning barriers and facilitators experienced during the implementation may be able to guide improvements to the implementation plan and the intervention itself [10,11].”
BackgroundApplication of participatory ergonomics to the redesign of the family-centred rounds process
Xie, A., Carayon, P., COX, E. D., Cartmill, R. S., Li, Y., Wetterneck, T. B., & Kelly, M. M. (2015) · Ergonomics · doi:10.1080/00140139.2015.1029534 · Cited by 53
“Finally, we performed a formative evaluation (Stetler et al. 2006) to assess and continuously improve the PE process (Andersen and Zebis 2014; Durlak and DuPre 2008; Griffin et al.”
MethodSobo, E. J., Bowman, C., & Gifford, A. L. (2008) · Social Science & Medicine · doi:10.1016/j.socscimed.2008.06.001 · Cited by 48
Self-citationDamschroder, L. J., Moin, T., Datta, S., Reardon, C. M., Steinle, N., Weinreb, J. E., Billington, C. J., Maciejewski, M. L., Yancy, W. S., Hughes, M. V., et al. (2015) · Implementation Science · doi:10.1186/s13012-015-0250-0 · Cited by 49
“This formative evaluation will incorporate the following features: 1) developmental evaluation to identify barriers and facilitators to implementation; 2) implementation and progress-focused evaluation to actively monitor implementation progress and adapt the program as needed; and 3) interpretive evaluation to provid…”
MethodPainter, P., Clark, L., & Olausson, J. M. (2014) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2014.01.433 · Cited by 50
Olmos‐Ochoa, T. T., Bharath, P., Ganz, D. A., et al. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-04967-y · Cited by 49
Mignogna, J., Martin, L. A., Harik, J., Hundt, N. E., Kauth, M., Naik, A. D., Sorocco, K., Benzer, J., & Cully, J. (2018) · Implementation Science · doi:10.1186/s13012-018-0768-z · Cited by 47
“Our work is limited in that we conducted a summative rather than formative evaluation [46] of providers’ perspectives on fidelity to the brief CBT intervention.”
MethodCompassionate End-of-Life Care: Mixed-Methods Multisite Evaluation of the 3 Wishes Project
Vanstone, M. G., Neville, T. H., Clarke, F. J., Swinton, M. E., Sadik, M., Takaoka, A., Smith, O. M., BAKER, A. J., LeBlanc, A., Foster, D., et al. (2019) · Annals of Internal Medicine · doi:10.7326/m19-2438 · Cited by 46
“Formative evaluation is a methodological technique used to assess development and implementation of a program early in its life cycle (6, 7).”
MethodMcNeely, J., Troxel, A. B., Kunins, H. V., Shelley, D., Lee, J. D., Walley, A. Y., Weinstein, Z., Billings, J., Davis, N. J., Marcello, R. K., et al. (2019) · Addiction Science & Clinical Practice · doi:10.1186/s13722-019-0135-7 · Cited by 42
Between and within-site variation in qualitative implementation research
Benzer, J. K., Beehler, S., Cramer, I. E., Mohr, D. C., Charns, M. P., & Burgess, J. F. (2013) · Implementation Science · doi:10.1186/1748-5908-8-4 · Cited by 49
“Formative evaluation was defined as an assessment of both potential and actual influences on implementation progress and effectiveness [33].”
BackgroundLukersmith, S., Fernández, A., Millington, M. J., & Salvador‐Carulla, L. (2015) · Disability and health journal · doi:10.1016/j.dhjo.2015.09.006 · Cited by 43
“A type of formative evaluation, the developmental evaluation, was undertaken to enhance the feasibility of the taxonomy.(46,47) Feasibility refers to the extent to which the use”
MethodEvaluating the Implementation of Home-Based Videoconferencing for Providing Mental Health Services
Interian, A., King, A. R., St. Hill, L. M., Robinson, C. H., & Damschroder, L. J. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700004 · Cited by 41
“Responding to this need, we conducted a formative evaluation of V2H implementation for mental health services (13).”
MethodChui, M. A., Stone, J. A., & Holden, R. J. (2017) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2016.11.006 · Cited by 43
Gören, J. L., Rose, A. J., Engle, R. L., Smith, E. G., Christopher, M. L. D., Rickles, N. M., Semla, T. P., & McCullough, M. B. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201500506 · Cited by 38
Dobalian, A., Bowman, C. C., Wyte‐Lake, T., Pearson, M. L., Dougherty, M. B., & Needleman, J. (2014) · BMC Nursing · doi:10.1186/s12912-014-0036-8 · Cited by 45
“This formative evaluation [16] was designed to qualitatively analyze data collected using in-person interviews and focus groups from 364 key informants involved in implementing these 15 partnerships.”
MethodSelf-citationIntegrating Opioid Use Disorder Treatment Into Primary Care Settings
Austin, E. J., Chen, J. A., Briggs, E. S., Ferro, L., Barry, P., Heald, A., Merrill, J. O., Curran, G. M., Saxon, A. J., Fortney, J. C., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.28627 · Cited by 43
Ragheb, J. W., Norcott, A. E., Benn, L., Shah, N., McKinney, A., Min, L. C., & Vlisides, P. E. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09681-4 · Cited by 51
“Study Design and Overview This was a pre-implementation diagnostic evaluation study [16], which sought to identify challenges, barriers, and facilitators to delirium screening and management at a major university hospital.”
MethodHenderson, C., Jackson, C. T., Slade, M., Young, A. S., & Strauss, J. L. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0264-5 · Cited by 41
“…however, this study indicates a package of strategies similar to those used in the VA Quality Enhancement Research Initiative (QUERI) program (Stetler et al. 2006), to include system level changes to facilitate MHAD completion and access; training to improve provider knowledge and attitudes;…”
MethodGoetz, M. B., Bowman, C., Hoang, T., Anaya, H. D., Osborn, T. M., Gifford, A. L., & Asch, S. M. (2008) · Implementation Science · doi:10.1186/1748-5908-3-16 · Cited by 43
“In addition, we are conducting formative evaluations to further refine our program and assess the organizational factors that determine the generalizability [38].”
InfluentialSelf-citationA Systematic Review of Nursing Home Palliative Care Interventions: Characteristics and Outcomes
Carpenter, J. G., Lam, K., Ritter, A. Z., & Ersek, M. T. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2019.11.015 · Cited by 46
Fidelity monitoring in complex interventions: a case study of the WISE intervention
Swindle, T., Selig, J. P., Rutledge, J. M., Whiteside-Mansell, L., & Curran, G. M. (2018) · Archives of Public Health · doi:10.1186/s13690-018-0292-2 · Cited by 40
Swindle, T., Johnson, S. L., Whiteside-Mansell, L., & Curran, G. M. (2017) · Implementation Science · doi:10.1186/s13012-017-0624-6 · Cited by 37
“Further, we will use formative evaluation principles to assess the feasibility, acceptability, and fidelity of the enhanced WISE strategy during real-time implementation [35].”
MethodSorensen, J. L., & Kosten, T. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022765 · Cited by 40
Taylor, N. J., Best, S., Martyn, M. M., Long, J. C., North, K. N., Braithwaite, J., & Gaff, C. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024681 · Cited by 37
Beehler, G. P., Murphy, J. L., King, P. R., Dollar, K. M., Kearney, L. K., Haslam, A., Wade, M., & Goldstein, W. R. (2019) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000747 · Cited by 39
Ersek, M. T., Hickman, S. E., Thomas, A., et al. (2017) · The Gerontologist · doi:10.1093/geront/gnx155 · Cited by 38
“This analytic approach was appropriate because our interview questions and coding schema followed our specific aims, which were derived from Stetler and colleagues’ framework for implementation-focused and interpretive formative evaluation (Stetler et al., 2006).”
BackgroundMethodInfluentialValentine, S. E., Borba, C. P. C., Dixon, L., Vaewsorn, A. S., Gallegos-Guajardo, J., Resick, P. A., Stirman, S. W., & Marques, L. M. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22337 · Cited by 41
“In addition, the study uses formative evaluation (Stetler et al., 2006) to assess potential and actual influences on the success of our process of implementation of culturally adapted CPT.”
MethodHagedorn, H., Stetler, C. B., Bangerter, A. K., Noorbaloochi, S., Stitzer, M. L., & Kivlahan, D. R. (2014) · Addiction Science & Clinical Practice · doi:10.1186/1940-0640-9-12 · Cited by 35
“In contrast to process evaluation, as defined in this study as part of a Hybrid I design, formative evaluation [9] is used in a Hybrid II or III design to gather information for monitoring and actively enhancing implementation during the course of the implementationfocused aspect of the study.”
BackgroundMethodSelf-citationImplementation of the Tobacco Tactics Program in the Department of Veterans Affairs
Veterans Integrated Services Network (VISN) 11 Tobacco Tactics Team, Duffy, S. A., Karvonen-Gutierrez, C. A., Ewing, L. A., & Smith, P. M. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-009-1075-9 · Cited by 38
Care coordination for pregnant veterans: VA’s Maternity Care Coordinator Telephone Care Program
Cordasco, K. M., Katzburg, J. R., Katon, J. G., Zephyrin, L., Chrystal, J. G., & Yano, E. M. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx081 · Cited by 38
HIV patient and provider feedback on a telehealth collaborative care for depression intervention
Drummond, K. L., Painter, J. T., Curran, G. M., Stanley, R., Gifford, A. L., Rodriguez‐Barradas, M. C., Rimland, D., Monson, T. P., & Pyne, J. M. (2016) · AIDS Care · doi:10.1080/09540121.2016.1255704 · Cited by 33
“FE is a process of formally identifying potential or actual influences upon implementation (Stetler et al., 2006).”
Background
Metadata
- Authors
- Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
- Affiliations
- University of Washington (US)
- VA Puget Sound Health Care System (US)
- VA San Diego Healthcare System (US)
- Edward Hines, Jr. VA Hospital (US)
- Minneapolis VA Medical Center (US)
- Baylor College of Medicine (US)
- Central Arkansas Veterans Healthcare System (US)
- Corresponding author
- Stetler, Cheryl B.
- ORCID
- Venue
- Journal of General Internal Medicine
- Published
- 2006-02
- Volume
- 21
- Issue
- S2
- Pages
- S1-S8
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC2557128
- OpenAlex
- W2084378034
- Cited by
- 642
- References
- 45
- Citations by year
- 2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
- FWCI
- 24.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
- MeSH terms
- Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
