The DOI matches its Crossref record; 44 of the 45 references it cites were found in a registry or an index; none of the 10 sources its claims cite is checked for support yet.
The role of formative evaluation in implementation research and the QUERI experience
Open accessOverview
AI snapshot
Written from the paper’s full text.
TL;DR
This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.
Key findings
- Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
- The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
- QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
- Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.
Methods
- Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
- Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
- Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.
Limitations
- Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
- Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
- Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own interpretation · 20 Own result · 8 Not checked yet · 7 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interview responses indicated that planning, including identification of resources and potential barriers and facilitators, was a critical and universally underutilized step in the intervention process.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Implementation-Focused Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the depression project, FE has generated information about factors influencing caseload capacity for nurse ''care managers'' who maintain regular contact with depressed patients to monitor treatment adherence, assess symptomatic improvement, provide patient education, and facilitate communication among patients, primary care clinicians, and mental health specialists.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: the weekly teleconferencing and problem resolutions changed the perceptions of clinical staff that this research was not related to day-to-day activities.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: the analysis identified barriers to successful implementation that related primarily to the intervention process and secondarily to characteristics of the intervention context.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The research team of CHF QUERI was able to negotiate a policy change, so access to a scale was no longer a barrier to self-monitoring fluid retention when patients were discharged.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Implementation-Focused Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Data from the interviews suggested tools and guidelines to improve planning and implementation skills.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One barrier to accomplishing the weight management goal was uncovered during weekly calls.The Prosthetic Department at 1 site did not interpret current policy in a manner that allowed provision of scales for CHF patients.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 13
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Formative evaluation allowed quick identification of stagnation of progress that suggested the need for increased facilitation or introduction of new tools to sustain enthusiasm for the intervention.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Seventeen prioritized short-and long-term recommendations were generated to improve the usefulness and usability of the 9 reminders.59”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
What the sources say
- Not checked yetcites [59] Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“IQS FE was also used to refine the intervention, to some extent, during the project, and to a major extent for the next ''nationwide rollout'' phase.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prototype toolkits for implementing QI and monitoring clinic practices and patient outcomes were distributed to 8 pilot clinics.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Value: while the FE provided some suggestions about how to modify the intervention and tools, the data were also useful to accurately ''tell the story'' of the individual clinics and to illustrate differences between clinics that were successful with implementation versus those that were not.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, we were able to see that lower performing IQS sites were slow to begin applying key concepts and had not completed an entire PDSA cycle well into an action period.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Value: i.The original toolkit was modified to improve its usefulness.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The calls also provided a model of problem-solving behaviors for the clinical staff (e.g.,Plan, Do, Study, Act) as well as information for the research team that enabled ''user friendly'' modifications of original project tools.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the project moved forward, it was clear that some sites were very successful in applying the treatment plan while others stagnated.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We identified 6 barriers to the effective use of the CR intervention, including workload, lack of time to follow-up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation provides techniques for obtaining such information and for overcoming limitations identified in early implementation/QI studies.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > THE VETERAN HEALTH ADMINISTRATION'S QUERI PROGRAM
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“As Stead et al. 10,11 suggest, traditional intervention research can fail to ''capture the detail and complexity of intervention inputs and tactics'' (10, p. 354), thereby missing the true nature of interventions as well as significant organizational factors important for replication.10,11”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > NEED FOR FE IN IMPLEMENTATION/QI RESEARCH
What the sources say
- Not checked yetcites Stead et al. 10,11
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementation researchers need to answer critical questions about the feasibility of implementation strategies, degree of real-time implementation, status and potential influence of contextual factors, response of project participants, and any adaptations necessary to achieve optimal change.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > THE VETERAN HEALTH ADMINISTRATION'S QUERI PROGRAM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation can save time and frustration as data highlight factors that impede the ability of clinicians to implement best practices.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it can make the realities and black box nature of implementation more transparent to decision makers;”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interpretive FE uses the results of all other FE stages.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementation-focused formative data also are used to keep the strategies on track and as a result optimize the likelihood of affecting change by resolving actionable barriers, enhancing identified levers of change, and refining components of the implemen-tation interventions.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, FE data collected at this stage offer several advantages.They can (a) highlight actual versus planned interventions, (b) enable implementation through identification of modifiable barriers, (c) facilitate any needed refinements in the original implementation intervention, (d) enhance interpretation of project results, and (e) identify critical details and guidance necessary for re”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Table 4. Implementation-Focused FE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such developmental data enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Developmental Evaluation
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, interpretive information can be collected at the end of the project about key stakeholder experiences.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proactive nature of FE is emphasized, as progress data become feedback about the degree of movement toward desired outcomes.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > 3. Intensity of implementation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, as indicated in some of the examples, various evaluative activities can serve multiple stages, which then merge in practice.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > FORMATIVE EVALUATION WITHIN QUERI
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation helps to meet the many challenges to effective implementation and its scientific study, thereby facilitating integration of research findings into practice and improvement of patient care.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Formative evaluation is a study approach that is often key to the success, interpretation, and replication of the results of implementation/QI projects.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > SUMMARY
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In QUERI, this action focus differentiates FE from ''process'' evaluations where data are not intended for concurrent use.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > FORMATIVE EVALUATION WITHIN QUERI
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, FE data provide alternative explanations for results, help to clarify the meaning of success in implementation, and enhance understanding of an implementation strategy's impact or ''worth.''”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
Implementation work tends to be disorderly and complicated.
“As implementation efforts can be a relatively messy and complex process”
The paper relies on this without support.
Factors identified in a general way are likely to differ from one implementation site to another.
“Even if available, a diagnostic analysis is suggested given the likelihood that generically identified factors will vary across implementation sites.”
The paper relies on this without support.
In well-controlled randomized controlled trials, researchers do not typically modify an experimental intervention once it has been approved.
“In the well-controlled RCT, researchers do not typically modify an experimental intervention once approved.”
The paper relies on this without support.
Formative evaluation is a new concept when applied to health services research.
“Formative evaluation is a new concept as applied to health services research and as such presents multiple challenges.”
The paper relies on this without support.
Formative evaluation is a time-consuming activity.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Some researchers need help understanding how formative evaluation can be incorporated into a study design.
“Some researchers may need help in understanding how FE can be incorporated into a study design.”
The paper relies on this without support.
Project leaders may need to be convinced of formative evaluation's utility before committing study resources.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Implementation and quality improvement researchers may encounter the erroneous view that formative evaluation involves only qualitative research or is not rigorous.
“Implementation and QI researchers may encounter the erroneous view that FE involves only qualitative research or that it is not rigorous, e.g., that it consists of ''just talking to a few people''.”
The paper relies on this without support.
Inferential statistical significance does not necessarily equate with clinical significance.
“However, as inferential statistical significance does not necessarily equate with clinical significance, it is useful to obtain perceptions of stakeholders relative to the ''meaning'' of statistical findings.”
The paper relies on this without support.
Research evidence indicates that people with negative attitudes about influenza vaccine safety, effectiveness, or utility are less likely to seek it annually.
“Implementation goal: research evidence indicates that persons who have negative attitudes about influenza vaccine (safety, effectiveness, utility) are less likely to seek it out annually.”
The paper relies on this without support.
Formative evaluation, as a descriptive assessment activity, does not by itself test hypotheses.
“Formative evaluation, as a descriptive assessment activity, does not per se test hypotheses.”
The paper relies on this without support.
Formative evaluation does not lack rigor.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
The paper relies on this without support.
Formative evaluation is not simply a matter of qualitative research or a specific qualitative methodology.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In addition, it has been suggested that current conceptions of research designs fail to “maximize clinical utility for practicing clinicians and other decision makers”18; for example, through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of “effective” practices.4,21,22”
From Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact (2012) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [14–16, 19, 20].”
From Guidance for conducting feasibility and pilot studies for implementation trials (2020) · cites this paper as [20]
In the citing paper
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
From The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework (2013) · cites this paper as [10]
In the citing paper
“Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [71]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“Mental health (MH) QUERI”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“NH, mental health”
Citations
Cites
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,639
Qualitative research & evaluation methods
Patton, M. Q. (2002) · Sage eBooks · Cited by 15,530
(1963) · Medical Entomology and Zoology · Cited by 15,085
(1979) · Medical Entomology and Zoology · Cited by 11,252
Gluesing, J. C. (2012) · Sage eBooks · doi:10.4135/9781412994231.n14 · Cited by 3,264
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836
Utilization-focused evaluation : the new century text
Patton, M. Q. (1997) · Cited by 1,469
Complex interventions: how “out of control” can a randomised controlled trial be?
Hawe, P., Shiell, A., & Riley, T. (2004) · BMJ · doi:10.1136/bmj.328.7455.1561 · Cited by 1,244
Evaluation: A Systematic Approach.
Patton, M. Q., Rossi, P. H., Freeman, H. E., & Wright, S. R. (1981) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2066947 · Cited by 933
Getting evidence into practice: the meaning of `context'
McCormack, B., Kitson, A., Harvey, G., Rycroft‐Malone, J., Titchen, A., & Seers, K. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02150.x · Cited by 765
Research works printed but not linked: 12
- Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
- Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.
- Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.
- Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.
- Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.
- Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
- Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.
- Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.
- Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
- LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.
- La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
- Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.
Cited by
Sorted by citations
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
“[3] as diagnostic analysis, implementationand progress-focused evaluations, and interpretive evaluation).”
BackgroundMethodCurran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058
Self-citationAn introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L., Hagedorn, H., Smith, J., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 1,611
“These studies can involve one or more of three broad types of evaluation, process evaluation, formative evaluation, and summative evaluation [13, 31].”
BackgroundSelf-citationProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894
“can address the fundamental issue of public health impact (Stetler et al. 2006).”
BackgroundImplementation science: What is it and why should I care?
Bauer, M. S., & Kirchner, J. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2019.04.025 · Cited by 1,167
Metadata
- Authors
- Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
- Affiliations
- University of Washington (US)
- VA Puget Sound Health Care System (US)
- VA San Diego Healthcare System (US)
- Edward Hines, Jr. VA Hospital (US)
- Minneapolis VA Medical Center (US)
- Baylor College of Medicine (US)
- Central Arkansas Veterans Healthcare System (US)
- Corresponding author
- Stetler, Cheryl B.
- ORCID
- Venue
- Journal of General Internal Medicine
- Published
- 2006-02
- Volume
- 21
- Issue
- S2
- Pages
- S1-S8
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC2557128
- OpenAlex
- W2084378034
- Cited by
- 642
- References
- 45
- Citations by year
- 2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
- FWCI
- 24.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
- MeSH terms
- Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
