The role of formative evaluation in implementation research and the QUERI experience
Open accessAI snapshot
Written from the paper’s full text.
TL;DR
This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.
Key findings
- Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
- The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
- QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
- Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.
Methods
- Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
- Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
- Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.
Limitations
- Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
- Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
- Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 10 cited sources checked · 0 supported · 0 flagged
- Findingown data
In the ATIP project, formative evaluation showed a correlation between the amount of external facilitation and improvement in antipsychotic dosing.
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
Quote in the paperNot a substantial claim - Findingown data
Interview responses showed that planning, including identifying resources and potential barriers and facilitators, was a critical and universally underused step in the intervention process.
“Interview responses indicated that planning, including identification of resources and potential barriers and facilitators, was a critical and universally underutilized step in the intervention process.”
Quote in the paperSubstantial claim - Findingown data
In the SCI QUERI respiratory vaccine initiative, sites differed greatly in how they installed and used the computerized clinical reminder.
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
Quote in the paperSubstantial claim - Findingown data
In the depression project, formative evaluation generated information about factors influencing caseload capacity for nurse care managers.
“In the depression project, FE has generated information about factors influencing caseload capacity for nurse ''care managers'' who maintain regular contact with depressed patients to monitor treatment adherence, assess symptomatic improvement, provide patient education, and facilitate communication among patients, primary care clinicians, and mental health specialists.”
Quote in the paperNot a substantial claim - Findingown data
In the CHF QUERI, weekly teleconferences and solving problems shifted clinical staff's view that the research was disconnected from daily work.
“Value: the weekly teleconferencing and problem resolutions changed the perceptions of clinical staff that this research was not related to day-to-day activities.”
Quote in the paperSubstantial claim
- Findingown data
In an interpretive post hoc evaluation, barriers to successful implementation were mainly related to the intervention process and secondarily to intervention context characteristics.
“Value: the analysis identified barriers to successful implementation that related primarily to the intervention process and secondarily to characteristics of the intervention context.”
Quote in the paperSubstantial claim - Findingreasoning
In the CHF QUERI, the team negotiated a policy change, removing lack of access to a scale as a barrier to self-monitoring fluid retention after discharge.
“The research team of CHF QUERI was able to negotiate a policy change, so access to a scale was no longer a barrier to self-monitoring fluid retention when patients were discharged.”
Quote in the paperNot a substantial claim - Findingown data
In the SCI QUERI respiratory vaccine initiative, every site had the current version of the influenza computerized clinical reminder installed.
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
Quote in the paperSubstantial claim - Findingown data
Interview data suggested tools and guidelines for improving planning and implementation skills.
“Data from the interviews suggested tools and guidelines to improve planning and implementation skills.”
Quote in the paperNot a substantial claim - Findingown data
In the CHF QUERI, a barrier to the weight management goal was that the Prosthetic Department at one site read current policy in a way that prevented giving scales to CHF patients.
“One barrier to accomplishing the weight management goal was uncovered during weekly calls.The Prosthetic Department at 1 site did not interpret current policy in a manner that allowed provision of scales for CHF patients.”
Quote in the paperNot a substantial claim - Findingnone given
In the QUERI example, sharing information and responding to clinic suggestions strengthened working relationships.
“The exchange of information and responsiveness of the implementation team to clinic suggestions improved working relationships.”
Quote in the paperNot a substantial claim - Findingown data
Sites with the most rapid adoption and generalization of quality improvement concepts tended to show significant improvements in care.
“Although the contrast between levels of activation was not always marked, the sites showing the most rapid adoption and generalization of quality improvement concepts tended to show significant improvements in care.”
Quote in the paperSubstantial claim - Findingown data
Higher performing IQS sites quickly adopted and applied basic quality improvement concepts such as PDSA.
“By using the SAS scoring system, we were able to identify that higher performing IQS sites rapidly adopted and applied basic quality improvement concepts, like the PDSA (PLAN, Do, Study, Act).”
Quote in the paperSubstantial claim - Findingown data
Formative evaluation showed the acceptability and utility of mental health QUERI tools and products.
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
Quote in the paperSubstantial claim - Findingown data
Formative evaluation showed the extent to which lack of clinician time inhibits dissemination and quality improvement activities.
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
Quote in the paperNot a substantial claim - Findingown data
Data on prevalent negative attitudes were used to create targeted questionnaires and educational materials to counter those attitudes.
“Value: data regarding prevalent negative attitudes were used to construct targeted questionnaires as well as educational materials to counter these attitudes.”
Quote in the paperNot a substantial claim - Interpretationnone given
Randomized clinical trials may leave unanswered questions that are important for system-wide uptake of targeted research.
“For example, randomized clinical trials (RCT) may leave questions important to system-wide uptake of targeted research unanswered.”
Quote in the paperSubstantial claim - Interpretationreasoning
In an action-oriented improvement program such as QUERI, summative evaluation data are necessary but not enough for implementation and quality improvement researchers.
“In an action-oriented improvement program, such as QUERI, summative data are essential but insufficient to meet the needs of implementation/QI researchers.”
Quote in the paperSubstantial claim - Premisecitation
When implementation is not assessed, a Type III error can result: study interpretations may be wrong because the intervention was not delivered as intended.
“Implementation assessment failure can lead to a ''Type III'' error, where erroneous study interpretations occur because the intervention was not implemented as planned.12,13”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ovretveit and Gustafson 12,13
Not checked in the free run.
- Interpretationreasoning
Because implementation efforts can be messy and complex, traditional study designs by themselves are often not sufficient to obtain evaluative information.
“As implementation efforts can be a relatively messy and complex process, traditional study designs alone are often inadequate to the task of obtaining evaluative information.”
Quote in the paperSubstantial claim- Rests on an assumption: Implementation work tends to be disorderly and complicated.
- Interpretationreasoning
The original plan to create one standard set of instructions for the influenza computerized clinical reminder was not suitable.
“This indicated that the initial plan to develop a standard set of instructions was inappropriate.”
Quote in the paperSubstantial claim - Interpretationreasoning
In the CHF QUERI, system-level changes made it more likely that patient education would improve self-monitoring of weight.
“Overall, changes at the system level improved the probability that patient education would lead to better self-monitoring of weight.”
Quote in the paperSubstantial claim - Premisenone given
Within QUERI, the stages of formative evaluation-development/diagnosis, implementation, progress, and interpretation-are progressive, integrated components of a single hybrid project.
“Within QUERI, these stages are progressive, integrated components of a single hybrid project.”
Quote in the paperSubstantial claim - Interpretationreasoning
Formative evaluation data can give insights into what determines behavior or system change.
“Overall, FE data may provide evidence regarding targeted components of a conceptual framework, insights into the determinants of behavior or system change, and hypotheses for future testing.”
Quote in the paperSubstantial claim - Interpretationnone given
Key informant or focus group interviews may allow negative unintended consequences to be anticipated.
“For example, negative unintended consequences might be prospectively identified by key informant or focus group interviews.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Instead of identifying modifiable components only after the study, formative evaluation gives timely feedback that reduces the chance of type III errors.
“Rather than identify such modifiable components on a post hoc basis, FE provides timely feedback to lessen the likelihood of type III errors (see Tables 4, 6, 7, and 9).”
Quote in the paperSubstantial claim - Interpretationnone given
Interpreting the black box happens by triangulating qualitative and quantitative formative evaluation data at the end point, including associations with impacts.
“Such ''black box'' interpretation occurs through the end point triangulation of qualitative and quantitative FE data, including associational relationships with impacts.”
Quote in the paperSubstantial claim - Interpretationreasoning
Formative data gathered during implementation help researchers more fully describe and understand the main obstacles to reaching goals and what change actually requires, including when project activities occur.
“Implementation-focused formative data enable researchers to describe and understand more fully the major barriers to goal achievement and what it actually takes to achieve change, including the timing of project activities.”
Quote in the paperSubstantial claim - Premisecitation
Assessing results only after implementation ends hides potentially valuable information about practice trends during the study, which could indicate whether an effort is building momentum or cannot be sustained.
“As Krumholz and Herrin 49 suggest, waiting until implementation is completed to assess results ''obscures potentially important information . . .about trends in practice during the study [that] could demonstrate if an effort is gaining momentum-or that it is not sustainable'' (see Tables 6 and 7).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Krumholz and Herrin 49
Not checked in the free run.
- Interpretationreasoning
Formative data can show how much time an intervention really requires, since poor results may stem from too little intervention time.
“In terms of timing, formative data can clarify the true length of time needed to complete an intervention, as failure to achieve results could relate to insufficient intervention time.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Formative evaluation data can generate hypotheses for later testing.
“Overall, FE data may provide evidence regarding targeted components of a conceptual framework, insights into the determinants of behavior or system change, and hypotheses for future testing.”
Quote in the paperSubstantial claim - Interpretationcitation
Pairing formative evaluation with an experimental study can produce a hybrid design for implementation research, in which the trial is combined with descriptive or observational work using both qualitative and quantitative methods, yielding a richer dataset for interpreting results.
“Rather, complementary use of FE within an experimental study can create a dual or hybrid style approach for implementation research.15 The experimental design is thus combined with descriptive or observational research that employs a mix of qualitative and quantitative techniques, creating a richer dataset for interpreting study results.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [15] Mittman B. Creating the evidence base for quality improvement collaboratives. Ann Int Med. 2004;140:897–901. source
Not checked in the free run.
- Interpretationnone given
Formative evaluation can identify early whether desired outcomes are being achieved, allowing implementation strategies to be refined as needed.
“It can also identify at an early stage whether desired outcomes are being achieved so that implementation strategies can be refined as needed;”
Quote in the paperSubstantial claim - Interpretationcitation
Interpretive formative evaluation can be treated as case study data that help build theory.
“contribute to theory building.58”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [58] Yin R. Case Study Research: Design and Methods. Thousand Oaks: Sage; 1994.
Not checked in the free run.
- Interpretationnone given
Formative evaluation can increase the likelihood of obtaining credible summative results about the effectiveness and transferability of an implementation strategy.
“and it can increase the likelihood of obtaining credible summative results about effectiveness and transferability of an implementation strategy.”
Quote in the paperNot a substantial claim - Interpretationreasoning
When data on what actually happened relative to the implementation plan and on contextual factors that influenced implementation are not collected, potential study users have little understanding of a particular implementation strategy.
“By failing to collect such data, potential study users have little understanding of a particular implementation strategy.”
Quote in the paperSubstantial claim - Interpretationreasoning
Documenting what implementation actually involves may uncover new intervention possibilities.
“By describing the actuality of implementation, new interventions may be revealed.”
Quote in the paperSubstantial claim - Interpretationreasoning
Implementation data on dose, intensity, and barriers can reveal what is impeding progress.
“Using implementation data on dose, intensity, and barriers, factors blocking progress may be identified.”
Quote in the paperSubstantial claim - Interpretationnone given
In improvement-oriented research, critical problems that would prevent an optimal test of the planned implementation can be found and addressed.
“However, in improvement-oriented research, critical problems that prevent an optimal test of the planned implementation can be identified and resolved.”
Quote in the paperSubstantial claim- Rests on an assumption: In well-controlled randomized controlled trials, researchers do not typically modify an experimental intervention once it has been approved.
- Interpretationcitation
In an experimental study, concurrent formative evaluation can supply working hypotheses that explain why implementation succeeded or failed, especially when implementation and evaluation plans are based on a conceptual framework.
“However, within an experimental study, in-depth data from a concurrent FE can provide working hypotheses to explain successes or failures, particularly when the implementation and evaluation plans are grounded in a conceptual framework.[55][56][57]”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [55] Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use. J Biomed Inform. 2003;36:218–27. source
Not checked in the free run.
- cites [56] Sanson-Fisher RW, Grimshaw JM, Eccles MP. The science of changing providers’ behaviour: the missing link in evidence-based practice. Med J Aust. 2004;180:205–6. source
Not checked in the free run.
- cites [57] Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME—PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice. BMC Health Serv Res. 2003;3:22. source
Not checked in the free run.
- Rests on an assumption: Formative evaluation, as a descriptive assessment activity, does not by itself test hypotheses.
- Interpretationreasoning
Formative evaluation data can provide evidence about specific components of a conceptual framework.
“Overall, FE data may provide evidence regarding targeted components of a conceptual framework, insights into the determinants of behavior or system change, and hypotheses for future testing.”
Quote in the paperSubstantial claim - Interpretationcitation
A participatory approach may also strengthen commitment among the intended users.
“This participatory approach may also facilitate commitment among targeted users.34”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [34] Brown K, Gerhardt M. Formative evaluation: an integrative practice model and case study. Personnel Psychol. 2002;55:951f. source
Not checked in the free run.
- May combine two claims. 16
- Classified as probably not a substantial claim. 13
May combine two claims.
FindingreasoningIn the QUERI example, formative evaluation quickly spotted stalled progress, indicating that more facilitation or new tools were needed to keep enthusiasm for the intervention.
“Formative evaluation allowed quick identification of stagnation of progress that suggested the need for increased facilitation or introduction of new tools to sustain enthusiasm for the intervention.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingcitationSeventeen prioritized short- and long-term recommendations were produced to improve the usefulness and usability of the nine reminders.
“Seventeen prioritized short-and long-term recommendations were generated to improve the usefulness and usability of the 9 reminders.59”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [59] Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9. source
Not checked in the free run.
May combine two claims.
Findingnone givenIQS formative evaluation was used to refine the intervention somewhat during the project and to a major extent for the next nationwide rollout phase.
“IQS FE was also used to refine the intervention, to some extent, during the project, and to a major extent for the next ''nationwide rollout'' phase.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataIn the QUERI example, prototype toolkits for quality improvement and for monitoring clinic practices and patient outcomes went to 8 pilot clinics.
“Prototype toolkits for implementing QI and monitoring clinic practices and patient outcomes were distributed to 8 pilot clinics.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFormative evaluation data were useful for accurately telling the story of individual clinics and illustrating differences between clinics that succeeded with implementation and those that did not.
“Value: while the FE provided some suggestions about how to modify the intervention and tools, the data were also useful to accurately ''tell the story'' of the individual clinics and to illustrate differences between clinics that were successful with implementation versus those that were not.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataLower performing IQS sites were slow to start applying key concepts and had not completed a full PDSA cycle well into an action period.
“In contrast, we were able to see that lower performing IQS sites were slow to begin applying key concepts and had not completed an entire PDSA cycle well into an action period.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningIn the QUERI example, the original toolkit was changed to make it more useful.
“Value: i.The original toolkit was modified to improve its usefulness.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the CHF QUERI, the calls modeled problem-solving for clinical staff and gave the research team information that allowed original project tools to be made more user-friendly.
“The calls also provided a model of problem-solving behaviors for the clinical staff (e.g.,Plan, Do, Study, Act) as well as information for the research team that enabled ''user friendly'' modifications of original project tools.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAs the project progressed, some sites were very successful in applying the treatment plan while others stagnated.
“As the project moved forward, it was clear that some sites were very successful in applying the treatment plan while others stagnated.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSix barriers to effective use of the clinical reminder intervention were identified: workload, lack of time to follow up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.
“We identified 6 barriers to the effective use of the CR intervention, including workload, lack of time to follow-up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningFormative evaluation offers methods to obtain such information and to overcome limitations found in early implementation and quality improvement studies.
“Formative evaluation provides techniques for obtaining such information and for overcoming limitations identified in early implementation/QI studies.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationTraditional intervention studies may fail to record the detailed and complex nature of what is delivered, and so may overlook both the actual character of interventions and key organizational factors needed for replication.
“As Stead et al. 10,11 suggest, traditional intervention research can fail to ''capture the detail and complexity of intervention inputs and tactics'' (10, p. 354), thereby missing the true nature of interventions as well as significant organizational factors important for replication.10,11”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Stead et al. 10,11
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
InterpretationreasoningImplementation researchers require evaluative information beyond clinical impact and beyond whether an adoption strategy worked, including feasibility of strategies, real-time implementation, contextual factors, participant response, and necessary adaptations.
“Implementation researchers need to answer critical questions about the feasibility of implementation strategies, degree of real-time implementation, status and potential influence of contextual factors, response of project participants, and any adaptations necessary to achieve optimal change.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenFormative evaluation can save time and reduce frustration because its data reveal factors that hinder clinicians from implementing best practices.
“Formative evaluation can save time and frustration as data highlight factors that impede the ability of clinicians to implement best practices.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenFormative evaluation can make the realities and black-box nature of implementation more transparent to decision makers.
“it can make the realities and black box nature of implementation more transparent to decision makers;”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenInterpretive formative evaluation draws on the results from every other formative evaluation stage.
“Interpretive FE uses the results of all other FE stages.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningFormative data collected during implementation help keep strategies on course and improve the chance of producing change by addressing barriers that can be acted on, strengthening known levers of change, and refining parts of the implementation intervention.
“Implementation-focused formative data also are used to keep the strategies on track and as a result optimize the likelihood of affecting change by resolving actionable barriers, enhancing identified levers of change, and refining components of the implemen-tation interventions.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningAt the implementation stage, formative evaluation data have multiple benefits: they reveal differences between planned and actual interventions, support implementation by finding barriers that can be changed, allow needed adjustments to the original intervention, aid interpretation of project outcomes, and provide crucial details and guidance for reproducing results elsewhere.
“In summary, FE data collected at this stage offer several advantages.They can (a) highlight actual versus planned interventions, (b) enable implementation through identification of modifiable barriers, (c) facilitate any needed refinements in the original implementation intervention, (d) enhance interpretation of project results, and (e) identify critical details and guidance necessary for re”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningData from developmental evaluation help researchers identify potential problems and, when possible, address them before interventions start at study sites.
“Such developmental data enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenAt the end of a project, interpretive information about key stakeholders' experiences can be gathered.
“In addition, interpretive information can be collected at the end of the project about key stakeholder experiences.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationreasoningProgress-focused formative evaluation is proactive: data on progress serve as feedback about how far the project has moved toward intended outcomes.
“The proactive nature of FE is emphasized, as progress data become feedback about the degree of movement toward desired outcomes.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningSome evaluation activities can contribute to more than one formative evaluation stage, and in practice the stages blend together.
“However, as indicated in some of the examples, various evaluative activities can serve multiple stages, which then merge in practice.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningFormative evaluation helps address many challenges to effective implementation and its scientific study, thereby aiding integration of research findings into practice and improvement of patient care.
“Formative evaluation helps to meet the many challenges to effective implementation and its scientific study, thereby facilitating integration of research findings into practice and improvement of patient care.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningFormative evaluation is often essential to the success, interpretation, and replication of implementation and quality improvement project results.
“Formative evaluation is a study approach that is often key to the success, interpretation, and replication of the results of implementation/QI projects.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisereasoningWithin QUERI, formative evaluation's emphasis on action sets it apart from process evaluations whose data are not meant to be used concurrently.
“In QUERI, this action focus differentiates FE from ''process'' evaluations where data are not intended for concurrent use.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationnone givenInterpretive formative evaluation data offer alternative explanations for findings, clarify what success in implementation means, and deepen understanding of an implementation strategy's impact or value.
“Specifically, FE data provide alternative explanations for results, help to clarify the meaning of success in implementation, and enhance understanding of an implementation strategy's impact or ''worth.''”
Quote in the paperSubstantial claim
The paper relies on this without support.
Implementation work tends to be disorderly and complicated.
“As implementation efforts can be a relatively messy and complex process”
The paper relies on this without support.
Factors identified in a general way are likely to differ from one implementation site to another.
“Even if available, a diagnostic analysis is suggested given the likelihood that generically identified factors will vary across implementation sites.”
The paper relies on this without support.
In well-controlled randomized controlled trials, researchers do not typically modify an experimental intervention once it has been approved.
“In the well-controlled RCT, researchers do not typically modify an experimental intervention once approved.”
The paper relies on this without support.
Formative evaluation is a new concept when applied to health services research.
“Formative evaluation is a new concept as applied to health services research and as such presents multiple challenges.”
The paper relies on this without support.
Formative evaluation is a time-consuming activity.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Some researchers need help understanding how formative evaluation can be incorporated into a study design.
“Some researchers may need help in understanding how FE can be incorporated into a study design.”
The paper relies on this without support.
Project leaders may need to be convinced of formative evaluation's utility before committing study resources.
“Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
The paper relies on this without support.
Implementation and quality improvement researchers may encounter the erroneous view that formative evaluation involves only qualitative research or is not rigorous.
“Implementation and QI researchers may encounter the erroneous view that FE involves only qualitative research or that it is not rigorous, e.g., that it consists of ''just talking to a few people''.”
The paper relies on this without support.
Inferential statistical significance does not necessarily equate with clinical significance.
“However, as inferential statistical significance does not necessarily equate with clinical significance, it is useful to obtain perceptions of stakeholders relative to the ''meaning'' of statistical findings.”
The paper relies on this without support.
Research evidence indicates that people with negative attitudes about influenza vaccine safety, effectiveness, or utility are less likely to seek it annually.
“Implementation goal: research evidence indicates that persons who have negative attitudes about influenza vaccine (safety, effectiveness, utility) are less likely to seek it out annually.”
The paper relies on this without support.
Formative evaluation, as a descriptive assessment activity, does not by itself test hypotheses.
“Formative evaluation, as a descriptive assessment activity, does not per se test hypotheses.”
The paper relies on this without support.
Formative evaluation does not lack rigor.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
The paper relies on this without support.
Formative evaluation is not simply a matter of qualitative research or a specific qualitative methodology.
“However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current conceptions of research designs fail to maximize clinical utility for practicing clinicians and other decision makers, for example through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of effective practices.
“In addition, it has been suggested that current conceptions of research designs fail to “maximize clinical utility for practicing clinicians and other decision makers”18; for example, through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of “effective” practices.4,21,22”
From Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact (2012) · cites this paper as [21]
The field of implementation science has identified the need for nuanced guidance because of differences between implementation trials and conventional clinical or public health efficacy trials.
“However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [14–16, 19, 20].”
From Guidance for conducting feasibility and pilot studies for implementation trials (2020) · cites this paper as [20]
The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings.
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
From The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework (2013) · cites this paper as [10]
Stetler et al. provide four progressive yet integrated stages of formative evaluation.
“Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [71]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Conflicting abbreviation for mental health in Table 8
MinorThe table introduces MH as the abbreviation for mental health, but its abbreviation key instead assigns that meaning to NH.
“Mental health (MH) QUERI”
“NH, mental health”
Cites
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,631 · see citing works
Qualitative research & evaluation methods
Patton, M. Q. (2002) · Sage eBooks · Cited by 15,530 · see citing works
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
(1979) · Medical Entomology and Zoology · Cited by 11,252 · see citing works
Gluesing, J. C. (2012) · Sage eBooks · doi:10.4135/9781412994231.n14 · Cited by 3,259 · see citing works
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836 · see citing works
Utilization-focused evaluation : the new century text
Patton, M. Q. (1997) · Cited by 1,469 · see citing works
Complex interventions: how “out of control” can a randomised controlled trial be?
Hawe, P., Shiell, A., & Riley, T. (2004) · BMJ · doi:10.1136/bmj.328.7455.1561 · Cited by 1,244 · see citing works
Evaluation: A Systematic Approach.
Patton, M. Q., Rossi, P. H., Freeman, H. E., & Wright, S. R. (1981) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2066947 · Cited by 933 · see citing works
Getting evidence into practice: the meaning of `context'
McCormack, B., Kitson, A., Harvey, G., Rycroft‐Malone, J., Titchen, A., & Seers, K. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02150.x · Cited by 765 · see citing works
Ingredients for change: revisiting a conceptual framework
Rycroft-Malone, J. (2002) · Quality and Safety in Health Care · doi:10.1136/qhc.11.2.174 · Cited by 701 · see citing works
Process evaluation on quality improvement interventions
Hulscher, M. (2003) · BMJ Quality & Safety · doi:10.1136/qhc.12.1.40 · Cited by 349 · see citing works
A Qualitative Study of Increasing β-Blocker Use After Myocardial Infarction
Bradley, E. H., Holmboe, E. S., Mattera, J. A., Roumanis, S. A., Radford, M. J., & Krumholz, H. M. (2001) · JAMA · doi:10.1001/jama.285.20.2604 · Cited by 292 · see citing works
Creating the Evidence Base for Quality Improvement Collaboratives
Mittman, B. S. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-11-200406010-00011 · Cited by 279 · see citing works
Effectiveness of quality improvement: learning from evaluations
Walshe, K. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.1.85 · Cited by 210 · see citing works
Kukafka, R., Johnson, S. B., Linfante, A., & Allegrante, J. P. (2003) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2003.09.002 · Cited by 206 · see citing works
Designing a quality improvement intervention: a systematic approach
van Bokhoven, M. A. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.3.215 · Cited by 204 · see citing works
Santacroce, S. J., Maccarelli, L. M., & Grey, M. J. (2004) · Nursing Research · doi:10.1097/00006199-200401000-00010 · Cited by 178 · see citing works
Walker, A. E., Grimshaw, J., Johnston, M., Pitts, N., Steen, N., & Eccles, M. (2003) · BMC Health Services Research · doi:10.1186/1472-6963-3-22 · Cited by 133 · see citing works
Demakis, J. G., McQueen, L., Kizer, K. W., & Feussner, J. R. (2000) · Medical Care · doi:10.1097/00005650-200006001-00003 · Cited by 131 · see citing works
Guideline Implementation: What the Literature Doesn’t Tell Us
Solberg, L. I. (2000) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(00)26044-6 · Cited by 129 · see citing works
Formative and Process Evaluation of Health Promotion and Disease Prevention Programs
Dehar, M.-A., Casswell, S., & Duignan, P. (1993) · Evaluation Review · doi:10.1177/0193841x9301700205 · Cited by 127 · see citing works
Using research to inform quality programmes
Øvretveit, J., & Gustafson, D. (2003) · BMJ · doi:10.1136/bmj.326.7392.759 · Cited by 120 · see citing works
Human Factors Barriers to the Effective Use of Ten HIV Clinical Reminders
Patterson, E. S., Nguyen, A. D., Halloran, J. P., & Asch, S. M. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1364 · Cited by 115 · see citing works
McQueen, L., Mittman, B. S., & Demakis, J. G. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1499 · Cited by 106 · see citing works
FORMATIVE EVALUATION: AN INTEGRATIVE PRACTICE MODEL AND CASE STUDY
BROWN, K. G., & GERHARDT, M. W. (2002) · Personnel Psychology · doi:10.1111/j.1744-6570.2002.tb00137.x · Cited by 86 · see citing works
A framework for evaluating community-based heart disease prevention programs
Altman, D. G. (1986) · Social Science & Medicine · doi:10.1016/0277-9536(86)90052-3 · Cited by 67 · see citing works
The challenge of evaluating complex interventions: a framework for evaluating media advocacy
Stead, M. (2002) · Health Education Research · doi:10.1093/her/17.3.351 · Cited by 59 · see citing works
Sharp, N. D., Piñeros, S. L., Hsu, C. W., Starks, H. E., & Sales, A. E. (2004) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2004.04023.x · Cited by 50 · see citing works
Self-citationNazareth, I., Freemantle, N., Duggan, C. A., Mason, J. M., & Haines, A. (2002) · Journal of Health Services Research & Policy · doi:10.1258/135581902320432769 · Cited by 46 · see citing works
Zapka, J. G., Goins, K. V., Pbert, L. A., & Ockene, J. K. (2004) · Health Promotion Practice · doi:10.1177/1524839904263713 · Cited by 35 · see citing works
Willenbring, M. L., Hagedorn, H., Postier, A. C., & Kenny, M. E. (2004) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2004.01.009 · Cited by 32 · see citing works
Self-citationBoyd, N. R., & Windsor, R. A. (2003) · Maternal and Child Health Journal · doi:10.1023/a:1023873112024 · Cited by 32 · see citing works
The science of changing providers’ behaviour: the missing link in evidence‐based practice
Sanson‐Fisher, R. W., Grimshaw, J. M., & Eccles, M. P. (2004) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2004.tb05884.x · Cited by 30 · see citing works
Lichtman, J. H., Roumanis, S. A., Radford, M. J., Riedinger, M. S., Weingarten, S. R., & Krumholz, H. M. (2001) · The Joint Commission Journal on Quality Improvement · doi:10.1016/s1070-3241(01)27005-9 · Cited by 28 · see citing works
Bhola, H. S. (1990) · Medical Entomology and Zoology · Cited by 28 · see citing works
Many A Slip Between Cup And Lip
Forsetlund, L., Talseth, K. O., Bradley, P., Nordheim, L. V., & Bjørndal, A. (2003) · Evaluation Review · doi:10.1177/0193841x02250528 · Cited by 24 · see citing works
Quality improvement studies: the need is there but so are the challenges
Krumholz, H. M., & Herrin, J. S. (2000) · The American Journal of Medicine · doi:10.1016/s0002-9343(00)00593-3 · Cited by 22 · see citing works
Uses of Process Evaluation in the Maryland WIC 5-a-Day Promotion Program
Havas, S., Anliker, J. A., Damron, D. J., et al. (2000) · Health Education & Behavior · doi:10.1177/109019810002700211 · Cited by 20 · see citing works
Issues in intervention research: Maintaining integrity
Kirchhoff, K. T., & Dille, C. A. (1994) · Applied Nursing Research · doi:10.1016/0897-1897(94)90018-3 · Cited by 14 · see citing works
Formative evaluation in school-based health promotion investigations
Evans, R. I., Raines, B. E., & Owen, A. E. (1989) · Preventive Medicine · doi:10.1016/0091-7435(89)90070-4 · Cited by 13 · see citing works
Impact of Organizational Redesign
Effken, J. A., & Stetler, C. B. (1997) · JONA The Journal of Nursing Administration · doi:10.1097/00005110-199707000-00009 · Cited by 11 · see citing works
Self-citationImplementing Evidence-Based Practices In Opioid Agonist Therapy Clinics
Willenbring, M. L., & Hagedorn, H. (2004) · doi:10.1093/oso/9780195165005.003.0035 · Cited by 3 · see citing works
Self-citationSeries on nursing administration Vol. 2
Webb, C. (1990) · Nurse Education Today · doi:10.1016/0260-6917(90)90066-y · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 45 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
- Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.
- Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.
- Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.
- Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.
- Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
- Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.
- Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.
- Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
- LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.
- La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
- Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.
Cited by
Sorted by citations
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · see citing works
“[3] as diagnostic analysis, implementationand progress-focused evaluations, and interpretive evaluation).”
BackgroundMethodCurran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058 · see citing works
Self-citationAn introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L. J., Hagedorn, H., Smith, J. L., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 2,023 · see citing works
“These studies can involve one or more of three broad types of evaluation, process evaluation, formative evaluation, and summative evaluation [13, 31].”
BackgroundSelf-citationProctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894 · see citing works
“can address the fundamental issue of public health impact (Stetler et al. 2006).”
BackgroundImplementation science: What is it and why should I care?
Bauer, M. S., & Kirchner, J. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2019.04.025 · Cited by 1,167 · see citing works
Feldstein, A. C., & Glasgow, R. E. (2008) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(08)34030-6 · Cited by 987 · see citing works
“that begin with a barrier and needs assessment are most effective.(5,31) The Re-Fracture Intervention Trial (REFIT; Table 2) Phase 1 RCT revealed that patient-specific clinical guideline advice to the primary care provider (PCP)”
BackgroundKontos, P. C., & Poland, B. D. (2009) · Implementation Science · doi:10.1186/1748-5908-4-1 · Cited by 760 · see citing works
“It is an approach to implementation evaluation (also referred to as formative evaluation) [63] that, when combined with outcomes evaluation, creates a powerful 'hybrid style approach for implementation research' [63] which provides a clearer direction for action because the decision maker not only has knowledge of the…”
BackgroundTen recommendations for using implementation frameworks in research and practice
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Albers, B., Nilsen, P., Broder-Fingert, S., Mukasa, B., & Aarons, G. A. (2020) · Implementation Science Communications · doi:10.1186/s43058-020-00023-7 · Cited by 488 · see citing works
Guidance for conducting feasibility and pilot studies for implementation trials
Pearson, N., Naylor, P.-J., Ashe, M. C., Fernandez, M., Yoong, S. L., & Wolfenden, L. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00634-w · Cited by 475 · see citing works
A guiding framework and approach for implementation research in substance use disorders treatment.
Damschroder, L. J., & Hagedorn, H. J. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022284 · Cited by 427 · see citing works
Self-citationStetler, C. B., Legro, M. W., Rycroft‐Malone, J., Bowman, C., Curran, G. M., Guihan, M., Hagedorn, H., Piñeros, S. L., & Wallace, C. M. (2006) · Implementation Science · doi:10.1186/1748-5908-1-23 · Cited by 409 · see citing works
“definitions within the implementation field, as well as a failure at times to differentiate planned versus actual operationalization of those interventions [ 11 ,12].”
BackgroundInfluentialSelf-citationA Guide for applying a revised version of the PARIHS framework for implementation
Stetler, C. B., Damschroder, L. J., Helfrich, C. D., & Hagedorn, H. J. (2011) · Implementation Science · doi:10.1186/1748-5908-6-99 · Cited by 402 · see citing works
“• Receptivity to the targeted innovation/change • Develop and organize a retrospective interpretive evaluation [20] to explore the perceived influence of Contextual features on implementation of the targeted EBP.”
BackgroundInfluentialSelf-citationKoorts, H., Eakin, E., Estabrooks, P. A., Timperio, A. F., Salmon, J., & Bauman, A. E. (2018) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-018-0678-0 · Cited by 337 · see citing works
“Formative and process evaluation approaches [36, 37], including intervention testing methodologies [12, 31, 34], inform strategies to address potential barriers to effective implementation in Step 4.”
MethodImplementation of the CALM intervention for anxiety disorders: a qualitative study
Curran, G. M., Sullivan, G., Mendel, P. J., Craske, M. G., Sherbourne, C. D., Stein, M. B., McDaniel, A., & Roy‐Byrne, P. (2012) · Implementation Science · doi:10.1186/1748-5908-7-14 · Cited by 349 · see citing works
“[28] refer to as “developmental formative evaluation”, that is, context-specific “diagnostic” evaluation of barriers and facilitators to implementation, which leverages the experiences and lessons learned in effectiveness trials to enhance the development of wider implementation and dissemination interventions.”
BackgroundRycroft-Malone, J., Seers, K., Chandler, J., Hawkes, C. A., Crichton, N., Allen, C., Bullock, I., & Strunin, L. (2013) · Implementation Science · doi:10.1186/1748-5908-8-28 · Cited by 303 · see citing works
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
BackgroundGetting a clinical innovation into practice: An introduction to implementation strategies
Kirchner, J. E., Smith, J. L., Powell, B. J., Waltz, T. J., & Proctor, E. K. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.06.042 · Cited by 263 · see citing works
“Formative evaluation (FE) is a rigorous assessment process typically involving collection of qualitative and quantitative data to identify the determinants of current practice, barriers, and facilitators for a practice change or implementation of a clinical innovation (Stetler et al., 2006).”
BackgroundInfluentialSelf-citationWoodward, E. N., Singh, R. S., Ndebele‐Ngwenya, P., Castillo, A. M., Dickson, K. S., & Kirchner, J. E. (2021) · Implementation Science Communications · doi:10.1186/s43058-021-00146-5 · Cited by 274 · see citing works
Cohen, D. J., Crabtree, B. F., Etz, R., Balasubramanian, B. A., Donahue, K. E., Leviton, L. C., Clark, E. C., Isaacson, N. F., Stange, K. C., & Green, L. W. (2008) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2008.08.005 · Cited by 248 · see citing works
Wierenga, D., Engbers, L. H., van Empelen, P., Duijts, S. F. A., Hildebrandt, V. H., & van Mechelen, W. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-1190 · Cited by 219 · see citing works
“Hence, effect evaluations should be accompanied by systematic and real-time process evaluations [20,21].”
BackgroundGreen, C. A., Duan, N., Gibbons, R. D., Hoagwood, K. E., Palinkas, L. A., & Wisdom, J. P. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0552-6 · Cited by 205 · see citing works
“Formative, process, and evaluative questions are all fair game (Stetler et al. 2006), and mixed methods designs capitalize on the strengths of each method used while attempting to reduce each method’s weaknesses.”
Metzler, C. W., Sanders, M. R., Rusby, J. C., & Crowley, R. N. (2011) · Behavior Therapy · doi:10.1016/j.beth.2011.05.004 · Cited by 197 · see citing works
“Gathering feedback from potential consumers is central to a formative development and evaluation process that seeks out and incorporates feedback from potential users and accounts for factors that will facilitate or interfere with program usage (Stetler et al., 2006).”
May, C. R., Mair, F. S., Dowrick, C. F., & Finch, T. L. (2007) · BMC Family Practice · doi:10.1186/1471-2296-8-42 · Cited by 183 · see citing works
“Similar methodological or procedural approaches are encountered with other frameworks that explore, for example, empirically identified facilitators and barriers to the successful implementation of complex interventions [10-12].”
Forman, J. H., Creswell, J. W., Damschroder, L. J., Kowalski, C. P., & Krein, S. L. (2008) · American Journal of Infection Control · doi:10.1016/j.ajic.2008.03.010 · Cited by 197 · see citing works
“This knowledge can help us design interventions, improve our understanding of how these interventions work and why they are effective or ineffective, and evaluate them and thereby better translate research into practice.(25) For example, in the area of hand hygiene, researchers can observe hospital staff to determine…”
BackgroundHawk, K. F., D’Onofrio, G., Chawarski, M. C., O’Connor, P. G., Cowan, E. A., Lyons, M. S., Richardson, L. D., Rothman, R. E., Whiteside, L., Owens, P. H., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.4561 · Cited by 182 · see citing works
Stetler, C. B., Mittman, B. S., & Francis, J. (2008) · Implementation Science · doi:10.1186/1748-5908-3-8 · Cited by 176 · see citing works
Self-citationDevelopment and application of the RE-AIM QuEST mixed methods framework for program evaluation
Forman, J. H., Heisler, M., Damschroder, L. J., Kaselitz, E. B., & Kerr, E. A. (2017) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2017.04.002 · Cited by 151 · see citing works
“Retrospective interpretive evaluation (Stetler et al., 2006).”
MethodHarnessing Implementation Science to Increase the Impact of Health Equity Research
Chinman, M., Woodward, E. N., Curran, G. M., & Hausmann, L. R. M. (2017) · Medical Care · doi:10.1097/mlr.0000000000000769 · Cited by 167 · see citing works
“The Hybrid Type I design is a traditional effectiveness trial of a clinical intervention accompanied by a formative evaluation that tracks various aspects of that clinical intervention’s implementation (eg, integrity, dose) which could affect its adoption.(39) As shown in Figure 1, a Hybrid Type I design may be approp…”
Elwy, A. R., Wasan, A. D., Gillman, A. G., Johnston, K. L., Dodds, N. E., McFarland, C. E., & Greco, C. M. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112532 · Cited by 154 · see citing works
Saint, S. K., Olmsted, R. N., Fakih, M. G., Kowalski, C. P., Watson, S. R., Sales, A. E., & Krein, S. L. (2009) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(09)35062-x · Cited by 146 · see citing works
Improving Community-Based Mental Health Care for Children: Translating Knowledge into Action
Garland, A. F., Haine‐Schlagel, R., Brookman-Frazee, L. I., Baker-Ericzén, M. J., Trask, E. V., & Fawley-King, K. (2012) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-012-0450-8 · Cited by 144 · see citing works
“…models that identify effective methods to implement EB interventions into UC highlight the need for contextual data on UC services to inform implementation processes and intervention ‘‘fit’’ (Aarons et al. 2011; Damschroder et al. 2009; Mendel et al. 2008; Powell et al. 2012; Stetler et al. 2006).”
Stetler, C. B., McQueen, L., Demakis, J. G., & Mittman, B. S. (2008) · Implementation Science · doi:10.1186/1748-5908-3-30 · Cited by 144 · see citing works
“▪ Selection of implementation designs and tools that would address the complexities of producing credible output while accounting for the realities of a real-time, dynamic health care setting [3,31];”
InfluentialSelf-citationRitchie, M. J., Parker, L. E., Edlund, C. N., & Kirchner, J. E. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2217-0 · Cited by 112 · see citing works
“Facilitators used formative evaluation techniques [47] to assess clinics’ current practice patterns and anticipate barriers and facilitators to PC-MHI implementation.”
MethodThe role of organizational research in implementing evidence-based practice: QUERI Series
Yano, E. M. (2008) · Implementation Science · doi:10.1186/1748-5908-3-29 · Cited by 112 · see citing works
Designing monitoring arrangements for collaborative learning about adaptation pathways
Hermans, L. M., Haasnoot, M., ter Maat, J., et al. (2016) · Environmental Science & Policy · doi:10.1016/j.envsci.2016.12.005 · Cited by 110 · see citing works
Training Research: Practical Recommendations for Maximum Impact
Beidas, R., Koerner, K., Weingardt, K. R., & Kendall, P. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-011-0338-z · Cited by 107 · see citing works
“For example, formative evaluation, a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts (Stetler et al. 2006), is a further research tool to augment hybrid efficacy-effectiveness models.”
BackgroundA process for developing an implementation intervention: QUERI Series
Curran, G. M., Mukherjee, S., Allee, E., & Owen, R. R. (2008) · Implementation Science · doi:10.1186/1748-5908-3-17 · Cited by 110 · see citing works
Quantitative approaches for the evaluation of implementation research studies
Smith, J. D., & Hasan, M. M. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.112521 · Cited by 115 · see citing works
Self-citationEvaluation of Pragmatic Telehealth Physical Therapy Implementation During the COVID-19 Pandemic
Miller, M. J., Pak, S. S., Keller, D. R., & Barnes, D. E. (2021) · Physical Therapy · doi:10.1093/ptj/pzaa193 · Cited by 106 · see citing works
“002d Non-Hispanic White 1639 (52) 826 (55) Black/African American 190 (6) 91 (6) Asian 666 (21) 240 (16) Hispanic/Latino 298 (9) 152 (10) Unknown 376 (12) 192 (13) Comorbidity burden <.”
Why do interventions work in some places and not others: A breastfeeding support group trial
Hoddinott, P., Britten, J., & Pill, R. (2009) · Social Science & Medicine · doi:10.1016/j.socscimed.2009.10.067 · Cited by 103 · see citing works
Dietary Management of Heart Failure: DASH Diet and Precision Nutrition Perspectives
Wickman, B. E., Enkhmaa, B., Ridberg, R. A., Romero, E., Cadeiras, M., Meyers, F., & Steinberg, F. M. (2021) · Nutrients · doi:10.3390/nu13124424 · Cited by 113 · see citing works
“Implementation science approaches are used to translate research effectively and swiftly into patient care to support the inclusion of new knowledge into systems of care to optimize health care and public health [114,115].”
BackgroundConnell, L., McMahon, N. E., Harris, J. E., Watkins, C. L., & Eng, J. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0090-3 · Cited by 99 · see citing works
“Stetler et al. [11] have highlighted the role of formative evaluation in implementation research, defining it as ‘ a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts.”
Measuring persistence of implementation: QUERI Series
Initiative, T. H. Q. E. R., Bowman, C. C., Sobo, E. J., Asch, S. M., & Gifford, A. L. (2008) · Implementation Science · doi:10.1186/1748-5908-3-21 · Cited by 100 · see citing works
Self-citationChan, C. L., & Ackermann, B. J. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00706 · Cited by 100 · see citing works
“…implementation stages of developing an occupation-specific prevention and management program may increase the likelihood of intervention outcomes being successful (Viera and Kumar, 2004; Stetler et al., 2006; Boocock et al., 2007; Jurg et al., 2008; Bell and Burnett, 2009; Harding et al., 2009).”
BackgroundMethodInfluentialHamilton, A. B., Brunner, J., Cain, C., Chuang, E., Luger, T. M., Canelo, I. A., Rubenstein, L. V., & Yano, E. M. (2017) · Translational Behavioral Medicine · doi:10.1007/s13142-017-0501-5 · Cited by 78 · see citing works
“velop “QI roadmaps”; (2) formative evaluation [28],”
Model for the evaluation of implementation programs and professional pharmacy services
Moullin, J. C., Sabater‐Hernández, D., & Benrimoj, S. I. (2015) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2015.08.003 · Cited by 80 · see citing works
Soi, C., Gimbel, S., Chilundo, B., Muchanga, V., Matsinhe, L., & Sherr, K. (2018) · Implementation Science · doi:10.1186/s13012-018-0846-2 · Cited by 74 · see citing works
“A post-implementation interpretive evaluation [25], which sought to explain implementation success or failure through the triangulation of project implementation stakeholders’ experiences, was carried out.”
MethodHarden, S. M., Johnson, S. B., Almeida, F. A., & Estabrooks, P. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0380-6 · Cited by 75 · see citing works
“However, there has been little research to determine if they result in practical interventions that are more likely to be adopted and implemented when compared to interventions developed through a pipeline model [20, 21].”
BackgroundFulop, N., Ramsay, A. I. G., Perry, C., Boaden, R., McKevitt, C., Rudd, A. G., Turner, S., Tyrrell, P. J., Wolfe, C. D. A., & Stephen Morris, A. (2015) · Implementation Science · doi:10.1186/s13012-016-0445-z · Cited by 71 · see citing works
“Understanding how evidence-based practice is implemented in complex settings such as healthcare is enhanced when its various components are considered (decision to change, intervention selection, planning and implementation of change, and outcomes) [2, 5, 6, 10].”
BackgroundBroyles, L. M., Rodriguez, K. L., Kraemer, K. L., Sevick, M. A., Price, P. A., & Gordon, A. J. (2012) · Addiction Science & Clinical Practice · doi:10.1186/1940-0640-7-7 · Cited by 71 · see citing works
“Consistent with these assumptions and previously identified barriers to the uptake and utilization of alcohol screening, BI, and RT [28,30,31], we developed our semi-structured interview guide to (1) assess current practices in alcohol-related care, and (2) prospectively identify the potential barriers and facilitator…”
MethodInformation Technology to Support Improved Care For Chronic Illness
Young, A. S., Chaney, E. F., Shoai, R., Bonner, L. M., Cohen, A. N., Doebbeling, B. N., Dorr, D. A., Goldstein, M. K., Kerr, E. A., Nichol, P., et al. (2007) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0303-4 · Cited by 71 · see citing works
“also go beyond it to evaluate the role of information systems in clinical and policy efforts to improve care.(39) In this project, the expert panel process resulted in the development of use cases for depression, schizophrenia, and diabetes with comorbidity.”
BackgroundLessons Learned about Implementing Research Evidence into Clinical Practice.
Hagedorn, H., Hogan, M. M., Smith, J. L., Bowman, C., Curran, G. M., Espadas, D., Kimmel, B., Kochevar, L. K., Legro, M. W., & Sales, A. E. (2006) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2006.00358.x · Cited by 76 · see citing works
Self-citationEvaluation of a Pilot Implementation to Integrate Alcohol-Related Care within Primary Care
Bobb, J. F., Lee, A., Lapham, G. T., Oliver, M. M., Ludman, E. J., Achtmeyer, C. E., Parrish, R. S., Caldeiro, R. M., Lozano, P., Richards, J. A., et al. (2017) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph14091030 · Cited by 69 · see citing works
“The two practice coaches interacted with staff in the three pilot PC sites regularly regarding elements of planned implementation that were challenging, and brought information on barriers and facilitators to a weekly implementation-focused FE meeting that included the principal investigator, a co-investigator, and a…”
BackgroundMazzucca, S., Tabak, R. G., Pilar, M. R., Ramsey, A. T., Baumann, A. A., Kryzer, E., Lewis, E. M., Padek, M. M., Powell, B. J., & Brownson, R. C. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00032 · Cited by 62 · see citing works
“org February 2018 | Volume 6 | Article 32 to the choice of design, and assessment of the designs currently being used in D&I research is needed so that future implementation efforts may better account for these differences as well as the contextual factors and multiple levels involved in this field of study (25, 26).”
BackgroundA synthesis of implementation science frameworks and application to global health gaps
Dintrans, P. V., Bossert, T. J., Sherry, J., & Kruk, M. E. (2019) · Global Health Research and Policy · doi:10.1186/s41256-019-0115-1 · Cited by 70 · see citing works
Poland, F. M., Spalding, N. J., Gregory, S., McCulloch, J., Sargen, K., & Vicary, P. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013498 · Cited by 68 · see citing works
Partnering with health system operations leadership to develop a controlled implementation trial
Bauer, M. S., Miller, C., Kim, B., Lew, R., Weaver, K. R., Coldwell, C. M., Henderson, K., Holmes, S. K., Seibert, M. N., Stolzmann, K. L., et al. (2015) · Implementation Science · doi:10.1186/s13012-016-0385-7 · Cited by 62 · see citing works
“For the interpretive formative evaluation [45], we will code interviews using our grounded thematic analysis [57] coding, paying particular attention to factors that might be barriers or facilitators to future implementation efforts [62].”
BackgroundMethodLuck, J., Hagigi, F., Parker, L. E., Yano, E. M., Rubenstein, L. V., & Kirchner, J. E. (2009) · Implementation Science · doi:10.1186/1748-5908-4-64 · Cited by 66 · see citing works
“Formative evaluation activities [21], such as a systematic program of stakeholder interviews and measurement of program penetration and utilization, and spread to additional sites are ongoing.”
Developing a national dissemination plan for collaborative care for depression: QUERI Series
Smith, J. L., Williams, J. W., Owen, R. R., Rubenstein, L. V., & Chaney, E. F. (2008) · Implementation Science · doi:10.1186/1748-5908-3-59 · Cited by 66 · see citing works
“Formative evaluation of DSG activities includes implementation-focused, progress-focused, and interpretive components [47].”
BackgroundMethodInfluentialSelf-citationSelf-compassion in sport: a scoping review
Cormier, D. L., Kowalski, K. C., Ferguson, L. J., Mosewich, A. D., McHugh, T.-L., & Röthlin, P. (2023) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2022.2161064 · Cited by 71 · see citing works
“We highly recommend that researchers follow best practices for intervention development and implementation, including formative evaluation (Stetler et al., 2006), process evaluation (Saunders et al., 2005), and adherence to the TiDieR checklist (Template for Intervention Description and…”
Evaluations of Implementation at Early-Adopting Lung Cancer Screening Programs
Gesthalter, Y. B., Koppelman, E. A., Bolton, R. E., Slatore, C. G., Yoon, S. H., Cain, H. C., Tanner, N. T., Au, D. H., Clark, J. A., & Wiener, R. S. (2017) · CHEST Journal · doi:10.1016/j.chest.2017.02.012 · Cited by 56 · see citing works
Jani, N., Mathur, S., Kahabuka, C., Makyao, N., & Pilgrim, N. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0246717 · Cited by 59 · see citing works
“It has been established that formative evaluations that are designed to address anticipated facilitators and barriers can help policy makers and health care providers increase the possibility of successful implementation and uptake of interventions [26–28].”
BackgroundCampbell, C. I., Saxon, A. J., Boudreau, D. M., Wartko, P. D., Bobb, J. F., Lee, A. K., Matthews, A. G., McCormack, J., Liu, D. S., Addis, M., et al. (2021) · Addiction Science & Clinical Practice · doi:10.1186/s13722-021-00218-w · Cited by 58 · see citing works
Menec, V. H., Novek, S., VESELYUK, D. M., et al. (2013) · Journal of Aging & Social Policy · doi:10.1080/08959420.2014.854606 · Cited by 59 · see citing works
“Formative evaluation focuses on an assessment of the process of implementing a program (Dehar, Casswell, & Duignan, 1993; Schoster, Altpeter, Meier, & Callahan, 2012; Stetler et al., 2006).”
BackgroundMethodLeveraging implementation science to advance antibiotic stewardship practice and research
Livorsi, D. J., Drainoni, M., Reisinger, H. S., Nanda, N., McGregor, J. C., Barlam, T. F., Morris, A. M., & Szymczak, J. E. (2021) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2021.480 · Cited by 50 · see citing works
Richards, J. A., Bobb, J. F., Lee, A. K., Lapham, G. T., Williams, E. C., Glass, J. E., Ludman, E. J., Achtmeyer, C. E., Caldeiro, R. M., Oliver, M. M., et al. (2019) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2019.04.015 · Cited by 52 · see citing works
“other systems integrating drug screening may benefit from lessons learned during the pilot, we described barriers and facilitators identified in formative evaluation (Stetler et al., 2006) and adaptations made to the implementation strategies.”
BackgroundHagedorn, H., Kenny, M. E., Gordon, A. J., Ackland, P. E., Noorbaloochi, S., Yu, W., & Harris, A. H. S. (2018) · Addiction Science & Clinical Practice · doi:10.1186/s13722-018-0127-z · Cited by 54 · see citing works
Self-citationDefining and Addressing Gaps in Care for Obsessive-Compulsive Disorder in the United States
Senter, M. S., Patel, S. R., Dixon, L. B., Myers, R. W., & Simpson, H. B. (2021) · Psychiatric Services · doi:10.1176/appi.ps.202000296 · Cited by 54 · see citing works
The Effect of Care Pathways for Hip Fractures: A Systematic Review
Leigheb, F., Vanhaecht, K., Sermeus, W., Lodewijckx, C., Deneckere, S., Boonen, S., Boto, P., Mendes, R. V., & Panella, M. (2012) · Calcified Tissue International · doi:10.1007/s00223-012-9589-2 · Cited by 52 · see citing works
Brown, A. H., Cohen, A. N., Chinman, M. J., Kessler, C., & Young, A. S. (2008) · Implementation Science · doi:10.1186/1748-5908-3-9 · Cited by 53 · see citing works
“At the conclusion of the project, we will conduct semistructured interviews with the clinicians and clinic managers regarding the usefulness of the EQUIP-2 strategy, their satisfaction with the implementation process, barriers to and facilitators of implementation, and recommendations for future refinements [2].”
BackgroundMethodInfluentialRobben, S. H., Perry, M., Huisjes, M., van Nieuwenhuijzen, L., Schers, H., van Weel, C., Rikkert, M. G. M. O., van Achterberg, T., Heinen, M. M., & Melis, R. J. F. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-251 · Cited by 52 · see citing works
“First, because information concerning barriers and facilitators experienced during the implementation may be able to guide improvements to the implementation plan and the intervention itself [10,11].”
BackgroundApplication of participatory ergonomics to the redesign of the family-centred rounds process
Xie, A., Carayon, P., COX, E. D., Cartmill, R. S., Li, Y., Wetterneck, T. B., & Kelly, M. M. (2015) · Ergonomics · doi:10.1080/00140139.2015.1029534 · Cited by 50 · see citing works
“Finally, we performed a formative evaluation (Stetler et al. 2006) to assess and continuously improve the PE process (Andersen and Zebis 2014; Durlak and DuPre 2008; Griffin et al.”
MethodSobo, E. J., Bowman, C., & Gifford, A. L. (2008) · Social Science & Medicine · doi:10.1016/j.socscimed.2008.06.001 · Cited by 48 · see citing works
Self-citationDamschroder, L. J., Moin, T., Datta, S., Reardon, C. M., Steinle, N., Weinreb, J. E., Billington, C. J., Maciejewski, M. L., Yancy, W. S., Hughes, M. V., et al. (2015) · Implementation Science · doi:10.1186/s13012-015-0250-0 · Cited by 49 · see citing works
“This formative evaluation will incorporate the following features: 1) developmental evaluation to identify barriers and facilitators to implementation; 2) implementation and progress-focused evaluation to actively monitor implementation progress and adapt the program as needed; and 3) interpretive evaluation to provid…”
MethodPainter, P., Clark, L., & Olausson, J. M. (2014) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2014.01.433 · Cited by 50 · see citing works
Olmos‐Ochoa, T. T., Bharath, P., Ganz, D. A., et al. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-04967-y · Cited by 49 · see citing works
Mignogna, J., Martin, L. A., Harik, J., Hundt, N. E., Kauth, M., Naik, A. D., Sorocco, K., Benzer, J., & Cully, J. (2018) · Implementation Science · doi:10.1186/s13012-018-0768-z · Cited by 47 · see citing works
“Our work is limited in that we conducted a summative rather than formative evaluation [46] of providers’ perspectives on fidelity to the brief CBT intervention.”
MethodCompassionate End-of-Life Care: Mixed-Methods Multisite Evaluation of the 3 Wishes Project
Vanstone, M. G., Neville, T. H., Clarke, F. J., Swinton, M. E., Sadik, M., Takaoka, A., Smith, O. M., BAKER, A. J., LeBlanc, A., Foster, D., et al. (2019) · Annals of Internal Medicine · doi:10.7326/m19-2438 · Cited by 46 · see citing works
“Formative evaluation is a methodological technique used to assess development and implementation of a program early in its life cycle (6, 7).”
MethodMcNeely, J., Troxel, A. B., Kunins, H. V., Shelley, D., Lee, J. D., Walley, A. Y., Weinstein, Z., Billings, J., Davis, N. J., Marcello, R. K., et al. (2019) · Addiction Science & Clinical Practice · doi:10.1186/s13722-019-0135-7 · Cited by 42 · see citing works
Between and within-site variation in qualitative implementation research
Benzer, J. K., Beehler, S., Cramer, I. E., Mohr, D. C., Charns, M. P., & Burgess, J. F. (2013) · Implementation Science · doi:10.1186/1748-5908-8-4 · Cited by 46 · see citing works
“Formative evaluation was defined as an assessment of both potential and actual influences on implementation progress and effectiveness [33].”
BackgroundLukersmith, S., Fernández, A., Millington, M. J., & Salvador‐Carulla, L. (2015) · Disability and health journal · doi:10.1016/j.dhjo.2015.09.006 · Cited by 43 · see citing works
“A type of formative evaluation, the developmental evaluation, was undertaken to enhance the feasibility of the taxonomy.(46,47) Feasibility refers to the extent to which the use”
MethodEvaluating the Implementation of Home-Based Videoconferencing for Providing Mental Health Services
Interian, A., King, A. R., St. Hill, L. M., Robinson, C. H., & Damschroder, L. J. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201700004 · Cited by 41 · see citing works
“Responding to this need, we conducted a formative evaluation of V2H implementation for mental health services (13).”
MethodChui, M. A., Stone, J. A., & Holden, R. J. (2017) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2016.11.006 · Cited by 43 · see citing works
Gören, J. L., Rose, A., Engle, R., Smith, E., Christopher, M., Rickles, N. M., Semla, T., & McCullough, M. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201500506 · Cited by 30 · see citing works
Dobalian, A., Bowman, C. C., Wyte‐Lake, T., Pearson, M. L., Dougherty, M. B., & Needleman, J. (2014) · BMC Nursing · doi:10.1186/s12912-014-0036-8 · Cited by 45 · see citing works
“This formative evaluation [16] was designed to qualitatively analyze data collected using in-person interviews and focus groups from 364 key informants involved in implementing these 15 partnerships.”
MethodSelf-citationIntegrating Opioid Use Disorder Treatment Into Primary Care Settings
Austin, E. J., Chen, J. A., Briggs, E. S., Ferro, L., Barry, P., Heald, A., Merrill, J. O., Curran, G. M., Saxon, A. J., Fortney, J. C., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.28627 · Cited by 43 · see citing works
Ragheb, J. W., Norcott, A. E., Benn, L., Shah, N., McKinney, A., Min, L. C., & Vlisides, P. E. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09681-4 · Cited by 45 · see citing works
“Study Design and Overview This was a pre-implementation diagnostic evaluation study [16], which sought to identify challenges, barriers, and facilitators to delirium screening and management at a major university hospital.”
MethodHenderson, C., Jackson, C. T., Slade, M., Young, A. S., & Strauss, J. L. (2010) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0264-5 · Cited by 41 · see citing works
“…however, this study indicates a package of strategies similar to those used in the VA Quality Enhancement Research Initiative (QUERI) program (Stetler et al. 2006), to include system level changes to facilitate MHAD completion and access; training to improve provider knowledge and attitudes;…”
MethodGoetz, M. B., Bowman, C., Hoang, T., Anaya, H. D., Osborn, T. M., Gifford, A. L., & Asch, S. M. (2008) · Implementation Science · doi:10.1186/1748-5908-3-16 · Cited by 43 · see citing works
“In addition, we are conducting formative evaluations to further refine our program and assess the organizational factors that determine the generalizability [38].”
InfluentialSelf-citationA Systematic Review of Nursing Home Palliative Care Interventions: Characteristics and Outcomes
Carpenter, J. G., Lam, K., Ritter, A. Z., & Ersek, M. T. (2020) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2019.11.015 · Cited by 46 · see citing works
Fidelity monitoring in complex interventions: a case study of the WISE intervention
Swindle, T., Selig, J. P., Rutledge, J. M., Whiteside-Mansell, L., & Curran, G. M. (2018) · Archives of Public Health · doi:10.1186/s13690-018-0292-2 · Cited by 40 · see citing works
Swindle, T., Johnson, S. L., Whiteside-Mansell, L., & Curran, G. M. (2017) · Implementation Science · doi:10.1186/s13012-017-0624-6 · Cited by 37 · see citing works
“Further, we will use formative evaluation principles to assess the feasibility, acceptability, and fidelity of the enhanced WISE strategy during real-time implementation [35].”
MethodSorensen, J. L., & Kosten, T. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022765 · Cited by 40 · see citing works
Taylor, N. J., Best, S., Martyn, M. M., Long, J. C., North, K. N., Braithwaite, J., & Gaff, C. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024681 · Cited by 37 · see citing works
Beehler, G. P., Murphy, J. L., King, P. R., Dollar, K. M., Kearney, L. K., Haslam, A., Wade, M., & Goldstein, W. R. (2019) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000747 · Cited by 39 · see citing works
Ersek, M. T., Hickman, S. E., Thomas, A., et al. (2017) · The Gerontologist · doi:10.1093/geront/gnx155 · Cited by 38 · see citing works
“This analytic approach was appropriate because our interview questions and coding schema followed our specific aims, which were derived from Stetler and colleagues’ framework for implementation-focused and interpretive formative evaluation (Stetler et al., 2006).”
BackgroundMethodInfluentialValentine, S. E., Borba, C. P. C., Dixon, L., Vaewsorn, A. S., Gallegos-Guajardo, J., Resick, P. A., Stirman, S. W., & Marques, L. M. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22337 · Cited by 38 · see citing works
“In addition, the study uses formative evaluation (Stetler et al., 2006) to assess potential and actual influences on the success of our process of implementation of culturally adapted CPT.”
MethodHagedorn, H., Stetler, C. B., Bangerter, A. K., Noorbaloochi, S., Stitzer, M. L., & Kivlahan, D. R. (2014) · Addiction Science & Clinical Practice · doi:10.1186/1940-0640-9-12 · Cited by 35 · see citing works
“In contrast to process evaluation, as defined in this study as part of a Hybrid I design, formative evaluation [9] is used in a Hybrid II or III design to gather information for monitoring and actively enhancing implementation during the course of the implementationfocused aspect of the study.”
BackgroundMethodSelf-citationImplementation of the Tobacco Tactics Program in the Department of Veterans Affairs
Veterans Integrated Services Network (VISN) 11 Tobacco Tactics Team, Duffy, S. A., Karvonen-Gutierrez, C. A., Ewing, L. A., & Smith, P. M. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-009-1075-9 · Cited by 38 · see citing works
Care coordination for pregnant veterans: VA’s Maternity Care Coordinator Telephone Care Program
Cordasco, K. M., Katzburg, J. R., Katon, J. G., Zephyrin, L., Chrystal, J. G., & Yano, E. M. (2018) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx081 · Cited by 37 · see citing works
HIV patient and provider feedback on a telehealth collaborative care for depression intervention
Drummond, K. L., Painter, J. T., Curran, G. M., Stanley, R., Gifford, A. L., Rodriguez‐Barradas, M. C., Rimland, D., Monson, T. P., & Pyne, J. M. (2016) · AIDS Care · doi:10.1080/09540121.2016.1255704 · Cited by 33 · see citing works
“FE is a process of formally identifying potential or actual influences upon implementation (Stetler et al., 2006).”
Background
- Authors
- Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
- Affiliations
- University of Washington (US)
- VA Puget Sound Health Care System (US)
- VA San Diego Healthcare System (US)
- Edward Hines, Jr. VA Hospital (US)
- Minneapolis VA Medical Center (US)
- Baylor College of Medicine (US)
- Central Arkansas Veterans Healthcare System (US)
- Corresponding author
- Stetler, Cheryl B.
- ORCID
- Venue
- Journal of General Internal Medicine
- Published
- 2006-02
- Volume
- 21
- Issue
- S2
- Pages
- S1-S8
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC2557128
- OpenAlex
- W2084378034
- Cited by
- 642
- References
- 45
- Citations by year
- 2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
- FWCI
- 24.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Keywords
- formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
- MeSH terms
- Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-30.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-30.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review