Cited by 642Health Policy Implementation ScienceHealth Sciences Research and EducationEvaluation and Performance Assessment

The role of formative evaluation in implementation research and the QUERI experience

Open access
Stetler, Cheryl B., Legro, Marcia W., Wallace, Carolyn M., Bowman, Candice, Guihan, Marylou, Hagedorn, Hildi, Kimmel, Barbara, Sharp, Nancy D. +1 more·2006·Journal of General Internal Medicine

AI snapshot

Written from the paper’s full text.

TL;DR

This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.

Key findings

  • Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
  • The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
  • QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
  • Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.

Methods

  • Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
  • Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
  • Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.

Limitations

  • Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
  • Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
  • Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11606-006-0267-9

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 10 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the ATIP project, formative evaluation showed a correlation between the amount of external facilitation and improvement in antipsychotic dosing.

    “In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Interview responses showed that planning, including identifying resources and potential barriers and facilitators, was a critical and universally underused step in the intervention process.

    “Interview responses indicated that planning, including identification of resources and potential barriers and facilitators, was a critical and universally underutilized step in the intervention process.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the SCI QUERI respiratory vaccine initiative, sites differed greatly in how they installed and used the computerized clinical reminder.

    “From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the depression project, formative evaluation generated information about factors influencing caseload capacity for nurse care managers.

    “In the depression project, FE has generated information about factors influencing caseload capacity for nurse ''care managers'' who maintain regular contact with depressed patients to monitor treatment adherence, assess symptomatic improvement, provide patient education, and facilitate communication among patients, primary care clinicians, and mental health specialists.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In the CHF QUERI, weekly teleconferences and solving problems shifted clinical staff's view that the research was disconnected from daily work.

    “Value: the weekly teleconferencing and problem resolutions changed the perceptions of clinical staff that this research was not related to day-to-day activities.”
    Quote in the paperSubstantial claim
  • May combine two claims. 16
  • Classified as probably not a substantial claim. 13
  • May combine two claims.

    Findingreasoning

    In the QUERI example, formative evaluation quickly spotted stalled progress, indicating that more facilitation or new tools were needed to keep enthusiasm for the intervention.

    “Formative evaluation allowed quick identification of stagnation of progress that suggested the need for increased facilitation or introduction of new tools to sustain enthusiasm for the intervention.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Seventeen prioritized short- and long-term recommendations were produced to improve the usefulness and usability of the nine reminders.

    “Seventeen prioritized short-and long-term recommendations were generated to improve the usefulness and usability of the 9 reminders.59”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9. source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    IQS formative evaluation was used to refine the intervention somewhat during the project and to a major extent for the next nationwide rollout phase.

    “IQS FE was also used to refine the intervention, to some extent, during the project, and to a major extent for the next ''nationwide rollout'' phase.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the QUERI example, prototype toolkits for quality improvement and for monitoring clinic practices and patient outcomes went to 8 pilot clinics.

    “Prototype toolkits for implementing QI and monitoring clinic practices and patient outcomes were distributed to 8 pilot clinics.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Formative evaluation data were useful for accurately telling the story of individual clinics and illustrating differences between clinics that succeeded with implementation and those that did not.

    “Value: while the FE provided some suggestions about how to modify the intervention and tools, the data were also useful to accurately ''tell the story'' of the individual clinics and to illustrate differences between clinics that were successful with implementation versus those that were not.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Lower performing IQS sites were slow to start applying key concepts and had not completed a full PDSA cycle well into an action period.

    “In contrast, we were able to see that lower performing IQS sites were slow to begin applying key concepts and had not completed an entire PDSA cycle well into an action period.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    In the QUERI example, the original toolkit was changed to make it more useful.

    “Value: i.The original toolkit was modified to improve its usefulness.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    In the CHF QUERI, the calls modeled problem-solving for clinical staff and gave the research team information that allowed original project tools to be made more user-friendly.

    “The calls also provided a model of problem-solving behaviors for the clinical staff (e.g.,Plan, Do, Study, Act) as well as information for the research team that enabled ''user friendly'' modifications of original project tools.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    As the project progressed, some sites were very successful in applying the treatment plan while others stagnated.

    “As the project moved forward, it was clear that some sites were very successful in applying the treatment plan while others stagnated.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Six barriers to effective use of the clinical reminder intervention were identified: workload, lack of time to follow up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.

    “We identified 6 barriers to the effective use of the CR intervention, including workload, lack of time to follow-up, inapplicability, limited training, interruption of patient-provider face time, and use of paper for physician orders at some sites.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Formative evaluation offers methods to obtain such information and to overcome limitations found in early implementation and quality improvement studies.

    “Formative evaluation provides techniques for obtaining such information and for overcoming limitations identified in early implementation/QI studies.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    Traditional intervention studies may fail to record the detailed and complex nature of what is delivered, and so may overlook both the actual character of interventions and key organizational factors needed for replication.

    “As Stead et al. 10,11 suggest, traditional intervention research can fail to ''capture the detail and complexity of intervention inputs and tactics'' (10, p. 354), thereby missing the true nature of interventions as well as significant organizational factors important for replication.10,11”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Stead et al. 10,11

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Implementation researchers require evaluative information beyond clinical impact and beyond whether an adoption strategy worked, including feasibility of strategies, real-time implementation, contextual factors, participant response, and necessary adaptations.

    “Implementation researchers need to answer critical questions about the feasibility of implementation strategies, degree of real-time implementation, status and potential influence of contextual factors, response of project participants, and any adaptations necessary to achieve optimal change.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Formative evaluation can save time and reduce frustration because its data reveal factors that hinder clinicians from implementing best practices.

    “Formative evaluation can save time and frustration as data highlight factors that impede the ability of clinicians to implement best practices.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Formative evaluation can make the realities and black-box nature of implementation more transparent to decision makers.

    “it can make the realities and black box nature of implementation more transparent to decision makers;”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Interpretive formative evaluation draws on the results from every other formative evaluation stage.

    “Interpretive FE uses the results of all other FE stages.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    Formative data collected during implementation help keep strategies on course and improve the chance of producing change by addressing barriers that can be acted on, strengthening known levers of change, and refining parts of the implementation intervention.

    “Implementation-focused formative data also are used to keep the strategies on track and as a result optimize the likelihood of affecting change by resolving actionable barriers, enhancing identified levers of change, and refining components of the implemen-tation interventions.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    At the implementation stage, formative evaluation data have multiple benefits: they reveal differences between planned and actual interventions, support implementation by finding barriers that can be changed, allow needed adjustments to the original intervention, aid interpretation of project outcomes, and provide crucial details and guidance for reproducing results elsewhere.

    “In summary, FE data collected at this stage offer several advantages.They can (a) highlight actual versus planned interventions, (b) enable implementation through identification of modifiable barriers, (c) facilitate any needed refinements in the original implementation intervention, (d) enhance interpretation of project results, and (e) identify critical details and guidance necessary for re”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    Data from developmental evaluation help researchers identify potential problems and, when possible, address them before interventions start at study sites.

    “Such developmental data enable researchers to understand potential problems and, where possible, overcome them prior to initiation of interventions in study sites.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    At the end of a project, interpretive information about key stakeholders' experiences can be gathered.

    “In addition, interpretive information can be collected at the end of the project about key stakeholder experiences.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Progress-focused formative evaluation is proactive: data on progress serve as feedback about how far the project has moved toward intended outcomes.

    “The proactive nature of FE is emphasized, as progress data become feedback about the degree of movement toward desired outcomes.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    Some evaluation activities can contribute to more than one formative evaluation stage, and in practice the stages blend together.

    “However, as indicated in some of the examples, various evaluative activities can serve multiple stages, which then merge in practice.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Formative evaluation helps address many challenges to effective implementation and its scientific study, thereby aiding integration of research findings into practice and improvement of patient care.

    “Formative evaluation helps to meet the many challenges to effective implementation and its scientific study, thereby facilitating integration of research findings into practice and improvement of patient care.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Formative evaluation is often essential to the success, interpretation, and replication of implementation and quality improvement project results.

    “Formative evaluation is a study approach that is often key to the success, interpretation, and replication of the results of implementation/QI projects.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Premisereasoning

    Within QUERI, formative evaluation's emphasis on action sets it apart from process evaluations whose data are not meant to be used concurrently.

    “In QUERI, this action focus differentiates FE from ''process'' evaluations where data are not intended for concurrent use.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationnone given

    Interpretive formative evaluation data offer alternative explanations for findings, clarify what success in implementation means, and deepen understanding of an implementation strategy's impact or value.

    “Specifically, FE data provide alternative explanations for results, help to clarify the meaning of success in implementation, and enhance understanding of an implementation strategy's impact or ''worth.''”
    Quote in the paperSubstantial claim
26 of 26
  • The paper relies on this without support.

    Implementation work tends to be disorderly and complicated.

    “As implementation efforts can be a relatively messy and complex process”
  • The paper relies on this without support.

    Factors identified in a general way are likely to differ from one implementation site to another.

    “Even if available, a diagnostic analysis is suggested given the likelihood that generically identified factors will vary across implementation sites.”
  • The paper relies on this without support.

    In well-controlled randomized controlled trials, researchers do not typically modify an experimental intervention once it has been approved.

    “In the well-controlled RCT, researchers do not typically modify an experimental intervention once approved.”
  • The paper relies on this without support.

    Formative evaluation is a new concept when applied to health services research.

    “Formative evaluation is a new concept as applied to health services research and as such presents multiple challenges.”
  • The paper relies on this without support.

    Formative evaluation is a time-consuming activity.

    “Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
  • The paper relies on this without support.

    Some researchers need help understanding how formative evaluation can be incorporated into a study design.

    “Some researchers may need help in understanding how FE can be incorporated into a study design.”
  • The paper relies on this without support.

    Project leaders may need to be convinced of formative evaluation's utility before committing study resources.

    “Formative evaluation is also a time-consuming activity and project leaders may need to be convinced of its utility before committing study resources.”
  • The paper relies on this without support.

    Implementation and quality improvement researchers may encounter the erroneous view that formative evaluation involves only qualitative research or is not rigorous.

    “Implementation and QI researchers may encounter the erroneous view that FE involves only qualitative research or that it is not rigorous, e.g., that it consists of ''just talking to a few people''.”
  • The paper relies on this without support.

    Inferential statistical significance does not necessarily equate with clinical significance.

    “However, as inferential statistical significance does not necessarily equate with clinical significance, it is useful to obtain perceptions of stakeholders relative to the ''meaning'' of statistical findings.”
  • The paper relies on this without support.

    Research evidence indicates that people with negative attitudes about influenza vaccine safety, effectiveness, or utility are less likely to seek it annually.

    “Implementation goal: research evidence indicates that persons who have negative attitudes about influenza vaccine (safety, effectiveness, utility) are less likely to seek it out annually.”
  • The paper relies on this without support.

    Formative evaluation, as a descriptive assessment activity, does not by itself test hypotheses.

    “Formative evaluation, as a descriptive assessment activity, does not per se test hypotheses.”
  • The paper relies on this without support.

    Formative evaluation does not lack rigor.

    “However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
  • The paper relies on this without support.

    Formative evaluation is not simply a matter of qualitative research or a specific qualitative methodology.

    “However, FE does not lack rigor nor is it simply a matter of qualitative research or a specific qualitative methodology.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current conceptions of research designs fail to maximize clinical utility for practicing clinicians and other decision makers, for example through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of effective practices.

    “In addition, it has been suggested that current conceptions of research designs fail to “maximize clinical utility for practicing clinicians and other decision makers”18; for example, through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of “effective” practices.4,21,22”
  • The field of implementation science has identified the need for nuanced guidance because of differences between implementation trials and conventional clinical or public health efficacy trials.

    “However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [14–16, 19, 20].”
  • The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings.

    “The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
  • Stetler et al. provide four progressive yet integrated stages of formative evaluation.

    “Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting abbreviation for mental health in Table 8

    Minor

    The table introduces MH as the abbreviation for mental health, but its abbreviation key instead assigns that meaning to NH.

1 of 1

Cites

Flagged first, then by citations

1 of the 45 references couldn’t be matched to a known paper.

Research works printed but not linked: 12

  1. Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
  2. Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.
  3. Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.
  4. Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.
  5. Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.
  6. Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
  7. Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.
  8. Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.
  9. Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
  10. LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.
  11. La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
  12. Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.

Cited by

Sorted by citations

Authors
Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
Affiliations
  • University of Washington (US)
  • VA Puget Sound Health Care System (US)
  • VA San Diego Healthcare System (US)
  • Edward Hines, Jr. VA Hospital (US)
  • Minneapolis VA Medical Center (US)
  • Baylor College of Medicine (US)
  • Central Arkansas Veterans Healthcare System (US)
Corresponding author
Stetler, Cheryl B.
Venue
Journal of General Internal Medicine
Published
2006-02
Volume
21
Issue
S2
Pages
S1-S8
ISSN
  • 0884-8734 (ISSN-L)
  • 1525-1497
OpenAlex
W2084378034
Cited by
642
References
45
Citations by year
2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
FWCI
24.45
Type
journal article
Language
en
Open access
Locations
  • Journal of General Internal Medicine (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • U.S. Department of Veterans Affairs
  • Health Services Research and Development
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
Keywords
formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
MeSH terms
Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-30.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review