CoChat

The role of formative evaluation in implementation research and the QUERI experience

Open access

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 2 of the 10 sources its claims cite were checked for support.

Overview

CoChat Summary

TL;DR

This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.

Key findings

  • Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
  • The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
  • QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
  • Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.

Methods

  • Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
  • Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
  • Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.

Limitations

  • Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
  • Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
  • Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11606-006-0267-9

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 0 Not valid · 7 Not checked · 60 Own results and interpretations

37 more claims

Create a free account to see all 68 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 13
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 61 references

Research works printed but not linked: 5

  1. 7. Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
  2. 35. Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
  3. 42. Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA prac- tice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1-8. Available at: http://www.hsrd.research.va.gov/publica- tions/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
  4. 53. La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
  5. 61. Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083-263.

Cited by

See all 642

Metadata

Authors
Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
Affiliations
  • University of Washington (US)
  • VA Puget Sound Health Care System (US)
  • VA San Diego Healthcare System (US)
  • Edward Hines, Jr. VA Hospital (US)
  • Minneapolis VA Medical Center (US)
  • Baylor College of Medicine (US)
  • Central Arkansas Veterans Healthcare System (US)
Corresponding author
Stetler, Cheryl B.
Venue
Journal of General Internal Medicine
Published
2006-02
Volume
21
Issue
S2
Pages
S1-S8
ISSN
  • 0884-8734 (ISSN-L)
  • 1525-1497
OpenAlex
W2084378034
Cited by
642
References
45
Citations by year
2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
FWCI
24.45
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (tdm)
Locations
  • Journal of General Internal Medicine (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • U.S. Department of Veterans Affairs
  • Health Services Research and Development
Topics
  • Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
Keywords
formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
MeSH terms
Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The role of formative evaluation in implementation research and the QUERI experience. https://app.cochat.ai/papers/10.1007/s11606-006-0267-9
The paper
Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine. https://doi.org/10.1007/s11606-006-0267-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.