The role of formative evaluation in implementation research and the QUERI experience
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 2 of the 10 sources its claims cite were checked for support.
Overview
CoChat Summary
TL;DR
This article argues that formative evaluation is essential alongside summative evaluation in implementation and quality improvement research, and it outlines four stages: developmental, implementation-focused, progress-focused, and interpretive. It illustrates these stages with Veterans Health Administration QUERI projects and discusses methodological challenges.
Key findings
- Summative evaluation alone cannot answer key implementation questions about feasibility, fidelity, context, participant response, and needed adaptations; formative evaluation fills this gap.
- The authors define four progressive stages: developmental (diagnostic), implementation-focused (discrepancies and barriers), progress-focused (monitoring impacts and indicators), and interpretive (explaining results and worth).
- QUERI examples show formative evaluation in use: SCI vaccine reminder work found local installation and use varied widely, leading to general principles rather than standard instructions; CHF weekly calls uncovered a Prosthetics policy barrier to scales and enabled policy change; a SUD toolkit was distributed to 8 pilot clinics with monthly contact; an HIV/AIDS clinical reminder study identified 6 barriers and 17 prioritized recommendations for 9 reminders; SCI attitude data were collected over 3 years.
- Formative evaluation can help avoid Type III errors, support in-course adjustments, and generate hypotheses for future testing, though it does not itself test hypotheses.
Methods
- Design: conceptual and methodological article reviewing limitations of traditional implementation research and presenting a rationale for formative evaluation; no primary trial is reported.
- Data or sample: illustrative examples from VHA QUERI projects, including SCI, CHF, SUD, mental health, HIV/AIDS, and ischemic heart disease, plus references to prior literature; QUERI began in 1998.
- Analysis: descriptive synthesis and case illustration of the four formative evaluation stages, with discussion of measurement issues and challenges; formative evaluation is described as using mixed qualitative and quantitative methods.
Limitations
- Stated: formative evaluation is new to health services research, can be time-consuming, and may require convincing project leaders; challenges include deciding when modifications change the intervention, setting limits on local adaptation, measuring flexibility consistently, and determining how formative evaluation activities influence results.
- Stated: there is no single definition or approach to formative evaluation, with variation in terminology and scope.
- Evident: the article is conceptual and relies on selected QUERI examples rather than a systematic review or controlled comparison, so it does not quantify formative evaluation’s independent effect on implementation outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 7 Not checked · 60 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“However, within an experimental study, in-depth data from a concurrent FE can provide working hypotheses to explain successes or failures, particularly when the implementation and evaluation plans are grounded in a conceptual framework.[55][56][57]”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Interpretive Evaluation
What the sources say
- Not checked yetcites [55] 55. Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology implementation framework in behavioral sci- ence: a systematic analysis of the literature on IT use. J Biomed Inform. 2003;36:218-27. source
- Not checked yetcites [56] 56. Sanson-Fisher RW, Grimshaw JM, Eccles MP. The science of changing providers' behaviour: the missing link in evidence-based practice. Med J Aust. 2004;180:205-6. source
- Could not verifycites [57] 57. Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME-PRocess modelling in ImpleMEntation research: selecting a the- oretical basis for interventions to change clinical practice. BMC Health Serv Res. 2003;3:22. sourceExperimental — accuracy not yet measured
Our copy of PRIME (Walker et al. 2003) is a process-modelling study protocol: its Background (Implementation research) argues for establishing a scientific/theoretical rationale for interventions, notes the two approaches needed to identify 'active ingredients' (process modelling and 'develop an understanding of how the interventions themselves work'), and cites the UK MRC framework for developing and evaluating complex interventions; 'Work leading up to this study' describes the third-molar RCT's post-intervention theory-of-planned-behaviour questionnaire. Nothing in the passages returned (searched: formative evaluation, concurrent/process evaluation, working hypotheses, theory-based evaluation of trials, conceptual framework grounding implementation and evaluation plans) states that in-depth data from a concurrent formative evaluation within an experimental study supplies working hypotheses explaining implementation successes or failures. The claim's point is therefore not addressed by our copy, in either direction. (LLM, conf=0.70)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the ATIP project, FE demonstrated a correlation between the amount of external facilitation and improvement in antipsychotic dosing, the extent to which lack of clinician time inhibits dissemination and QI activities, and the acceptability and utility of MH QUERI tools/products.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interview responses indicated that planning, including identification of resources and potential barriers and facilitators, was a critical and universally underutilized step in the intervention process.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > CHALLENGES OF CONDUCTING FE
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these data it was clear that the current version of the reminder was installed at all locations, but that local methods of installing and using the CCR varied widely.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > Implementation-Focused Evaluation
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the depression project, FE has generated information about factors influencing caseload capacity for nurse ''care managers'' who maintain regular contact with depressed patients to monitor treatment adherence, assess symptomatic improvement, provide patient education, and facilitate communication among patients, primary care clinicians, and mental health specialists.”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
37 more claims
Create a free account to see all 68 claims and to verify the 8 references still open.
- Probably not a substantial claim. 13
- It may state two separate claims as one. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In addition, it has been suggested that current conceptions of research designs fail to “maximize clinical utility for practicing clinicians and other decision makers”18; for example, through a failure to focus on external validity or implementation-related barriers and facilitators to routine use and sustainability of “effective” practices.4,21,22”
From Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact (2012) · cites this paper as [21]
The cited source says what the claim attributes to it.
In the citing paper
“However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [14–16, 19, 20].”
From Guidance for conducting feasibility and pilot studies for implementation trials (2020) · cites this paper as [20]
In the citing paper
“The ability to learn from implementation studies would be improved by greater attention on theoretically driven formative and/or process evaluations to help determine how interventions work in different contexts, and to promote the transferability of findings to other settings [10,26].”
From The role of evidence, context, and facilitation in an implementation trial: implications for the development of the PARIHS framework (2013) · cites this paper as [10]
In the citing paper
“Stetler et al. [71] provide four progressive yet integrated stages of formative evaluation.”
From Ten recommendations for using implementation frameworks in research and practice (2020) · cites this paper as [71]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“Mental health (MH) QUERI”
§ The Role of Formative Evaluation in Implementation Research and the QUERI Experience > QUERI, quality enhancement research initiative; FE, formative evaluation; SCI, spinal cord injury.
“NH, mental health”
Citations
Cites
(1984) · Cited by 81,983
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,535
Rogers, E. M., Singhal, A., & Quinlan, M. M. (2019) · An Integrated Approach to Communication Theory and Research · doi:10.4324/9780203710753-35 · Cited by 37,322
(1963) · Medical Entomology and Zoology · Cited by 15,085
(1979) · Medical Entomology and Zoology · Cited by 11,252
Gluesing, J. C. (2012) · Leadership in Science and Technology: A Reference Handbook · doi:10.4135/9781412994231.n14 · Cited by 3,210
Evaluation: A Systematic Approach
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (1979) · Cited by 2,943
Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003) · American Journal of Public Health · doi:10.2105/ajph.93.8.1261 · Cited by 1,836
Hinkemeyer, M. T. (1971) · The Clearing House: A Journal of Educational Strategies, Issues and Ideas · doi:10.1080/00098655.1971.11478291 · Cited by 1,835
Utilization-focused evaluation : the new century text
Patton, M. Q. (1997) · Cited by 1,469
Research works printed but not linked: 5
- 7. Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
- 35. Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
- 42. Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA prac- tice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1-8. Available at: http://www.hsrd.research.va.gov/publica- tions/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
- 53. La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
- 61. Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083-263.
Cited by
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496
“[3] as diagnostic analysis, implementationand progress-focused evaluations, and interpretive evaluation).”
BackgroundMethodCurran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058
Self-citationClinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-008-0197-4 · Cited by 1,894
“can address the fundamental issue of public health impact (Stetler et al. 2006).”
BackgroundAn introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L., Hagedorn, H., Smith, J., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 1,611
“These studies can involve one or more of three broad types of evaluation, process evaluation, formative evaluation, and summative evaluation [13, 31].”
BackgroundSelf-citation
Metadata
- Authors
- Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L.
- Affiliations
- University of Washington (US)
- VA Puget Sound Health Care System (US)
- VA San Diego Healthcare System (US)
- Edward Hines, Jr. VA Hospital (US)
- Minneapolis VA Medical Center (US)
- Baylor College of Medicine (US)
- Central Arkansas Veterans Healthcare System (US)
- Corresponding author
- Stetler, Cheryl B.
- ORCID
- Venue
- Journal of General Internal Medicine
- Published
- 2006-02
- Volume
- 21
- Issue
- S2
- Pages
- S1-S8
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC2557128
- OpenAlex
- W2084378034
- Cited by
- 642
- References
- 45
- Citations by year
- 2026: 38 · 2025: 41 · 2024: 61 · 2023: 61 · 2022: 50 · 2021: 54 · 2020: 53 · 2019: 43 · 2018: 34 · 2017: 29 · 2016: 21 · 2015: 27 · 2014: 28 · 2013: 23 · 2012: 20
- FWCI
- 24.45
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Health Services Research and Development
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- formative assessment, implementation research, translating research into practice, Veterans Health Administration, developmental evaluation, interpretive evaluation
- MeSH terms
- Health Services Research, Hospitals, Veterans, Humans, United States, United States Department of Veterans Affairs, Outcome Assessment, Health Care, Total Quality Management, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The role of formative evaluation in implementation research and the QUERI experience. https://app.cochat.ai/papers/10.1007/s11606-006-0267-9
- The paper
- Stetler, C. B., Legro, M. W., Wallace, C. M., Bowman, C., Guihan, M., Hagedorn, H., Kimmel, B., Sharp, N. D., & Smith, J. L. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine. https://doi.org/10.1007/s11606-006-0267-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.