Six practical recommendations for improved implementation outcomes reporting
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Background: Implementation outcomes research spans an exciting mix of fields, disciplines, and geographical space. Although the number of studies that cite the 2011 taxonomy has expanded considerably, the problem of harmony in describing outcomes persists. This paper revisits that problem by focusing on the clarity of reporting outcomes in studies that examine them. Published recommendations for improved reporting and specification have proven to be an important step in enhancing the rigor of implementation research. We articulate reporting problems in the current implementation outcomes literature and describe six practical recommendations that address them. Recommendations: Our first recommendation is to clearly state each implementation outcome and provide a definition that the study will consistently use. This includes providing an explanation if using the taxonomy in a new way or merging terms. Our second recommendation is to specify how each implementation outcome will be analyzed relative to other constructs. Our third recommendation is to specify “the thing” that each implementation outcome will be measured in relation to. This is especially important if you are concurrently studying interventions and strategies, or if you are studying interventions and strategies that have multiple components. Our fourth recommendation is to report who will provide data and the level at which data will be collected for each implementation outcome, and to report what kind of data will be collected and used to assess each implementation outcome. Our fifth recommendation is to state the number of time points and frequency at which each outcome will be measured. Our sixth recommendation is to state the unit of observation and the level of analysis for each implementation outcome. Conclusion: This paper advances implementation outcomes research in two ways. First, we illustrate elements of the 2011 research agenda with concrete examples drawn from a wide swath of current literature. Second, we provide six pragmatic recommendations for improved reporting. These recommendations are accompanied by an audit worksheet and a list of exemplar articles that researchers can use when designing, conducting, and assessing implementation outcomes studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“the updated CFIR guides users to define the innovation (aka “the thing” [20, 25] being implemented), including the boundary between the innovation and implementation strategies.”
From The updated consolidated framework for implementation research based on user feedback (2022) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“When juxtaposed with Proctor et al.’s 2011 recommendations and a recent paper on recommendations for reporting implementation outcomes [67], our data provide a basis for projecting priorities for a “next stage agenda” on implementation outcomes 2022–2023.”
From Ten years of implementation outcomes research: a scoping review (2023) · cites this paper as [67]
In the citing paper
“Similarly, because implementation outcomes were often inconsistently defined and poorly reported [28–30], we broadened our scope to include studies that identified team constructs as implementation determinants (i.e., barriers/facilitators) without explicitly defining and measuring an implementation outcome.”
From Teamwork and implementation of innovations in healthcare and human service settings: a systematic review (2024) · cites this paper as [28]
In the citing paper
“There was also poor measurement and reporting of implementation outcomes in many studies, consistent with a recent review of research on implementation outcomes [28, 29].”
From Teamwork and implementation of innovations in healthcare and human service settings: a systematic review (2024) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
Self-citationSPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363 · cited 1×
“This contrasts with more narrowly focused reporting guidelines, like the Consolidated Standards of Reporting Trials for randomized control trials [16–18] and the Standard Protocol Items: Recommendations for Interventional Trials for clinical trial protocols [19].”
MethodKlein, K. J., & Kozlowski, S. W. J. (2000) · Cited by 4,914
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058
Boutron, I., Altman, D. G., Moher, D., Schulz, K. F., Ravaud, P., & for the CONSORT NPT Group* (2017) · Annals of Internal Medicine · doi:10.7326/m17-0046 · Cited by 1,538
Standards for Reporting Implementation Studies (StaRI) Statement
Pinnock, H., Barwick, M., Carpenter, C. R., Eldridge, S., Grandes, G., Griffiths, C. J., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., et al. (2017) · BMJ · doi:10.1136/bmj.i6795 · Cited by 1,446 · cited 3×
“The StaRI is intentionally open to the range of research designs and methods that implementation researchers use [14, 15].”
BackgroundMethodDesign and analysis of stepped wedge cluster randomized trials
Hussey, M. A., & Hughes, J. P. (2007) · Contemporary Clinical Trials · doi:10.1016/j.cct.2006.05.007 · Cited by 1,393
Dissemination and Implementation Research in HealthTranslating Science to Practice
Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012) · doi:10.1093/acprof:oso/9780199751877.001.0001 · Cited by 1,355
Self-citationSystematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223 · cited 1×
“Reporting the phase [27] or stage [28] can also help to clarify when during the implementation lifecycle outcomes are observed or are most salient.”
Background
Cited by
The updated consolidated framework for implementation research based on user feedback
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022) · Implementation Science · doi:10.1186/s13012-022-01245-0 · Cited by 4,452
“Distinguish the innovation (the “thing” that continues when implementation is complete) [20, 25] from the implementation process and strategies used to implement the innovation [26, 27] (activities that end after implementation is complete) [28].”
BackgroundMethodTen years of implementation outcomes research: a scoping review
Proctor, E. K., Bunger, A. C., Lengnick-Hall, R., Gerke, D. R., Martin, J. K., Phillips, R. J., & Swanson, J. C. (2023) · Implementation Science · doi:10.1186/s13012-023-01286-z · Cited by 424
“See Lengnick-Hall et al. [67] for detail on the coding challenges we encountered, along with recommendations to improve reporting.”
BackgroundSelf-citationThe implementation of interprofessional education: a scoping review
Bogossian, F., New, K., George, K., Barr, N., Dodd, N., Hamilton, A. L., Nash, G., Masters, N., Pelly, F., Reid, C., et al. (2023) · Advances in Health Sciences Education · doi:10.1007/s10459-022-10128-4 · Cited by 155
Ashcraft, L. E., Goodrich, D. E., Hero, J., Phares, A., Bachrach, R. L., Quinn, D. A., Qureshi, N., Ernecoff, N. C., Lederer, L. G., Scheunemann, L. P., et al. (2024) · Implementation Science · doi:10.1186/s13012-024-01369-5 · Cited by 89
“Other tracking approaches may leverage technology (e.g., cell phones, tablets, EMR templates) [78, 97] or find novel, pragmatic approaches to collect recommended strategy specifications over time (e.g.., dose, deliverer, and mechanism) [1, 9, 27, 98, 99].”
BackgroundMcGuier, E. A., Kolko, D. J., Aarons, G. A., Schachter, A., Klem, M. L., Diabes, M. A., Weingart, L. R., Salas, E., & Wolk, C. B. (2024) · Implementation Science · doi:10.1186/s13012-024-01381-9 · Cited by 70
Metadata
- Authors
- Lengnick-Hall, R., Gerke, D. R., Proctor, E. K., Bunger, A. C., Phillips, R. J., Martin, J. K., & Swanson, J. C.
- Affiliations
- University of Denver (US)
- Washington University in St. Louis (US)
- The Ohio State University (US)
- Venue
- Implementation Science
- Published
- 2022-02-08
- Volume
- 17
- Issue
- 1
- Pages
- 16-16
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC8822722
- OpenAlex
- W4210914141
- Cited by
- 86
- References
- 31
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 40 · 2023: 15 · 2022: 4
- FWCI
- 22.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Implementation Science (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institute of Mental Health (T32MH019960, R25 MH080916–08)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- implementation outcomes, reporting quality, outcome definitions, outcome measurement, unit of analysis
- MeSH terms
- Humans, Outcome Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Six practical recommendations for improved implementation outcomes reporting. https://app.cochat.ai/papers/10.1186/s13012-021-01183-3
- The paper
- Lengnick-Hall, R., Gerke, D. R., Proctor, E. K., Bunger, A. C., Phillips, R. J., Martin, J. K., & Swanson, J. C. (2022). Six practical recommendations for improved implementation outcomes reporting. Implementation Science. https://doi.org/10.1186/s13012-021-01183-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.