Development of a framework and coding system for modifications and adaptations of evidence-based interventions
Open accessThe DOI matches its Crossref record; 59 of the 60 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Evidence-based interventions are frequently modified or adapted during the implementation process. Changes may be made to protocols to meet the needs of the target population or address differences between the context in which the intervention was originally designed and the one into which it is implemented [Addict Behav 2011, 36(6):630-635]. However, whether modification compromises or enhances the desired benefits of the intervention is not well understood. A challenge to understanding the impact of specific types of modifications is a lack of attention to characterizing the different types of changes that may occur. A system for classifying the types of modifications that are made when interventions and programs are implemented can facilitate efforts to understand the nature of modifications that are made in particular contexts as well as the impact of these modifications on outcomes of interest. METHODS: We developed a system for classifying modifications made to interventions and programs across a variety of fields and settings. We then coded 258 modifications identified in 32 published articles that described interventions implemented in routine care or community settings. RESULTS: We identified modifications made to the content of interventions, as well as to the context in which interventions are delivered. We identified 12 different types of content modifications, and our coding scheme also included ratings for the level at which these modifications were made (ranging from the individual patient level up to a hospital network or community). We identified five types of contextual modifications (changes to the format, setting, or patient population that do not in and of themselves alter the actual content of the intervention). We also developed codes to indicate who made the modifications and identified a smaller subset of modifications made to the ways that training or evaluations occur when evidence-based interventions are implemented. Rater agreement analyses indicated that the coding scheme can be used to reliably classify modifications described in research articles without overly burdensome training. CONCLUSIONS: This coding system can complement research on fidelity and may advance research with the goal of understanding the impact of modifications made when evidence-based interventions are implemented. Such findings can further inform efforts to implement such interventions while preserving desired levels of program or intervention effectiveness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which commonly involve a series of non-systematic adaptations (Stirman, Miller, Toder, & Calloway, 2013).”
From User‐centered design for psychosocial intervention development and implementation. (2016) · cites this paper as Stirman, Miller, Toder, & Calloway, 2013
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reported influences on sustainability include characteristics of the innovation and the individuals involved, leadership, capacity, processes, and context [2, 9, 13, 14].”
From Factors influencing sustainability and scale-up of rural primary healthcare memory clinics: perspectives of clinic team members (2022) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, while the 2013 and 2017 frameworks [15, 16] laid out distinct forms of modification that could be made to the content or mode of delivery, they did not specify when those modifications were designed to address important differences between the original population and the stakeholders in the current implementation effort (i.e., cultural adaptations).”
From The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions (2019) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“More work is needed to determine what types of adaptation may specifically enhance sustainability and what types may erode the benefits or delivery of the intervention over time (135, 137).”
From The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care (2018) · cites this paper as [137]
The cited source says what the claim attributes to it.
In the citing paper
“However, new approaches to identifying and coding EBI modifications and adaptations are promising in regard to rigorous study of adaptations and their impacts [63].”
From “Scaling-out” evidence-based interventions to new populations or new health care delivery systems (2017) · cites this paper as [63]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 2×
“stantial agreement’ and ‘almost perfect’ agreement for the level and nature of modifications, respectively [52].”
BackgroundThe Discovery of Grounded Theory: Strategies for Qualitative Research.
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,361
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
(1998) · Sage eBooks · Cited by 38,631
(1990) · Cited by 11,110
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1×
“The two raters overlapped on 20% of the identified segments, and we computed Cohen’s kappa coefficients [31] to determine interrater reliability for each rating category.”
MethodDusenbury, L. (2003) · Health Education Research · doi:10.1093/her/18.2.237 · Cited by 1,671 · cited 1×
“cessary [4,6-10], few studies have examined the impact”
BackgroundWiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355 · cited 3×
“The tension between modification and fidelity is a critical issue in implementation science [4,54,55].”
BackgroundSelf-citationThe Cultural Adaptation of Prevention Interventions: Resolving Tensions Between Fidelity and Fit
Castro, F. G., Barrera, M., & Martinez, C. R. (2004) · Prevention Science · doi:10.1023/b:prev.0000013980.12412.cd · Cited by 1,228 · cited 4×
“terms of their complexity and comprehensiveness [27].”
BackgroundInfluentialShediac-Rizkallah, M. C., & Bone, L. R. (1998) · Health Education Research · doi:10.1093/her/13.1.87 · Cited by 1,129
Research works printed but not linked: 1
- Schoenwald SK GA, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA: Toward the effective and efficient measurement of implementation fidelity. Administration Policy Mental Health and Mental Health Serv Res. 2011, 38 (1): 32-43. 10.1007/s10488-010-0321-0. · DOI · Paper page
Cited by
Process evaluation of complex interventions: Medical Research Council guidance
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O'Cathain, A., Tinati, T., Wight, D., et al. (2015) · BMJ · doi:10.1136/bmj.h1258 · Cited by 6,204
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., et al. (2021) · BMJ · doi:10.1136/bmj.n2061 · Cited by 5,808
RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review
Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., Ory, M. G., & Estabrooks, P. A. (2019) · Frontiers in Public Health · doi:10.3389/fpubh.2019.00064 · Cited by 2,528
“Rapid, iterative use and analysis of brief practical measures of RE-AIM factors can inform adaptations (55, 66, 71).”
MethodWiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019) · Implementation Science · doi:10.1186/s13012-019-0898-y · Cited by 1,329
Self-citationSystematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019) · Implementation Science · doi:10.1186/s13012-018-0842-6 · Cited by 1,223
Metadata
- Authors
- Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A.
- Affiliations
- National Center for PTSD (US)
- VA Boston Healthcare System (US)
- University of Pennsylvania (US)
- University of Massachusetts Boston (US)
- Corresponding author
- Stirman, Shannon Wiltsey
- Venue
- Implementation Science
- Published
- 2013-06-10
- Volume
- 8
- Issue
- 1
- Pages
- 65-65
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3686699
- OpenAlex
- W2109880945
- Cited by
- 691
- References
- 59
- Citations by year
- 2026: 49 · 2025: 50 · 2024: 61 · 2023: 79 · 2022: 76 · 2021: 97 · 2020: 72 · 2019: 69 · 2018: 40 · 2017: 40 · 2016: 19 · 2015: 22 · 2014: 12 · 2013: 3
- FWCI
- 33.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs (R25 MH080916-01A2)
- Washington University in St. Louis (R25 MH080916-01A2)
- National Institutes of Health
- Quality Enhancement Research Initiative (R25 MH080916-01A2)
- National Institute of Mental Health (MH080916, R00 MH 01800, R00MH080100, R25MH080916)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Keywords
- evidence-based interventions, intervention adaptation, coding scheme, implementation fidelity, community settings, rater agreement, intervention effectiveness
- MeSH terms
- Mental Disorders, Behavioral Medicine, Community Mental Health Services, Decision Making, Delivery of Health Care, Humans, Evidence-Based Medicine, Clinical Coding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Development of a framework and coding system for modifications and adaptations of evidence-based interventions. https://app.cochat.ai/papers/10.1186/1748-5908-8-65
- The paper
- Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science. https://doi.org/10.1186/1748-5908-8-65
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

