Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: a Delphi study
Open accessThe DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The integration of behavioral health services into primary care is increasingly popular, yet fidelity of implementation in this area has been infrequently assessed due to the few measurement tools available. A sentinel indicator of fidelity of implementation is provider adherence, or utilization of prescribed procedures and engagement in model-specific behaviors. This study aimed to develop the first self-report measure of behavioral health provider adherence for co-located, collaborative care, a commonly adopted model of behavioral health service delivery in primary care. METHODS: A preliminary 56-item measure was developed by the research team to represent critical components of adherence among behavioral health providers. To ensure the content validity of the measure, a modified Delphi study was conducted using a panel of co-located, collaborative care model experts. During three rounds of emailed surveys, panel members provided qualitative feedback regarding item content while rating each item's relevance for behavioral health provider practice. Items with consensus ratings of 80% or greater were included in the final adherence measure. RESULTS: The panel consisted of 25 experts representing the Department of Veterans Affairs, the Department of Defense, and academic and community health centers (total study response rate of 76%). During the Delphi process, two new items were added to the measure, four items were eliminated, and a high level of consensus was achieved on the remaining 54 items. Experts identified 38 items essential for model adherence, six items compatible (although not essential) for model adherence, and 10 items that represented prohibited behaviors. Item content addressed several domains, but primarily focused on behaviors related to employing a time-limited, brief treatment model, the scope of patient concerns addressed, and interventions used by providers. CONCLUSIONS: This study yielded the first content valid self-report measure of critical components of collaborative care adherence for use by behavioral health providers in primary care. Although additional psychometric evaluation is necessary, this measure may assist implementation researchers in clarifying how provider behaviors contribute to clinical outcomes. This measure may also assist clinical stakeholders in monitoring implementation and identifying ways to support frontline providers in delivering high quality services.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirically supported anxiety interventions from specialty care (Norton & Price, 2007) are not readily translated to PCBH, where interventions must be brief and time-limited (e.g., deliverable in 6 or fewer 30-minute sessions; Beehler et al., 2013; Rowan & Runyan, 2005).”
From Brief modular anxiety intervention for primary care: Hybrid I pilot randomized controlled trial of feasibility, acceptability, effectiveness, and implementation potential (2024) · cites this paper as Beehler et al., 2013
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“MAST-V and other CBT interventions are a good fit for PCBH practice because they flow naturally from functional assessment, can be delivered quickly, and provide concrete self-management skills for patients to practice at home (Beehler et al., 2013; Rowan & Runyan, 2005).”
From Brief modular anxiety intervention for primary care: Hybrid I pilot randomized controlled trial of feasibility, acceptability, effectiveness, and implementation potential (2024) · cites this paper as Beehler et al., 2013
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On average, MAST-V treatment comprised 5 sessions, consistent with the goal of time-limited services in primary care (Beehler et al., 2013).”
From Brief modular anxiety intervention for primary care: Hybrid I pilot randomized controlled trial of feasibility, acceptability, effectiveness, and implementation potential (2024) · cites this paper as Beehler et al., 2013
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009) · Implementation Science · doi:10.1186/1748-5908-4-50 · Cited by 14,496 · cited 1×
“Fidelity refers to the de gre to which an intervention is implemented as intended [19], and is reflected in a variety of implementation fr ameworks and evaluation models [20,21].”
BackgroundProctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721
the delphi method: techniques and applications
Linstone, H. A., & Turoff, M. (1975) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 6,590
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951
The Delphi Technique: Making Sense of Consensus
Hsu, C.-C., & Sandford, B. A. (2007) · University of Massachusetts Amherst · doi:10.7275/pdz9-th90 · Cited by 4,500 · cited 1×
“An advantage to this approach is that it provides anonymity, thereby decreasing the likelihood that some participants will modify their responses based on the opinions of highl y nfluential experts [35].”
BackgroundPractical Assessment, Research & Evaluation, 2000-2001.
Rudner, L. M., & Schafer, W. D. (2001) · Practical assessment, research & evaluation · Cited by 3,905
Bellg, A. J., Treatment Fidelity Workgroup of the NIH Behavior Change Consortium, Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., Ogedegbe, G., Orwig, D., Ernst, D., et al. (2004) · Health Psychology · doi:10.1037/0278-6133.23.5.443 · Cited by 2,620 · cited 1×
“The US National Institutes of Health Behavior Change Consortium has describe d five aspects of fidelity as follows: study design, provider training, treatment de livery, treatment receipt, and treatment enactment [30].”
BackgroundThe Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,584
A conceptual framework for implementation fidelity
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007) · Implementation Science · doi:10.1186/1748-5908-2-40 · Cited by 2,559
Methods of coping with social desirability bias: A review
Nederhof, A. J. (1985) · European Journal of Social Psychology · doi:10.1002/ejsp.2420150303 · Cited by 2,340 · cited 1×
“When used in future wider-scale research on patient outcomes, rather than as a tool for training, the nature of the PPAQ as a self-administered instrument may be beneficial for reducing the impact of social desirability [45], particularly if the research design allows for administration over the internet or other appr…”
Method
Research works printed but not linked: 1
- VA Health Services Research and Development Service: Collaborative Care for Depression in the Primary Care Setting. A Primer on VA’s Translating Initiatives for Depression into Effective Solutions (TIDES) Project. 2008, Boston: Center for Information Dissemination and Education Resources, VA Health Services Research and Development Service, Office of Research and Development, U.S. Dept. of Veterans Affairs
Cited by
Hadlaczky, G., Hökby, S., Mkrtchian, A., Carli, V., & Wasserman, D. (2014) · International Review of Psychiatry · doi:10.3109/09540261.2014.924910 · Cited by 433
“…literature reviews or, regarding issues where comprehensive evidence does not exist, on expert consensus achieved using the Delphi method (Beehler et al., 2013; Berk et al., 2011; Clyne et al., 2012; Colucci et al., 2010a, 2010b, 2011; Hart et al., 2009, 2010; Hemmings et al., 2009; Jorm…”
MethodQualitative and Mixed Methods in Mental Health Services and Implementation Research
Palinkas, L. A. (2014) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2014.910791 · Cited by 365
Raine, R., Fitzpatrick, R., Barratt, H., Bevan, G., Black, N., Boaden, R., Bower, P., Campbell, M., Denis, J.-L., Devers, K., et al. (2016) · Health Services and Delivery Research · doi:10.3310/hsdr04160 · Cited by 186
Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties
Mettert, K., Lewis, C., Dorsey, C., Halko, H., & Weiner, B. (2020) · Implementation Research and Practice · doi:10.1177/2633489520936644 · Cited by 162
Hunter, C. L., Funderburk, J. S., Polaha, J., Bauman, D. E., Goodie, J. L., & Hunter, C. M. (2017) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-017-9512-0 · Cited by 126
Self-citation
Metadata
- Authors
- Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L.
- Affiliations
- Buffalo VA Medical Center (US)
- University of Rochester Medicine (US)
- Syracuse VA Medical Center (US)
- Corresponding author
- Beehler, Gregory P
- Venue
- Implementation Science
- Published
- 2013-02-13
- Volume
- 8
- Issue
- 1
- Pages
- 19-19
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC3640894
- OpenAlex
- W2147045828
- Cited by
- 41
- References
- 49
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2022: 3 · 2021: 7 · 2020: 2 · 2019: 3 · 2018: 4 · 2017: 4 · 2016: 5 · 2015: 3 · 2014: 3 · 2013: 1
- FWCI
- 7.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Open Scholarship Institutional Repository (Washington University in St. Louis) (submittedVersion, cc-by)
- Funders
- U.S. Department of Veterans Affairs
- Health Resources and Services Administration
- VA Western New York Healthcare System
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Policy Implementation Science
- Health Sciences › Health Professions › General Health Professions › Interprofessional Education and Collaboration
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Healthcare Systems and Technology
- Keywords
- provider adherence, collaborative care, behavioral health services, primary care, implementation fidelity, self-report measures, content validity, behavioral health providers, short-term treatment, implementation monitoring
- MeSH terms
- Behavior Therapy, Mental Disorders, Delivery of Health Care, Delphi Technique, Humans, Mental Health Services, Primary Health Care, Professional Practice, Practice Guidelines as Topic, Guideline Adherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: a Delphi study. https://app.cochat.ai/papers/10.1186/1748-5908-8-19
- The paper
- Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: a Delphi study. Implementation Science. https://doi.org/10.1186/1748-5908-8-19
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
