Reliability of Adherence and Competence Assessment in Cognitive Behavioral Therapy: Influence of Clinical Experience
Abstract
The use of highly experienced expert judges was suggested for the assessment of therapists' adherence and competence. However, such an approach implies high costs. It can be questioned whether only experts are able to evaluate therapists' adherence and competence reliably. To test this, 4 judges evaluated therapist adherence and competence in 30 randomly selected videotapes of cognitive therapy sessions for depression. In that, 2 judges exhibited high clinical experience (experts), whereas 2 judges did not (novices). We could demonstrate that novices evaluated an aggregated adherence and competence measure with high reliability. However, several adherence and competence aspects were not assessed with satisfactory reliability by novices. Although adherence ratings of experts and novices showed high concordance, the concordance of competence ratings was only moderate. Results revealed that therapists' adherence could be evaluated satisfactorily by trained novices with some restrictions, but not their competence.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Both evaluation forms were developed following the Cognitive-Behavioral Maintenance Adherence Scale.
“Both evaluation forms were developed following the Cognitive-Behavioral Maintenance Adherence Scale (30).”
From Efficacy of Cognitive-Behavioral Therapy for the Prophylaxis of Migraine in Adults: A Three-Armed Randomized Controlled Trial (2022) · cites this paper as [30]
Cites
Hamilton, M. (1960) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp.23.1.56 · Cited by 32,817 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Foundations of Clinical Research: Applications to Practice,
(2002) · Survey of Ophthalmology · doi:10.1016/s0039-6257(02)00362-4 · Cited by 9,485 · see citing works
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903 · see citing works
Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence
Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.61.4.620 · Cited by 873 · see citing works
Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.75.6.829 · Cited by 533 · see citing works
The Relationship Between Homework Compliance and Therapy Outcomes: An Updated Meta-Analysis
Mausbach, B. T., Moore, R., Roesch, S., Cardenas, V., & Patterson, T. L. (2010) · Cognitive Therapy and Research · doi:10.1007/s10608-010-9297-z · Cited by 309 · see citing works
Barriers to implementing treatment integrity procedures: Survey of treatment outcome researchers.
Perepletchikova, F., Hilt, L. M., Chereji, E., & Kazdin, A. E. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0015232 · Cited by 185 · see citing works
Weck, F., Hautzinger, M., Heidenreich, T., & Stangier, U. (2010) · Zeitschrift für Klinische Psychologie und Psychotherapie · doi:10.1026/1616-3443/a000055 · Cited by 57 · see citing works
Self-citationConsiderations in treatment integrity: Implications and recommendations for PTSD research
Barber, J. P., Triffleman, E., & Marmar, C. R. (2007) · Journal of Traumatic Stress · doi:10.1002/jts.20295 · Cited by 55 · see citing works
Evaluating Rater Competency for CNS Clinical Trials
Targum, S. D. (2006) · Journal of Clinical Psychopharmacology · doi:10.1097/01.jcp.0000219049.33008.b7 · Cited by 28 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A systematic review of methods for assessing competence in cognitive-behavioural therapy.
Muse, K., & McManus, F. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.01.010 · Cited by 189 · see citing works
Stangier, U., Hilling, C., Heidenreich, T., Risch, A. K., Barocka, A., Schlösser, R., Kronfeld, K., Ruckes, C., Berger, H., Röschke, J., et al. (2013) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2013.12060734 · Cited by 174 · see citing works
Self-citationKlan, T., Gaul, C., Liesering-Latta, E., Both, B., Held, I., Hennemann, S., & Witthöft, M. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.852616 · Cited by 22 · see citing works
“Both evaluation forms were developed following the Cognitive-Behavioral Maintenance Adherence Scale (30).”
MethodAn Integrative Cognitive Behavioral Therapy Program for Adults With Migraine: A Feasibility Study
Klan, T., Liesering‐Latta, E., Gaul, C., Martin, P. R., & Witthöft, M. (2019) · Headache: The Journal of Head and Face Pain · doi:10.1111/head.13532 · Cited by 21 · see citing works
- Authors
- Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U.
- Affiliations
- Goethe University Frankfurt (DE)
- Corresponding author
- Weck, Florian
- Hilling, Christine
- Rudari, Visar
- Stangier, Ulrich
- Venue
- Journal of Nervous & Mental Disease
- Published
- 2011-04
- Volume
- 199
- Issue
- 4
- Pages
- 276-279
- ISSN
- 0022-3018 (ISSN-L)
- OpenAlex
- W2077412993
- Cited by
- 44
- References
- 11
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 4 · 2022: 4 · 2021: 1 · 2019: 6 · 2018: 1 · 2017: 3 · 2016: 4 · 2015: 5 · 2014: 5 · 2013: 4 · 2012: 1
- FWCI
- 4.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The Journal of Nervous and Mental Disease (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- therapist adherence, therapist competence, cognitive behavioral therapy, inter-rater reliability, novice raters, expert raters, depression treatment
- MeSH terms
- Adult, Aged, Clinical Competence, Depressive Disorder, Female, Humans, Male, Middle Aged, Observer Variation, Cognitive Behavioral Therapy, Practice Guidelines as Topic, Guideline Adherence, Secondary Prevention, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
