The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Rater effects in evaluating therapist competencies using structured case reports: A mixed-methods study
Abstract
Objective: This study applied a mixed-methods approach to investigate rater effects that might affect case report evaluation in China. Method: In the quantitative phase, we randomly assigned 210 mental health professionals to the experienced or novice ratee condition to rate the same structured case report on assessment and formulation, intervention strategy, relationship, and self-reflection. The qualitative phase was subsequently conducted to help make sense of the results. We interviewed twelve supervisors about how they would rate a case report and factors they thought would influence their evaluation. Results: Linear model with non-constant variances revealed that the manipulation of ratee experience did not significantly bias ratings. Raters whose self-identified theoretical orientation aligned with that of the report provided higher and less dispersed ratings for intervention strategy and therapeutic relationship competencies. As their years of experience increased, theoretically congruent raters tended to give lower and more dispersed ratings. Thematic analysis revealed highly variable cognitive processes, and identified rater theoretical expertise, evaluation context, rater subjectivity and dual relationships as important factors to consider. Conclusion: This study indicate that theoretically congruent raters would be desirable, but raters with more years of experience should be cautioned against giving stringent ratings for theoretically familiar cases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051
Braun, V., & Clarke, V. (2021) · Counselling and Psychotherapy Research · doi:10.1002/capr.12360 · Cited by 3,198
Mixed methods research designs in counseling psychology.
Hanson, W. E., Creswell, J. W., Clark, V. L. P., Petska, K. S., & Creswell, J. D. (2005) · Journal of Counseling Psychology · doi:10.1037/0022-0167.52.2.224 · Cited by 1,375
Thagard, P. R. (1989) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00057046 · Cited by 1,075
MINERVA-DM: A memory processes model for judgments of likelihood.
Dougherty, M. R. P., Gettys, C. F., & Ogden, E. E. (1999) · Psychological Review · doi:10.1037/0033-295x.106.1.180 · Cited by 356
The role of deliberate practice in the development of highly effective psychotherapists.
Chow, D. L., Miller, S. D., Seidel, J. A., Kane, R. T., Thornton, J. A., & Andrews, W. P. (2015) · Psychotherapy · doi:10.1037/pst0000015 · Cited by 313
Magnitude and moderators of bias in observer ratings: A meta-analysis.
Hoyt, W. T., & Kerns, M.-D. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.4.403 · Cited by 197
Eells, T. D., Lombart, K. G., Kendjelic, E. M., Turner, L., & Lucas, C. P. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.4.579 · Cited by 175
Therapist Expertise in Psychotherapy Revisited
Hill, C. E., Spiegel, S. B., Hoffman, M. A., Kivlighan, D. M., & Gelso, C. J. (2017) · The Counseling Psychologist · doi:10.1177/0011000016641192 · Cited by 157
Field supervisors' assessments of trainee performance: Are they reliable and valid?
Gonsalvez, C. J., & Freestone, J. (2007) · Australian Psychologist · doi:10.1080/00050060600827615 · Cited by 84
Research works printed but not linked: 4
- Lu, Y., Liu, M., Gu, T., & Zhu, X. (2024). Assessment of professional competence for clinical and counseling psychology professionals. Journal of Psychological Science, 47(1), 237–245.
- 10.3969/j.issn.1000-6729.2010.12.013 · DOI
- Wang, M., Jiang, G., Yan, Y., & Zhou, Z. (2015). The way for certifying counselors and psychotherapists in China. Chinese Mental Health Journal, 20(07), 503–509.
- Zhu, X., Lu, Y., & Tang, W. (2022). Xinli zixunshi de zhuanye xunlian [Professional training of psychological counselors]. In F. Fan, X. Zhu, & J. He (Eds.), Zixun xinli xue [Counseling psychology]. (pp. 465-489). East China Normal University Press.
Cited by
Sorted by citations
Rater analysis using Item Response Theory in a clinical context
Kabic, M. (2025) · University of Klagenfurt
Metadata
- Authors
- Chen, Y., Shi, Y., & Lu, Y.
- Affiliations
- Zhejiang University (CN)
- Corresponding author
- Lu, Yun
- Venue
- Psychotherapy Research
- Publisher
- Informa UK Limited
- Published
- 2025-01-07
- Volume
- 35
- Issue
- 8
- Pages
- 1371-1382
- ISSN
- 1050-3307 (ISSN-L)
- 1468-4381
- OpenAlex
- W4406189204
- Semantic Scholar
- 216801246d5e701f0d4cbdf9528508672fae46b7
- Cited by
- 1
- References
- 28
- Citations by year
- 2025: 1
- FWCI
- 0.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychotherapy Research (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Clinical Psychology › Counseling Practices and Supervision
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- rater effects, rater experience, rater subjectivity, dual relationships, therapeutic relationship, intervention strategies, rater bias
- MeSH terms
- Adult, China, Clinical Competence, Female, Humans, Male, Middle Aged, Psychotherapy, Observer Variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
