Effects of rater training: Creating new response sets and decreasing accuracy.
Abstract
The extent of leniency error, the extent of halo effect, and the accuracy of ratings on behavioral expectation scales were compared across three groups of student raters (N - 72). One group of raters (RET) underwent a training program on rating error, involving definitions, graphic illustrations, and numerical examples of leniency and halo. A second group (RAT) heard lectures on the multidimensionality of teacher performance, generated and defined dimensions of performance, discussed behavioral examples of each dimension, and attempted to develop stereotypes of effective and ineffective performance. The third group received no training. Following the training, two hypothetical ratees, described in written vignettes, were each rated on 13 dimensions of performance. Ratings from the RET group had significantly less leniency and halo error than ratings from the RAT group and the control group. However, when ratings were compared to previously developed true scores on the ratees, significantly less accuracy was found for the RET group than for the other two groups. No significant differences were found between the RAT group and the control group ratings. Results are discussed in terms of the facilitation of a response set in raters trained by denning and illustrating errors through rating distributions. The response set results in lower mean ratings (less leniency) and lower scale intercorrelations (lower halo) while lowering the accuracy of ratings. Suggestions for the improvement of training programs are made.
Cited by (15)
- Estimating within-group interrater reliability with and without response biasJames, Lawrence R., Demaree, Robert G., Wolf, Gerrit · 1984 · Journal of Applied Psychology
- On the accuracy of personality judgment: A realistic approachFunder, David C. · 1995 · Psychological Review
- Errors and mistakes: Evaluating the accuracy of social judgment.Funder, David C. · 1987 · Psychological Bulletin
- Performance appraisal and performance management: 100 years of progress?DeNisi, Angelo S., Murphy, Kevin R. · 2017 · Journal of Applied Psychology
- A cognitive view of the performance appraisal process: A model and research propositionsDeNisi, Angelo S., Cafferty, Thomas P., Meglino, Bruce M. · 1984 · Organizational Behavior and Human Performance
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effects of rater training: Creating new response sets and decreasing accuracy. https://app.cochat.ai/papers/10.1037//0021-9010.65.1.60
- The paper
- Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology. https://doi.org/10.1037//0021-9010.65.1.60
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.