Effects of rater training: Creating new response sets and decreasing accuracy.

Bernardin, H. John, Pence, Earl C.·1980·Journal of Applied Psychology

Abstract

The extent of leniency error, the extent of halo effect, and the accuracy of ratings on behavioral expectation scales were compared across three groups of student raters (N - 72). One group of raters (RET) underwent a training program on rating error, involving definitions, graphic illustrations, and numerical examples of leniency and halo. A second group (RAT) heard lectures on the multidimensionality of teacher performance, generated and defined dimensions of performance, discussed behavioral examples of each dimension, and attempted to develop stereotypes of effective and ineffective performance. The third group received no training. Following the training, two hypothetical ratees, described in written vignettes, were each rated on 13 dimensions of performance. Ratings from the RET group had significantly less leniency and halo error than ratings from the RAT group and the control group. However, when ratings were compared to previously developed true scores on the ratees, significantly less accuracy was found for the RET group than for the other two groups. No significant differences were found between the RAT group and the control group ratings. Results are discussed in terms of the facilitation of a response set in raters trained by denning and illustrating errors through rating distributions. The response set results in lower mean ratings (less leniency) and lower scale intercorrelations (lower halo) while lowering the accuracy of ratings. Suggestions for the improvement of training programs are made.

Cited by (15)

  1. Estimating within-group interrater reliability with and without response bias
    James, Lawrence R., Demaree, Robert G., Wolf, Gerrit · 1984 · Journal of Applied Psychology
  2. On the accuracy of personality judgment: A realistic approach
    Funder, David C. · 1995 · Psychological Review
  3. Errors and mistakes: Evaluating the accuracy of social judgment.
    Funder, David C. · 1987 · Psychological Bulletin
  4. Performance appraisal and performance management: 100 years of progress?
    DeNisi, Angelo S., Murphy, Kevin R. · 2017 · Journal of Applied Psychology
  5. A cognitive view of the performance appraisal process: A model and research propositions
    DeNisi, Angelo S., Cafferty, Thomas P., Meglino, Bruce M. · 1984 · Organizational Behavior and Human Performance

See all 221

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Effects of rater training: Creating new response sets and decreasing accuracy. https://app.cochat.ai/papers/10.1037//0021-9010.65.1.60
The paper
Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology. https://doi.org/10.1037//0021-9010.65.1.60

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.