Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe.

Abstract

The interrater reliabilities of ratings of 9,975 ratees from 79 organizations were examined as a function of length of exposure to the ratee. It was found that there was a strong, nonlinear relationship between months of exposure and interrater reliability. The correlation between a logarithmic transformation of months of experience and reliability was .73 for one type of ratings and .65 for another type. The relationship was strongest during the first 12 months on the job. Implications for estimating reliabilities in individual and meta-analytic studies and for performance appraisal were presented, and possible explanations of the reliability-variance relationship were advanced

Cited by (18)

1. The Handbook of Research Synthesis and Meta-Analysis
   2019
2. The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach.
   Rotundo, Maria, Sackett, Paul R. · 2002 · Journal of Applied Psychology
3. The relation between emotional intelligence and job performance: A meta-analysis
   O'Boyle, Ernest H., Humphrey, Ronald H., Pollack, Jeffrey M., Hawver, Thomas H., Story, Paul A. · 2011 · Journal of Organizational Behavior
4. DOES PERFORMANCE IMPROVE FOLLOWING MULTISOURCE FEEDBACK? A THEORETICAL MODEL, META‐ANALYSIS, AND REVIEW OF EMPIRICAL FINDINGS
   SMITHER, JAMES W., LONDON, MANUEL, REILLY, RICHARD R. · 2005 · Personnel Psychology
5. Measurement error in psychological research: Lessons from 26 research scenarios.
   Schmidt, Frank L., Hunter, John E. · 1996 · Psychological Methods

See all 221