On various intraclass correlation reliability coefficients.

Abstract

This paper briefly reviews the most frequently used and misused intraclass cor-relation- analysis of variance (ANOVA) reliability measures. Recommendations are made for the one-way ANOVA intraclass correlation and against the use of several coefficients: Winer's so-called "anchor point method, " Kuder-Richardson Formula Number 20, and the Spearman-Brown prediction formula. The computation of the intraclass correlation coefficient via two-way ANOVA is not encouraged. Several uses and misuses of reliability coefficients applied to dichotomous data are also illustrated. Bartko (1966, 1974) has presented some analysis of variance (ANOVA) intraclass cor-relation reliability coefficients that avoid some serious deficiencies not uncommonly found in reliability measures. In his second edition, Winer (1971, pp, 289-296) presented some intraclass correlation results which appear to have deficiencies. His so-called "adjustment for anchor points " approach will produce an intraclass correlation reliability coefficient of unity (as expected) for the case in which the judges (raters) agree perfectly about a group of subjects. However, the method will also yield an intraclass correlation of unity for the case in which the judges display a con-stant additive bias. In general with Winer's approach, any ad-justment of original rating data that leaves the rater's variance-covariance matrix un-altered will produce the same intraclass cor-relation coefficient, and thus numerous vari-ations (of which additive bias is a subset) of the original data set can and will yield the same intraclass correlation. Bias and Unity Reliability As a first illustration on a more elementary level, the phenomenon discussed above can be observed with the product-moment correla-tion, which is a sometimes used but not rec-ommended measure of reliability. Consider Requests for reprints should be sent to John J.

Cited by (41)

1. Intraclass correlations: Uses in assessing rater reliability.
   Shrout, Patrick E., Fleiss, Joseph L. · 1979 · Psychological Bulletin
2. The Yale-Brown Obsessive Compulsive Scale: I. Development, Use, and Reliability
   Goodman, Wayne K. · 1989 · Archives of General Psychiatry
3. Forming inferences about some intraclass correlation coefficients.
   McGraw, Kenneth O., Wong, S. P. · 1996 · Psychological Methods
4. Estimating within-group interrater reliability with and without response bias
   James, Lawrence R., Demaree, Robert G., Wolf, Gerrit · 1984 · Journal of Applied Psychology
5. Answers to 20 questions about interrater reliability and interrater agreement
   LeBreton, James M., Senter, Jenell L. · 2008 · Organizational Research Methods

See all 825