A new coefficient of interrater agreement: The challenge of highly unequal category proportions.

We derive a general structure that encompasses important coefficients of interrater agreement such as the S-coefficient, Cohen's kappa, Scott's pi, Fleiss' kappa, Krippendorff's alpha, and Gwet's AC1. We show that these coefficients share the same set of assumptions about rater behavior; they onl…