Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation

The question of data reliability is of first importance to assess the quality of manually annotated corpora.Although Cohen ' s κ is the prevailing reliability measure used in NLP, alternative statistics have been proposed.This paper presents an experimental study with four measures (Cohen's κ, Sc…