The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Inter-annotator Agreement
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“…proceed with the coefficients that are the most appropriate for gauging the reliability of the annotation process, that is Fleiss’s κ and Krippendorff’s α. Note: The term “kappa” (κ) may refer to several distinct agreement coefficients, most commonly those of Cohen (1960) and Fleiss (1971).”
BackgroundAnswering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“And while confidence intervals for agreement coefficients can be estimated through resampling of the annotated items (Hayes and Krippendorff, 2007), this method cannot be used to quantify annotator variation, because resampling annotators would result in measuring agreement between an annotator and…”
MethodByrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 1×
“The debates on the matter have been raging for decades, in particular on how to treat individual differences between annotators (see for example Fleiss, 1975; Krippendorff, 1978, 2004b; Zwick, 1988; Byrt et al, 1993; Hsu and Field, 2003; Di Eugenio and Glass, 2004; Craggs and McGee Wood, 2005).”
BackgroundAssessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Self-citationReliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027 · cited 3×
“Krippendorff (2004a) explicitly builds the requirement that annotators be interchangeable into the definition of α, insisting that all the knowledge required for the annotation task be derived from the written manuals.”
BackgroundThe Benefits of a Model of Annotation
Passonneau, R. J., & Carpenter, B. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00185 · Cited by 227
What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation
Bayerl, P. S., & Paul, K. I. (2011) · Computational Linguistics · doi:10.1162/coli_a_00074 · Cited by 126
Cited by
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
Adapting Self-Regulated Learning in an Age of Generative Artificial Intelligence Chatbots
Lai, J. W. (2024) · Future Internet · doi:10.3390/fi16060218 · Cited by 51
“IAA is used in various fields such as natural language processing, information retrieval, and content analysis, where multiple annotators need to independently reach a measurable degree of agreement when labeling or codifying data [41,42].”
MethodLeveraging Foundation Models for Clinical Text Analysis
Raza, S., & Bashir, S. R. (2023) · 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi57859.2023.00103 · Cited by 2
Metadata
- Authors
- Artstein, R.
- Venue
- Handbook of Linguistic Annotation
- Publisher
- Springer Netherlands
- Published
- 2017
- Pages
- 297-313
- OpenAlex
- W2626407095
- Cited by
- 190
- References
- 9
- FWCI
- 29.30
- Type
- book chapter
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Inter-annotator Agreement. https://app.cochat.ai/papers/10.1007/978-94-024-0881-2_11
- The paper
- Artstein, R. (2017). Inter-annotator Agreement. Handbook of Linguistic Annotation. https://doi.org/10.1007/978-94-024-0881-2_11
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.