CoChat
Sign Up
Cited by 1,665Reliability and Agreement in MeasurementComputational and Text Analysis MethodsNatural Language Processing Techniques

Inter-Coder Agreement for Computational Linguistics

Open access
Artstein, Ron, Poesio, Massimo·2008·Computational Linguistics
DOI 10.1162/coli.07-034-r2
Open this paper in CoChat — free accountOpen PDF

Abstract

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/coli.07-034-r2

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Artstein and Poesio (2008) discussed several drawbacks when metrics without chance-correction like accuracy and correlation.

    “Artstein and Poesio (2008) discussed several drawbacks when metrics without chance-correction like accuracy and correlation.”

    From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Artstein and Poesio, 2008

    Section 2.3, “Agreement Without Chance Correction,” discusses two drawbacks of percentage agreement: it varies with the number of categories and can be misleading when categories are unevenly distributed. Section 3.2 also explains why high accuracy can coexist with unreliable coding. The paper does not discuss the drawbacks of correlation coefficients here; a note in Section 5.3 says that discussion is in an extended version. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • A weighted version of the multi-coders κ statistics is still missing.

    “This weighted generalization restricts however to a two coders scheme (Artstein and Poesio, 2008): a weighted version of the multi-coders κ statistics is still missing.”

    From Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation (2014) · cites this paper as Artstein and Poesio, 2008

    Section 2.6.2 describes weighted Cohen’s κ using two coders, supporting the first part of the citing sentence. But the paper does not say that a weighted multi-coder κ is missing; Section 3.1 discusses a weighted, individual-coder-distribution variant used in a study with 18 subjects. The claimed absence is therefore not supported by this source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Krippendorff's α and Cohen's weighted κ are the standard panel-level inter-rater stability metrics for ordinal scoring.

    “Krippendorff's α [Krippendorff, 2011] and Cohen's weighted κ are the standard panel-level inter-rater stability metrics for ordinal scoring”

    From A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays (2026) · cites this paper as Artstein and Poesio, 2008

    Section 2.6 describes both α and weighted κ as measures that can account for different magnitudes of disagreement, and §2.6.1 says α applies to multiple coders. But the paper says weighted κ is defined for two coders, so it does not support calling both measures panel-level metrics. Section 5.2 also recommends α, rather than establishing both as standard for ordinal scoring; page numbers were not available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Metrics without chance-correction cannot be compared across studies because some of the agreement will be due to chance, and that chance is affected by factors that vary from one study to another.

    “Despite being intuitive to understand, their values cannot be compared across studies, because some of the agreement will be due to chance, and that chance is affected by factors that vary from one study to another.”

    From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Artstein and Poesio, 2008

  • Agreements among annotators were moderate for economic, substantial for foreign, and fair for social.

    “The agreements among annotators for the economic and foreign dimensions are moderate & substantial (Artstein and Poesio, 2008), respectively; for social, the 'fair' agreement was noticed during annotation, and additional discussion for each paragraph was then held.”

    From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Artstein and Poesio, 2008

    The citation is attached to the economic and foreign agreement ratings; the citing sentence presents the social rating as its own observation. Section 5.3 discusses how to interpret agreement coefficients but gives no ratings for economic or foreign annotations. Searches for those dimensions and for “moderate” and “substantial” found no supporting passage in our copy. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 129

  1. Artstein, Ron and Massimo Poesio. 2005. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005, pages 141-150, Edinburgh.
  2. Artstein, Ron and Massimo Poesio. 2006. Identifying reference to abstract objects in dialogue. In brandial 2006: Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pages 56-63, Potsdam.
  3. Babarczy, Anna, John Carroll, and Geoffrey Sampson. 2006. Definitional, personal, and mechanical constraints on part of speech annotation performance. Natural Language Engineering, 12(1):77-90.
  4. Bartko, John J. and William T. Carpenter, Jr. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307-317.
  5. Beeferman, Doug, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1-3):177-210.
  6. Bennett, E. M., R. Alpert, and A. C. Goldstein. 1954. Communications through limited questioning. Public Opinion Quarterly, 18(3):303-308.
  7. Bloch, Daniel A. and Helena Chmura Kraemer. 1989. 2 × 2 kappa coefficients: Measures of agreement or association. Biometrics, 45(1):269-287.
  8. Brennan, Robert L. and Dale J. Prediger. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3):687-699.
  9. Bruce, Rebecca and Janyce Wiebe. 1998. Word-sense distinguishability and inter-coder agreement. In Proceedings of EMNLP, pages 53-60, Granada.
  10. Bruce, Rebecca F. and Janyce M. Wiebe. 1999. Recognizing subjectivity: A case study in manual tagging. Natural Language Engineering, 5(2):187-205.
  11. Buitelaar, Paul. 1998. CoreLex : Systematic Polysemy and Underspecification. Ph.D. thesis, Brandeis University, Waltham, MA.
  12. Bunt, Harry C. 2000. Dynamic interpretation and dialogue theory. In Martin M. Taylor, Franc ¸oise Néel, and Don G. Bouwhuis, editors, The Structure of Multimodal Dialogue II. John Benjamins, Amsterdam, pages 139-166.
  13. Bunt, Harry C. 2005. A framework for dialogue act specification. In Proceedings of the Joint ISO-ACL Workshop on the Representation and Annotation of Semantic Information, Tilburg. Available at: http://let.uvt.nl/research/ti/ sigsem/wg/discussionnotes4.htm.
  14. Byron, Donna K. 2002. Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the ACL, pages 80-87, Philadelphia, PA.
  15. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
  16. Carletta, Jean, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1):13-32.
  17. Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Jan C. J. van Kuppevelt and Ronnie W. Smith, editors, Current and New Directions in Discourse and Dialogue. Kluwer, Dordrecht, pages 85-112.
  18. Cicchetti, Domenic V. and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551-558.
  19. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
  20. Cohen, Jacob. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213-220.
  21. Core, Mark G. and James F. Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, AAAI, Cambridge, MA. Available at: http://www. cs.umd.edu/∼traum/CA/fpapers.html.
  22. Craggs, Richard and Mary McGee Wood. 2004. A two-dimensional annotation scheme for emotion in dialogue. In Papers from the 2004 AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, pages 44-49.
  23. Craggs, Richard and Mary McGee Wood. 2005. Evaluating discourse and dialogue coding schemes. Computational Linguistics, 31(3):289-295.
  24. Davies, Mark and Joseph L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38(4):1047-1051.
  25. Di Eugenio, Barbara. 2000. On the usage of Kappa to evaluate agreement on coding tasks. In Proceedings of LREC, volume 1, pages 441-444, Athens.
  26. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
  27. Di Eugenio, Barbara, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason. 1998. An empirical investigation of proposals in collaborative dialogues. In Proceedings of 36th Annual Meeting of the ACL, pages 325-329, Montreal.
  28. Dice, Lee R. 1945. Measures of the amount of ecologic association between species. Ecology, 26(3):297-302.
  29. Doran, Christine, John Aberdeen, Laurie Damianos, and Lynette Hirschman. 2001. Comparing several aspects of human-computer and human-human dialogues. In Proceedings of the 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark. Available at: http://www.sigdial.org/workshops/ workshop2/proceedings.
  30. Eckert, Miriam and Michael Strube. 2000. Dialogue acts, synchronizing units, and anaphora resolution. Journal of Semantics, 17(1):51-89.
  31. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA.
  32. Fleiss, Joseph L. 1975. Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31(3):651-659.
  33. Geertzen, Jeroen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages 126-133, Sydney.
  34. Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204.
  35. Hearst, Marti A. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33-64.
  36. Hovy, Eduard, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of HLT-NAACL, Companion Volume: Short Papers, pages 57-60, New York.
  37. Hsu, Louis M. and Ronald Field. 2003. Interrater agreement measures: Comments on kappa n , Cohen's kappa, Scott's π, and Aickin's α. Understanding Statistics, 2(3):205-219.
  38. Jaccard, Paul. 1912. The distribution of the flora in the Alpine zone. New Phytologist, 11(2):37-50.
  39. Kilgarriff, Adam. 1999. 95% replicability for manual word sense tagging. In Proceedings of the Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 277-278, Bergen, Norway.
  40. Klavans, Judith L., Samuel Popper, and Rebecca Passonneau. 2003. Tackling the internet glossary glut: Automatic extraction and evaluation of genus phrases. In Proceedings of the SIGIR-2003 Workshop on the Semantic Web, Toronto.
  41. Krippendorff, Klaus. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61-70.
  42. Krippendorff, Klaus. 1978. Reliability of binary attribute data. Biometrics, 34(1):142-144. Letter to the editor, with a reply by Joseph L. Fleiss.
  43. Krippendorff, Klaus. 1980. Content Analysis: An Introduction to Its Methodology, chapter 12. Sage, Beverly Hills, CA.
  44. Krippendorff, Klaus. 2004a. Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA.
  45. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. CLAWS4: The tagging of the British National Corpus. In Proceedings of COLING 1994: The 15th International Conference on Computational Linguistics, Volume 1, pages 622-628, Kyoto.
  46. Levin, James A. and James A. Moore. 1978. Dialogue-games: Metacommunication structures for natural language interaction. Cognitive Science, 1(4):395-420.
  47. Manning, Christopher D. and Hinrich Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
  48. Marcu, Daniel, Magdalena Romera, and Estibaliz Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In Workshop on Levels of Representation in Discourse, pages 71-78, University of Edinburgh.
  49. Marcus, Mitchell P., Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313-330.
  50. Melamed, I. Dan and Philip Resnik. 2000. Tagger evaluation given hierarchical tagsets. Computers and the Humanities, 34(1-2):79-84. Available at: http://www.sahs/utmb.edu/PELLINORE/ Intro to research/wad/wad/ home.htm.
  51. Mieskes, Margot and Michael Strube. 2006. Part-of-speech tagging of transcribed speech. In Proceedings of LREC, pages 935-938, Genoa.
  52. Mihalcea, Rada, Timothy Chklovski, and Adam Kilgarriff. 2004. The SENSEVAL-3 English lexical sample task. In Proceedings of SENSEVAL-3, pages 25-28, Barcelona.
  53. Miltsakaki, Eleni, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. Annotating discourse connectives and their arguments. In Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 9-16, Boston, MA.
  54. Navarretta, Costanza. 2000. Abstract anaphora resolution in Danish. In Proceedings of the 1st SIGdial Workshop on Discourse and Dialogue, Hong Kong, pages 56-65.
  55. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT-NAACL 2004, pages 145-152, Boston, MA.
  56. Palmer, Martha, Hoa Trang Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137-163.
  57. Passonneau, Rebecca J. 2004. Computing reliability for coreference annotation. In Proceedings of LREC, volume 4, pages 1503-1506, Lisbon.
  58. Passonneau, Rebecca J. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of LREC, Genoa, pages 831-836.
  59. Passonneau, Rebecca J., Nizar Habash, and Owen Rambow. 2006. Inter-annotator agreement on a multilingual semantic annotation task. In Proceedings of LREC, Genoa, pages 1951-1956.
  60. Passonneau, Rebecca J. and Diane J. Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of 31st Annual Meeting of the ACL, pages 148-155, Columbus, OH.
  61. Passonneau, Rebecca J. and Diane J. Litman. 1996. Empirical analysis of three dimensions of spoken discourse: Segmentation, coherence and linguistic devices. In Eduard H. Hovy and Donia R. Scott, editors, Computational and Conversational Discourse: Burning Issues - An Interdisciplinary Account, volume 151 of NATO ASI Series F: Computer and Systems Sciences. Springer, Berlin, chapter 7, pages 161-194.
  62. Passonneau, Rebecca J. and Diane J. Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1):103-139.
  63. Pevzner, Lev and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19-36.
  64. Poesio, Massimo. 2004a. Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation, pages 72-79, Barcelona.
  65. Poesio, Massimo. 2004b. The MATE/GNOME proposals for anaphoric annotation, revisited. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 154-162, Cambridge, MA.
  66. Poesio, Massimo and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76-83, Ann Arbor, MI.
  67. Poesio, Massimo and Natalia N. Modjeska. 2005. Focus, activation, and this-noun phrases: An empirical study. In Ant ónio Branco, Tony McEnery, and Ruslan Mitkov, editors, Anaphora Processing, volume 263 of Current Issues in Linguistic Theory. John Benjamins, pages 429-442, Amsterdam and Philadelphia.
  68. Poesio, Massimo, A. Patel, and Barbara Di Eugenio. 2006. Discourse structure and anaphora in tutorial dialogues: An empirical analysis of two theories of the global focus. Research in Language and Computation, 4(2-3):229-257.
  69. Poesio, Massimo and Renata Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183-216.
  70. Rajaratnam, Nageswari. 1960. Reliability formulas for independent decision data when reliability data are matched. Psychometrika, 25(3):261-271.
  71. Reidsma, Dennis and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguistics, 34(3):319-326.
  72. Reinhart, T. 1981. Pragmatics and linguistics: An analysis of sentence topics. Philosophica, 27(1):53-93.
  73. Reynar, Jeffrey C. 1998. Topic Segmentation: Algorithms and Applications. Ph.D. thesis, University of Pennsylvania, Philadelphia.
  74. Ries, Klaus. 2002. Segmenting conversations by topic, initiative and style. In Anni R. Coden, Eric W. Brown, and Savitha Srinivasan, editors, Information Retrieval Techniques for Speech Applications, volume 2273 of Lecture Notes in Computer Science. Springer, Berlin, pages 51-66.
  75. Rosenberg, Andrew and Ed Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proceedings of HLT-NAACL 2004: Short Papers, pages 77-80, Boston, MA.
  76. Scott, William A. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321-325.
  77. Shriberg, Elizabeth, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97-100, Cambridge, MA.
  78. Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences, 2nd edition, chapter 9.8. McGraw-Hill, New York.
  79. Stent, Amanda J. 2001. Dialogue Systems as Conversational Partners: Applying Conversation Acts Theory to Natural Language Generation for Task-Oriented Mixed-Initiative Spoken Dialogue. Ph.D. thesis, Department of Computer Science, University of Rochester.
  80. Stevenson, Mark and Robert Gaizauskas. 2000. Experiments on sentence boundary detection. In Proceedings of 6th ANLP, pages 84-89, Seattle, WA.
  81. Stolcke, Andreas, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-373.
  82. Teufel, Simone, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Proceedings of Ninth Conference of the EACL, pages 110-117, Bergen.
  83. Teufel, Simone and Marc Moens. 2002. Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4):409-445.
  84. Traum, David R. and Elizabeth A. Hinkelman. 1992. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3):575-599.
  85. Véronis, Jean. 1998. A study of polysemy judgments and inter-annotator agreement. In Proceedings of SENSEVAL-1, Herstmonceux Castle, England. Available at: http://www.itri.brighton.ac.uk/ events/senseval/ARCHIVE/PROCEEDINGS/.
  86. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference, pages 45-52, Columbia, MD.
  87. Zwick, Rebecca. 1988. Another look at interrater agreement. Psychological Bulletin, 103(3):374-378.
  88. Alessandro Lenci, Marco Baroni, Giulia Cazzolli, Giovanna Marotta. 2013. BLIND: a set of semantic feature norms from the congenitally blind. Behavior Research Methods . [CrossRef] · DOI
  89. Tommaso Fornaciari, Massimo Poesio. 2013. Automatic deception detection in Italian court cases. Artificial Intelligence and Law . [CrossRef] · DOI
  90. Pierre Andrews, Juan Pane. 2013. Sense induction in folksonomies: a review. Artificial Intelligence Review . [CrossRef] · DOI
  91. JENNIFER THEWISSEN. 2013. Capturing L2 Accuracy Developmental Patterns: Insights From an Error-Tagged EFL Learner Corpus. The Modern Language Journal 97:S1, 77-101. [CrossRef] · DOI
  92. James Hunter, Yvonne Freer, Albert Gatt, Ehud Reiter, Somayajulu Sripada, Cindy Sykes. 2012. Automatic generation of natural language nursing shift summaries in neonatal intensive care: BT-Nurse. Artificial Intelligence in Medicine 56:3, 157-172. [CrossRef] · DOI · Paper page
  93. Ester Boldrini, Alexandra Balahur, Patricio Martínez-Barco, Andrés Montoyo. 2012. Using EmotiBlog to annotate and analyse subjectivity in the new textual genres. Data Mining and Knowledge Discovery 25:3, 603-634. [CrossRef] · DOI
  94. Guangchao Charles Feng. 2012. Factors affecting intercoder reliability: a Monte Carlo experiment. Quality & Quantity . [CrossRef] · DOI
  95. R. Hoehndorf, M. Dumontier, G. V. Gkoutos. 2012. Evaluation of research in biomedical ontologies. Briefings in Bioinformatics . [CrossRef] · DOI
  96. Gemma Boleda, Sabine Schulte im Walde, Toni Badia. 2012. Modeling Regular Polysemy: A Study on the Semantic Classification of Catalan Adjectives. Computational Linguistics 38:3, 575-616. [Abstract] [Full Text] [PDF] [PDF Plus] · DOI
  97. Philipp Spanger, Masaaki Yasuhara, Ryu Iida, Takenobu Tokunaga, Asuka Terai, Naoko Kuriyama. 2012. REX-J: Japanese referring expression corpus of situated dialogs. Language Resources and Evaluation 46:3, 461-491. [CrossRef] · DOI
  98. Marjorie McShane, Stephen Beale, Sergei Nirenburg, Bruce Jarrell, George Fantry. 2012. Inconsistency as a diagnostic tool in a society of intelligent agents. Artificial Intelligence in Medicine 55:3, 137-148. [CrossRef] · DOI
  99. An-Shou Cheng, Kenneth R. Fleischmann, Ping Wang, Emi Ishita, Douglas W. Oard. 2012. The role of innovation and wealth in the net neutrality debate: A content analysis of human values in congressional and FCC hearings. Journal of the American Society for Information Science and Technology 63:7, 1360-1373. [CrossRef] · DOI
  100. U. Hahn, K. B. Cohen, Y. Garten, N. H. Shah. 2012. Mining the pharmacogenomics literature-- a survey of the state of the art. Briefings in Bioinformatics 13:4, 460-494. [CrossRef] · DOI
  101. Christian M. Meyer, Iryna GurevychOntoWiktionary 131-161. [CrossRef] · DOI
  102. Manfred Stede, Andreas Peldszus. 2012. The role of illocutionary status in the usage conditions of causal connectives and in coherence relations. Journal of Pragmatics . [CrossRef] · DOI
  103. Amal Zouaq, Dragan Gasevic, Marek Hatala. 2011. Towards open ontology learning and filtering. Information Systems 36:7, 1064-1081. [CrossRef] · DOI
  104. Jon Patrick, Min Li. 2011. An Ontology for clinical questions about the contents of patient notes. Journal of Biomedical Informatics . [CrossRef] · DOI
  105. Kees van Deemter, Albert Gatt, Ielka van der Sluis, Richard Power. 2011. Generation of Referring Expressions: Assessing the Incremental Algorithm. Cognitive Science no-no. [CrossRef] · DOI
  106. Federica Cavicchio, Massimo Poesio. 2011. The Rovereto Emotion and Cooperation Corpus: a new resource to investigate cooperation and emotions. Language Resources and Evaluation . [CrossRef] · DOI
  107. Stefanie Dipper, Heike Zinsmeister. 2011. Annotating abstract anaphora. Language Resources and Evaluation . [CrossRef] · DOI
  108. Danielle Mowery, Janyce Wiebe, Shyam Visweswaran, Henk Harkema, Wendy W. Chapman. 2011. Building an automated SOAP classifier for emergency department reports. Journal of Biomedical Informatics . [CrossRef] · DOI
  109. Aleksandar Kovačević, Zora Konjović, Branko Milosavljević, Goran Nenadic. 2011. Mining methodologies from NLP publications: A case study in automatic terminology recognition. Computer Speech & Language . [CrossRef] · DOI
  110. Edgar Meij, Marc Bron, Laura Hollink, Bouke Huurnink, Maarten de Rijke. 2011. Mapping queries to the Linking Open Data cloud: A case study using DBpedia. Web Semantics: Science, Services and Agents on the World Wide Web . [CrossRef] · DOI
  111. Dilek Hakkani-Tür, Giuseppe RiccardiActive Learning 195-224. [CrossRef] · DOI
  112. Melanie J Martin. 2011. Reliability and type of consumer health documents on the World Wide Web: an annotation study. Journal of Biomedical Semantics 2:Suppl 3, S5. [CrossRef] · DOI
  113. Mike Thelwall, Kevan Buckley, Georgios Paltoglou. 2011. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology n/a-n/a. [CrossRef] · DOI · Paper page
  114. Gerhard Kremer, Marco Baroni. 2010. A set of semantic norms for German and Italian. Behavior Research Methods . [CrossRef] · DOI
  115. Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology 61:12, 2544-2558. [CrossRef] · DOI · Paper page
  116. Mark Stevenson, Yikun Guo. 2010. Disambiguation in the biomedical domain: The role of ambiguity type. Journal of Biomedical Informatics 43:6, 972-981. [CrossRef] · DOI
  117. Larry H. Smith, W. John Wilbur. 2010. Finding related sentence pairs in MEDLINE. Information Retrieval 13:6, 601-617. [CrossRef] · DOI
  118. Marta Recasens, M. Antònia Martí. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation 44:4, 315-345. [CrossRef] · DOI
  119. Emi Ishita, Douglas W. Oard, Kenneth R. Fleischmann, An-Shou Cheng, Thomas Clay Templeton. 2010. Investigating multi-label classification for human values. Proceedings of the American Society for Information Science and Technology 47:1, 1-4. [CrossRef] · DOI
  120. Serguei V.S. Pakhomov, Ted Pedersen, Bridget McInnes, Genevieve B. Melton, Alexander Ruggieri, Christopher G. Chute. 2010. Towards a framework for developing semantic relatedness reference standards. Journal of Biomedical Informatics . [CrossRef] · DOI
  121. Wilbert Spooren, Liesbeth Degand. 2010. Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory 6:2, 241-266. [CrossRef] · DOI
  122. Mark Stevenson, Yikun Guo. 2010. Disambiguation of ambiguous biomedical terms using examples generated from the UMLS Metathesaurus. Journal of Biomedical Informatics 43:5, 762-773. [CrossRef] · DOI
  123. Serguei V. S. Pakhomov, Glenn E. Smith, Dustin Chacon, Yara Feliciano, Neill Graff-Radford, Richard Caselli, David S. Knopman. 2010. Computerized Analysis of Speech and Language to Identify Psycholinguistic Correlates of Frontotemporal Lobar Degeneration. Cognitive and Behavioral Neurology 23:3, 165-177. [CrossRef] · DOI · Paper page
  124. Hannah Tipney, Lawrence HunterKnowledge-Driven Approaches to Genome-Scale Analysis 33-65. [CrossRef] 38. References 655-741. [CrossRef] · DOI
  125. BONNIE J. DORR, REBECCA J. PASSONNEAU, DAVID FARWELL, REBECCA GREEN, NIZAR HABASH, STEPHEN HELMREICH, EDUARD HOVY, LORI LEVIN, KEITH J. MILLER, TERUKO MITAMURA, OWEN RAMBOW, ADVAITH SIDDHARTHAN. 2010. Interlingual annotation of parallel text corpora: a new framework for annotation and evaluation. Natural Language Engineering 16:03, 197-243. [CrossRef] · DOI
  126. Beata Beigman Klebanov, Eyal Beigman. 2009. From Annotator Agreement to Noise Models. Computational Linguistics 35:4, 495-503. [Abstract] [PDF] [PDF Plus] · DOI
  127. Łukasz Dębowski. 2009. Valence extraction using EM selection and co-occurrence matrices. Language Resources and Evaluation 43:4, 301-327. [CrossRef] · DOI
  128. Sumithra Velupillai, Hercules Dalianis, Martin Hassel, Gunnar H. Nilsson. 2009. Developing a standard for de-identifying electronic patient records written in Swedish: Precision, recall and F-measure in a manual and computerized annotation trial. International Journal of Medical Informatics 78:12, e19-e26. [CrossRef] · DOI
  129. Gemma Boleda, Sabine Schulte im Walde, Toni Badia. 2008. An Analysis of Human Judgements on Semantic Classification of Catalan Adjectives. Research on Language and Computation 6:3-4, 247-271. [CrossRef] · DOI

Cited by

Sorted by citations

Authors
Artstein, R., & Poesio, M.
Affiliations
University of Essex (GB)
Corresponding author
  • Artstein, Ron
  • Poesio, Massimo
ORCID
Venue
Computational Linguistics
Publisher
MIT Press
Published
2008-12
Volume
34
Issue
4
Pages
555-596
ISSN
  • 0891-2017 (ISSN-L)
  • 1530-9312
DOI
10.1162/coli.07-034-r2
OpenAlex
W2141766660
Cited by
1,665
References
8
Citations by year
2026: 153 · 2025: 89 · 2024: 85 · 2023: 133 · 2022: 128 · 2021: 147 · 2020: 117 · 2019: 92 · 2018: 87 · 2017: 77 · 2016: 89 · 2015: 72 · 2014: 99 · 2013: 76 · 2012: 70
FWCI
56.33
Type
journal article
Language
en
Open access
Locations
  • Computational Linguistics (publishedVersion)
  • Open Access at Essex (University of Essex) (acceptedVersion)
  • Iris (University of Trento) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (GR/S76434/01)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
inter-coder agreement, Krippendorff's alpha, Cohen's kappa, corpus annotation, inter-annotator reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review