Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks
Open accessAbstract
Annotators' disagreement in linguistic data has been recently the focus of multiple initiatives aimed at raising awareness on issues related to 'majority voting' when aggregating diverging annotations.Disagreement can indeed reflect different aspects of linguistic annotation, from annotators' subjectivity to sloppiness or lack of enough context to interpret a text.In this work we first propose a taxonomy of possible reasons leading to annotators' disagreement in subjective tasks.Then, we manually label part of a Twitter dataset for offensive language detection in English following this taxonomy, identifying how the different categories are distributed.Finally we run a set of experiments aimed at assessing the impact of the different types of disagreement on classification performance.In particular, we investigate how accurately tweets belonging to different categories of disagreement can be classified as offensive or not, and how injecting data with different types of disagreement in the training set affects performance.We also perform offensive language detection as a multi-task framework, using disagreement classification as an auxiliary task.Warning: This paper contains examples that may be offensive or upsetting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In real scenarios, annotators may disagree for a variety of reasons, not all of which relate to a subpopulation identity.
“In real scenarios, annotators may disagree for a variety of reasons (Sandri et al., 2023), not all of which relate to a subpopulation identity.”
From NUTMEG: Separating Signal From Noise in Annotator Disagreement (2025) · cites this paper as Sandri et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter
Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Pardo, F. M. R., Rosso, P., & Sanguinetti, M. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2007 · Cited by 889 · see citing works
The Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817 · see citing works
On the psycholinguistics of sarcasm.
Gibbs, R. W. (1986) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.115.1.3 · Cited by 633 · see citing works
Meaning in Language. An Introduction to Semantics and Pragmatics
Ardila, J. A. G. (2011) · Journal of Pragmatics · doi:10.1016/j.pragma.2011.03.014 · Cited by 536 · see citing works
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., & Smith, N. A. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2022.naacl-main.431 · Cited by 463 · see citing works
Levinson, S. C. (1987) · Journal of Linguistics · doi:10.1017/s0022226700011324 · Cited by 425 · see citing works
Ambiguity, polysemy, and vagueness
Tuggy, D. (1993) · Cognitive Linguistics · doi:10.1515/cogl.1993.4.3.273 · Cited by 400 · see citing works
Swearing: A Cross-Cultural Linguistic Study
Ljung, M. (2010) · Palgrave Macmillan eBooks · doi:10.1057/9780230292376 · Cited by 316 · see citing works
A coding scheme for question–response sequences in conversation
Stivers, T., & Enfield, N. J. (2010) · Journal of Pragmatics · doi:10.1016/j.pragma.2010.04.002 · Cited by 280 · see citing works
Learning Whom to Trust with MACE
Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., & Hovy, E. H. (2013) · Archivio istituzionale della Ricerca - Bocconi · Cited by 276 · see citing works
Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation
Aroyo, L. M., & Welty, C. (2015) · AI Magazine · doi:10.1609/aimag.v36i1.2564 · Cited by 248 · see citing works
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 237 · see citing works
Learning from Disagreement: A Survey
Uma, A., Fornaciari, T., Hovy, D., Paun, S., Plank, B., & Poesio, M. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12752 · Cited by 154 · see citing works
The reliability of anaphoric annotation, reconsidered
Poesio, M., & Artstein, R. (2005) · doi:10.3115/1608829.1608840 · Cited by 124 · see citing works
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
Plank, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.731 · Cited by 121 · see citing works
Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach
Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103 · see citing works
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks
Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.13 · Cited by 97 · see citing works
Deep Dominance - How to Properly Compare Deep Neural Models
Dror, R., Shlomov, S., & Reichart, R. (2019) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p19-1266 · Cited by 96 · see citing works
Linguistically debatable or just plain wrong?
Plank, B., Hovy, D., & Søgaard, A. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2083 · Cited by 86 · see citing works
D’hondt, S., Östman, J., & Verschueren, J. (2009) · Handbook of pragmatics highlights · doi:10.1075/hoph.4 · Cited by 85 · see citing works
We Need to Consider Disagreement in Evaluation
Basile, V., Fell, M., Fornaciari, T., Hovy, D., Paun, S., Plank, B., Poesio, M., & Uma, A. (2021) · doi:10.18653/v1/2021.bppf-1.3 · Cited by 80 · see citing works
Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP
van der Goot, R., Üstün, A., Ramponi, A., Sharaf, I., & Plank, B. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.22 · Cited by 73 · see citing works
Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., & Poesio, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.204 · Cited by 66 · see citing works
Sentiment Analysis: It’s Complicated!
Kenyon-Dean, K., Ahmed, E. A., Fujimoto, S., Georges-Filteau, J., Glasz, C., Kaur, B., Lalande, A., Bhanderi, S., Belfer, R., Kanagasabai, N., et al. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1171 · Cited by 64 · see citing works
Impact of politically biased data on hate speech classification
Wich, M., Bauer, J., & Groh, G. (2020) · Workshop on Abusive Language Online · doi:10.18653/v1/2020.alw-1.7 · Cited by 54 · see citing works
Exploiting 'subjective' annotations
Reidsma, D., & den Akker, R. O. (2008) · doi:10.3115/1611628.1611631 · Cited by 53 · see citing works
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement
Leonardelli, E., Menini, S., Aprosio, A. P., Guerini, M., & Tonelli, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.822 · Cited by 51 · see citing works
Self-citationA Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation
Poesio, M., Chamberlain, J., Paun, S., Yu, J., Uma, A., & Kruschwitz, U. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1176 · Cited by 45 · see citing works
Jamison, E., & Gurevych, I. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1035 · Cited by 43 · see citing works
Do You Really Want to Hurt Me? Predicting Abusive Swearing in Social Media
Pamungkas, E. W., Basile, V., & Patti, V. (2020) · Institutional Research Information System University of Turin (University of Turin) · Cited by 41 · see citing works
Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing
Schwartz, R., Abend, O., Reichart, R., & Rappoport, A. (2011) · Cited by 41 · see citing works
An Optimal Transportation Approach for Assessing Almost Stochastic Order
del Barrio, E., Cuesta-Albertos, J. A., & Matrán, C. (2018) · Studies in systems, decision and control · doi:10.1007/978-3-319-73848-2_3 · Cited by 39 · see citing works
Klebanov, B. B., Beigman, E., & Diermeier, D. A. (2008) · doi:10.3115/1611628.1611630 · Cited by 35 · see citing works
TSNLP - Test Suites for Natural Language ProcessingPreprint
Lehmann, S., Oepen, S., Regnier-Prost, S., Netter, K., Lux-Pogodalla, V., Klein, J., Falkedal, K., Fouvry, F., Estival, D., Dauphin, E., et al. (1996) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9607018 · Cited by 33 · see citing works
Investigating Reasons for Disagreement in Natural Language Inference
Jiang, N., & de Marneffe, M.-C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00523 · Cited by 32 · see citing works
A New Measure of Polarization in the Annotation of Hate Speech
Akhtar, S., Basile, V., & Patti, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-35166-3_41 · Cited by 29 · see citing works
Features or Spurious Artifacts? Data-centric Baselines for Fair and Robust Hate Speech Detection
Ramponi, A., & Tonelli, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.221 · Cited by 22 · see citing works
Self-citationDifficult Cases: From Data to Learning, and Back
Klebanov, B. B., & Beigman, E. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-2064 · Cited by 17 · see citing works
Sang, Y., & Stanton, J. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96957-8_36 · Cited by 15 · see citing works
Truth in Disagreement: Crowdsourcing Labeled Data for Natural Language Processing
Dumitrache, A. (2019) · Data Archiving and Networked Services (DANS) · Cited by 13 · see citing works
Predicting word sense annotation agreement
Alonso, H. M., Johannsen, A., de Lacalle, O. L., & Agirre, E. (2015) · doi:10.18653/v1/w15-2711 · Cited by 9 · see citing works
Ramponi, A., & Leonardelli, E. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.42 · Cited by 4 · see citing works
Self-citationEditorial: Bias, Subjectivity and Perspectives in Natural Language Processing
Basile, V., Caselli, T., Balahur, A., & Ku, L. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.926435 · Cited by 4 · see citing works
Application and Analysis of a Multi-layered Scheme for Irony on the Italian Twitter Corpus TWITTIRÒ
Cignarella, A. T., Bosco, C., Patti, V., & Lai, M. (2018) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/582d597s9i68
Donaldson, T., & Lepore, E. (n.d.) · Routledge Companion to Philosophy of Language · doi:10.4324/9780203206966.ch1_10
Storey, K. (1995) · The International Journal of Speech, Language and the Law · doi:10.1558/ijsll.v2i1.74
Flagged first, then by citations
Research works printed but not linked: 3
- Gavin Abercrombie, Valerio Basile, Sara Tonelli, Ver- ena Rieser, and Alexandra Uma, editors. 2022. Pro- ceedings of the 1st Workshop on Perspectivist Ap- proaches to NLPerspectives@LREC 2022, Marseille, France, 20th June 2022. European Language Re- sources Association.
- S. C. Levinson. 2004. Deixis. In The handbook of pragmatics, pages 97-121.
- Dennis Ulmer, Christian Hardmeier, and Jes Frellsen. 2022. deep-significance: Easy and meaningful sig- nifcance testing in the age of neural networks. In ML Evaluation Standards Workshop at the Tenth Interna- tional Conference on Learning Representations.
Cited by
Sorted by citations
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47 · see citing works
SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)
Leonardelli, E., Abercrombie, G., Almanea, D., Basile, V., Fornaciari, T., Plank, B., Rieser, V., Uma, A., & Poesio, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.314 · Cited by 31 · see citing works
Self-citationLee, N., Jung, C., Myung, J., Jin, J., Camacho-Collados, J., Kim, J., & Oh, A. H. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.236 · Cited by 30 · see citing works
Pei, J., & Jurgens, D. (2023) · Linguistic Annotation Workshop · doi:10.18653/v1/2023.law-1.25 · Cited by 25 · see citing works
Chou, H.-C., Goncalves, L., Leem, S.-G., Salman, A. N., Lee, C.-C., & Busso, C. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3411290 · Cited by 19 · see citing works
Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models
Palla, K., Redondo-García, J. L., Hauff, C., Fabbri, F., Damianou, A., Lindström, H., Taber, D., & Lalmas, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732054 · Cited by 19 · see citing works
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains
Jacovi, A., Bitton, Y., Bohnet, B., Herzig, J., Honovich, O., Tseng, M., Collins, M., Aharoni, R., & Geva, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.254 · Cited by 14 · see citing works
Can Large Language Models Capture Dissenting Human Voices?
Lee, N., An, N., & Thorne, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.278 · Cited by 14 · see citing works
Plurals: A System for Guiding LLMs via Simulated Social Ensembles
Ashkinaze, J., Fry, E., Edara, N., Gilbert, É., & Budak, C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713675 · Cited by 12 · see citing works
AI Alignment at Your Discretion
Buyl, M., Khalaf, H., Verdun, C. M., Paes, L. M., Machado, C. C. V., & Calmon, F. P. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732194 · Cited by 8 · see citing works
Xu, S., T.y.s.s, S., Ichim, O., Risini, I., Plank, B., & Grabmair, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.594 · Cited by 8 · see citing works
Federated Learning for Exploiting Annotators’ Disagreements in Natural Language Processing
Rodríguez-Barroso, N., Martínez‐Cámara, E., Camacho-Collados, J., Luzón, M. V., & Herrera, F. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00664 · Cited by 8 · see citing works
Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology
Ruggeri, F., Misino, E., Muti, A., Korre, K., Torroni, P., & Barrón‐Cedeño, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.804 · Cited by 6 · see citing works
Evaluating medical AI systems in dermatology under uncertain ground truth
Stutz, D., Cemgil, A. T., Roy, A. G., Matejovicova, T., Barsbey, M., Strachan, P., Schaekermann, M., Freyberg, J., Rikhye, R. V., Freeman, B., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103556 · Cited by 6 · see citing works
Villate-Castillo, G., Ser, J. D., & Sanz, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130542 · Cited by 5 · see citing works
See the words through my eyes: The role of personal traits in abusive language detection
Yao, T., Binnewies, S., Foo, E., & Alavi, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127188 · Cited by 4 · see citing works
Rizvi, N., Flores, A. M., Rizvi, M., Ousidhoum, N. D., & Munyaka, I. N. S. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757478 · Cited by 3 · see citing works
Heinisch, P., Orlikowski, M., Romberg, J., & Cimiano, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.687 · Cited by 3 · see citing works
Khadar, M., Runningen, D., Tang, J., Chancellor, S., & Kaur, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757707 · Cited by 2 · see citing works
Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines
Sung, Y. Y., Boyd-Graber, J. L., & Hassan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1010 · Cited by 2 · see citing works
Using Annotator Labels Instead of Golden Labels for Fine Emotion Detection
Lu, A. L. H., & Iwaihara, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0865-2_19 · Cited by 1 · see citing works
NUTMEG: Separating Signal From Noise in Annotator Disagreement
Ivey, J., Gauch, S., & Jurgens, D. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.144
From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis
Wan, R., Wang, H., Huang, T.-H. K., & Gao, J. (2025) · Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) · doi:10.18653/v1/2025.hcinlp-1.20
Leonardelli, E., & Casula, C. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.261
Self-citationUnraveling Disagreement Constituents in Hateful Speech
Rizzi, G., Astorino, A., Rosso, P., & Fersini, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_3
Fujihara, T., Ito, A., & Nose, T. (2024) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3711542.3711565
Mohtaj, S. (2025) · T-labs series in telecommunication services · doi:10.1007/978-3-032-00850-3_2
SHAP-RC: A Framework for Explaining Annotator Disagreement in Sexism Detection
Sawant, M., Younus, A., Caton, S. J., & Qureshi, M. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08333-3_10
Advancing Team Conflict Detection by Leveraging Multifeature Embeddings, LMs, and LLMs
Verma, A., Yadav, P., & Kankanhalli, A. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3620421
Evaluating Toxicity Understanding of LLM Agents
Mothilal, R. K., Ahmed, S. I., & Guha, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00121
A Linguistic Analysis of Human and LLM Annotations on English Sentence Difficulty
Han, Y., Heo, I., & Jung, S. (2026) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp68355.2026.00040
FI-RC: Analysing Annotator Disagreement through a Culturally Aware LexiconPreprint
Sawant, M., Caton, S. J., & Qureshi, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-9236082/v1
Indriyani, N., & Ratnasari, C. I. (2026) · Journal of Computers and Digital Business · doi:10.56427/jcbd.v5i2.1009
Belay, T. D., Ahmad, I. S., Abdulmumin, I., Ayele, A. A., Gelbukh, A. F., Ricárdez-Vázquez, E., Kolesnikova, O., Muhammad, S. H., & Yimam, S. M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.731
- Authors
- Sandri, M., Leonardelli, E., Tonelli, S., & Jezek, E.
- Affiliations
- University of Pavia (IT)
- Fondazione Bruno Kessler (IT)
- Venue
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Published
- 2023
- Pages
- 2428-2441
- OpenAlex
- W4386566656
- Cited by
- 31
- References
- 44
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 6 · 2023: 7
- FWCI
- 4.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- annotator disagreement, linguistic annotation, offensive language detection, Twitter, majority voting, annotation quality, multi-task learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
