CoChat
Sign Up
Cited by 121Machine Learning and Data ClassificationReliability and Agreement in MeasurementComputational and Text Analysis Methods

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation

Open access
Plank, Barbara·2022·Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2022.emnlp-main.731
Open this paper in CoChat — free accountOpen PDF

Abstract

Human variation in labeling is often considered noise.Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics.However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers.In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward.This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation.However, few works consider all of these dimensions jointly; and existing research is fragmented.We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward.As datasets are becoming increasingly available, we hope that this synthesized view on the "problem" will lead to an open discussion on possible strategies to devise fundamentally new directions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.emnlp-main.731

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 10 references

Research works printed but not linked: 77

  1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? Perspective-aware mod- els to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896. · arXiv
  2. Cecilia Ovesdotter Alm. 2011. Subjective natural lan- guage problems: Motivations, applications, charac- terizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Compu- tational Linguistics: Human Language Technologies, pages 107-112.
  3. Dina Almanea and Massimo Poesio. 2022. The ARMIS dataset of misogyny in arabic tweets. In Proc. of LREC.
  4. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15-24.
  5. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computa- tional linguistics, 34(4):555-596.
  6. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Empirical Methods of Natural Language Processing: EMNLP 2022. Association for Computational Linguistics.
  7. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021a. Toward a perspectivist turn in ground truthing for predictive computing. arXiv preprint arXiv:2109.04270. · arXiv
  8. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poe- sio, Alexandra Uma, et al. 2021b. We need to con- sider disagreement in evaluation. In 1st Workshop on Benchmarking: Past, Present and Future, pages 15-21. Association for Computational Linguistics.
  9. Elisa Bassignana and Barbara Plank. 2022. CrossRE: A Cross-Domain Dataset for Relation Extraction. In Findings of the Association for Computational Lin- guistics: EMNLP 2022. Association for Computa- tional Linguistics.
  10. Eyal Beigman and Beata Beigman-Klebanov. 2009. Learning with annotation noise. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Confer- ence on Natural Language Processing of the AFNLP, pages 280-287, Suntec, Singapore. Association for Computational Linguistics.
  11. Beata Beigman Klebanov, Eyal Beigman, and Danie Diermaier. 2008. Analyzing disagreements. In Pro- ceedings of the Coling 2008 workshop on Human Judgements in Computational Linguistics, pages 2-7.
  12. Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Ko- rhonen, and Boris Katz. 2016. Anchoring and agree- ment in syntactic annotations. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2215-2224, Austin, Texas. Association for Computational Linguistics. · DOI
  13. Debopam Das, Manfred Stede, and Maite Taboada. 2017. The good, the bad, and the disagreement: Complex ground truth in rhetorical structure anal- ysis. In Proceedings of the 6th Workshop on Recent Advances in RST and Related Formalisms, pages 11- 19, Santiago de Compostela, Spain. Association for Computational Linguistics. · DOI
  14. Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
  15. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? the prag- matic complexity of veridicality assessment. Compu- tational Linguistics, 38(2):301-333. · DOI
  16. Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
  17. Emily Denton, Mark Díaz, Ian Kivlichan, Vinodku- mar Prabhakaran, and Rachel Rosen. 2021. Whose ground truth? accounting for individual and col- lective identities underlying dataset annotation. In NeurIPS workshop on Data-Centric AI.
  18. Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad Twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Compu- tational Linguistics: Technical Papers, pages 1169- 1179, Osaka, Japan. The COLING 2016 Organizing Committee.
  19. Liviu P. Dinu, Ioan-Bogdan Iordache, Ana Sabina Uban, and Marcos Zampieri. 2021. A computational ex- ploration of pejorative language in social media. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 3493-3498, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI
  20. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2018. Crowdsourcing semantic label propagation in relation classification. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 16-21, Brussels, Belgium. Association for Computa- tional Linguistics. · DOI
  21. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics (NAACL). Human Language Technologies, pages 1626-1644, Online. Association for Computational Linguistics.
  22. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics. · DOI
  23. R Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. Garbage in, garbage out? do machine learning ap- plication papers in social computing report where human-labeled training data comes from? In Pro- ceedings of the 2020 Conference on Fairness, Ac- countability, and Transparency, pages 325-336.
  24. Jacob Goldberger and Ehud Ben-Reuven. 2016. Train- ing deep neural-networks using a noise adaptation layer. ICLR 2017.
  25. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems, pages 1-19.
  26. Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. 2018a. Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pages 5836-5846.
  27. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018b. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527- 8537.
  28. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource an- notation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 291-297.
  29. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natu- ral language inference. Transactions of the Associa- tion for Computational Linguistics, 10:1081-1097.
  30. Shailza Jolly, Sandro Pezzelle, and Moin Nabi. 2021. EaSe: A diagnostic tool for VQA based on answer diversity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2407-2414, Online. Association for Computational Linguistics. · DOI
  31. Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted rasch measurement and multi- task deep learning: a hate speech application. arXiv preprint arXiv:2009.10277. · arXiv
  32. Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020. Cali- brated language model fine-tuning for in-and out- of-distribution data. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1326-1340, Online. As- sociation for Computational Linguistics. · DOI · Paper page
  33. Ranjay A Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A Shamma, Li Fei-Fei, and Michael S Bernstein. 2016. Embracing error to enable rapid crowdsourcing. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 3167-3179.
  34. John P Lalor, Hao Wu, and Hong Yu. 2017. Soft la- bel memorization-generalization for natural language inference. arXiv preprint arXiv:1702.08563. · arXiv
  35. Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering anno- tator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 81-90.
  36. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators' disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  37. Tong Liu, Christopher Homan, Cecilia Ovesdotter Alm, Megan Lytle, Ann Marie White, and Henry Kautz. 2016. Understanding discourse on work and job- related well-being in public social media. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1044-1053, Berlin, Germany. Association for Computational Linguistics. · DOI
  38. Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In International conference on intelligent text process- ing and computational linguistics, pages 171-189. Springer.
  39. Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation qual- ity in multi-label annotations. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 3659-3668, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  40. Héctor Martínez Alonso, Anders Johannsen, and Bar- bara Plank. 2016. Supersense tagging with inter- annotator disagreement. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 43-48, Berlin, Germany. Association for Computational Linguistics. · DOI
  41. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? col- lecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 4, pages 139-148.
  42. Clara Meister, Elizabeth Salesky, and Ryan Cotterell. 2020. Generalized entropy regularization or: There's nothing special about label smoothing. arXiv preprint arXiv:2005.00820. · arXiv
  43. Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. Obtaining Well Calibrated Proba- bilities Using Bayesian Binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial In- telligence, AAAI'15, page 2901-2907. AAAI Press.
  44. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP). Association for Computational Linguistics.
  45. Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable annota- tions. In SAD/CrowdBias@ HCOMP, pages 19-31.
  46. Rebecca J Passonneau, Ansaf Salleb-Aouissi, Vikas Bhardwaj, and Nancy Ide. 2010. Word sense annota- tion of polysemous words by multiple annotators. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).
  47. Silviu Paun, Ron Artstein, and Massimo Poesio. 2022. Statistical methods for annotation analysis. Synthesis Lectures on Human Language Technologies, 15(1):1- 217.
  48. Siyao Peng, Yang Janet Liu, and Amir Zeldes. 2022. Gcdt: A chinese rst treebank for multigenre and mul- tilingual discourse parsing. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the As- sociation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP).
  49. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human un- certainty makes classification more robust. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9616-9625.
  50. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014a. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 742-751. Associa- tion for Computational Linguistics. · DOI
  51. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014b. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 507-511.
  52. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking am- biguity into account. In Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76-83.
  53. Massimo Poesio, Jon Chamberlain, Silviu Paun, Jun- tao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778-1789, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI
  54. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI
  55. James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. O'Reilly Media.
  56. Ciyang Qing, Ulle Endriss, Raquel Fernández, and Justin Kruger. 2014. Empirical analysis of aggrega- tion methods for collective annotation. In COLING.
  57. Dennis Reidsma and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguis- tics, 34(3):319-326.
  58. Dennis Reidsma and Rieks op den Akker. 2008. Ex- ploiting 'subjective' annotations. In Proc. of the Workshop on Human Judgments in Computational Linguistics, pages 8-16. · DOI
  59. Paul Resnick, Yuqing Kong, Grant Schoenebeck, and Tim Weninger. 2021. Survey equivalence: A proce- dure for measuring classifier accuracy against human labels. arXiv preprint arXiv:2106.01254. · arXiv
  60. Filipe Rodrigues and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  61. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguistics. · DOI
  62. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics. · DOI
  63. David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 670-674, Online. Association for Computa- tional Linguistics. · DOI
  64. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy label- ers. In Proceedings of the 14th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, KDD '08, page 614-622, New York, NY, USA. Association for Computing Machinery. · DOI
  65. Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5716-5728, Florence, Italy. Association for Computational Linguistics. · DOI
  66. Rion Snow, Brendan O'connor, Dan Jurafsky, and An- drew Y Ng. 2008. Cheap and fast-but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254-263.
  67. Pia Sommerauer, Antske Fokkens, and Piek Vossen. 2020. Would you describe a leopard as yellow? eval- uating crowd-annotations with justified and informa- tive disagreement. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4798-4809, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics. · DOI
  68. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics. · DOI
  69. Alexandra Uma, Tommaso Fornaciari, Anca Dumi- trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021a. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338- 347, Online. Association for Computational Linguis- tics. · DOI
  70. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft-loss functions. In Proc. of HCOMP.
  71. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021b. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470.
  72. Bonnie Webber and Aravind Joshi. 2012. Discourse structure and computation: Past, present and future. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 42-54, Jeju Island, Korea. Association for Computational Linguistics.
  73. Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. 2021. Investigating annotator bias in abusive language datasets. In Proceedings of the International Conference on Recent Advances in Nat- ural Language Processing (RANLP 2021), pages 1515-1525, Held Online. INCOMA Ltd.
  74. Daniel Zeman. 2010. Hard problems of tagset conver- sion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources, pages 181-185.
  75. Mike Zhang and Barbara Plank. 2021. Cartography ac- tive learning. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 395- 406, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  76. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of anno- tation: From zero to many labels. arXiv preprint arXiv:2109.04408. · arXiv
  77. Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagree- ment in natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908-4915, Online. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

See all 121

Metadata

Authors
Plank, B.
Affiliations
Munich Center for Machine Learning (DE)
Corresponding author
Plank, Barbara
ORCID
Plank, Barbara
Venue
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2022
Pages
10671-10682
DOI
10.18653/v1/2022.emnlp-main.731
OpenAlex
W4385573849
Cited by
121
References
10
Citations by year
2026: 33 · 2025: 28 · 2024: 18 · 2023: 41
FWCI
12.03
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
HORIZON EUROPE European Research Council (101043235)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
noisy annotations, ground truth, annotator disagreement, annotated dataset, data quality, machine learning model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review