The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
Machine learning classifiers for human-facing tasks such as comment toxicity and misinformation often score highly on metrics such as ROC AUC but are received poorly in practice. Why this gap? Today, metrics such as ROC AUC, precision, and recall are used to measure technical performance; however, human-computer interaction observes that evaluation of human-facing systems should account for people’s reactions to the system. In this paper, we introduce a transformation that more closely aligns machine learning classification metrics with the values and methods of user-facing performance measures. The disagreement deconvolution takes in any multi-annotator (e.g., crowdsourced) dataset, disentangles stable opinions from noise by estimating intra-annotator consistency, and compares each test set prediction to the individual stable opinions from each annotator. Applying the disagreement deconvolution to existing social computing datasets, we find that current metrics dramatically overstate the performance of many human-facing machine learning tasks: for example, performance on a comment toxicity task is corrected from .95 to .73 ROC AUC.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“now outperform human judgment on many artifcial intelligence (AI) tasks such as image classifcation [21], the results for social computing tasks remain problematic.”
BackgroundProceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 1×
“Recent eforts seek to reduce bias [48]; make models more robust to novel situations [29]; help models better interpret culturally situated language and behav-”
BackgroundA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812 · cited 1×
“Recent eforts seek to reduce bias [48]; make models more robust to novel situations [29]; help models better interpret culturally situated language and behav-”
BackgroundFake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304 · cited 2×
“Unlike prior work, which focuses its critique on the ML architecture [62], we derive our claim by measuring properties of the datasets.”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Research has also sought to characterize best practices for designers and organizations developing such classifers [2, 4].”
BackgroundTest–retest reliability and validity of the Pittsburgh Sleep Quality Index in primary insomnia
Backhaus, J., Junghanns, K., Broocks, A., Riemann, D., & Hohagen, F. (2002) · Journal of Psychosomatic Research · doi:10.1016/s0022-3999(02)00330-6 · Cited by 1,756
Çömlekçi, M. F. (2019) · Kırklareli University Institutional Repository (Kırklareli University) · Cited by 1,607
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“Future work could evaluate classifcations using annotator-specifc utility values for each decision a classifer makes [35], and weight errors accordingly.”
MethodProceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities
Zhou, X., & Zafarani, R. (2021) · ACM Computing Surveys · doi:10.1145/3395046 · Cited by 1,184
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165 · cited 2×
“This dataset [61] is a classic dataset in crowdsourcing optimization, used to support machine”
BackgroundMethodSoftware Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 2×
“Research has also sought to characterize best practices for designers and organizations developing such classifers [2, 4].”
BackgroundA Basis for Analyzing Test-Retest Reliability
Guttman, L. (1945) · Psychometrika · doi:10.1007/bf02288892 · Cited by 1,122
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes
Wobbrock, J. O., Wilson, A. D., & Li, Y. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294238 · Cited by 822 · cited 1×
“In many cases, common metrics used for technical performance are already aligned with user-facing performance, as strong gesture recognition [70] or text entry prediction rates [64] also represent strong user feedback.”
BackgroundGillespie, T. (2019) · doi:10.12987/9780300235029 · Cited by 806
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777 · cited 1×
“tions by modeling their various sources [5, 19, 55].”
Fails, J. A., & Olsen, D. R. (2003) · Proceedings of the 8th international conference on Intelligent user interfaces · doi:10.1145/604045.604056 · Cited by 633
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter
Waseem, Z. (2016) · Proceedings of the First Workshop on NLP and Computational Social Science · doi:10.18653/v1/w16-5618 · Cited by 569 · cited 2×
“There are many potential sources of this disagreement: hate speech defnitions vary signifcantly by country [58], and feminists’ annotations difer from antiracism activists’ annotations which both difer from amateur annotations [69].”
BackgroundMethodGillespie, T. (2018) · Cited by 511
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk
Callison-Burch, C. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09 · doi:10.3115/1699510.1699548 · Cited by 412 · cited 1×
“veloping a metric that accounts for levels of disagreement directly remains challenging because of label uncertainty [36]: even wellintentioned labelers can make accidental errors, misunderstand questions, or even disagree with themselves later when asked the same question a second time [14].”
BackgroundNuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
Borkan, D., Dixon, L., Sorensen, J., Thain, N., & Vasserman, L. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317593 · Cited by 309 · cited 14×
“When we evaluate this model against a test set created using the disagreement deconvolution, it achieves a ROC AUC of .726, precision of .514, and recall of .499, far lower performance than the metrics computed against an aggregated test set suggest.”
BackgroundInfluentialCaplan, R., & Gillespie, T. (2020) · Social Media + Society · doi:10.1177/2056305120936636 · Cited by 283 · cited 1×
“YouTube creates classifers that decide which videos need to be demonetized [15].”
BackgroundDetecting Spam Bots in Online Social Networking Sites: A Machine Learning Approach
Wang, A. H. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-13739-6_25 · Cited by 257 · cited 1×
“Across tasks such as identifying toxic comments [31, 66], bot accounts [67], and misinformation [72], researchers and platforms [40] increasingly turn to ML to aid their eforts [32].”
Background#thyghgapp: Instagram Content Moderation and Lexical Variation in Pro-Eating Disorder Communities
Chancellor, S., Pater, J. A., Clear, T., Gilbert, E., & De Choudhury, M. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819963 · Cited by 255 · cited 1×
“Even well-intentioned platforms will continue to make highly-publicized mistakes [16] until they can better align metrics with reality.”
BackgroundFake News: A Survey of Research, Detection Methods, and Opportunities.Preprint
Zhou, X., & Zafarani, R. (2018) · arXiv (Cornell University) · Cited by 252
Data quality from crowdsourcing: a study of annotation selection criteria
Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009) · Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09 · doi:10.3115/1564131.1564137 · Cited by 251 · cited 1×
“veloping a metric that accounts for levels of disagreement directly remains challenging because of label uncertainty [36]: even wellintentioned labelers can make accidental errors, misunderstand questions, or even disagree with themselves later when asked the same question a second time [14].”
BackgroundDeVito, M. A., Birnholtz, J., Hancock, J. T., French, M., & Liu, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173694 · Cited by 244 · cited 1×
“Many are unaware of their existence [26], and many others hold informal folk theories as to how they operate [22].”
BackgroundCREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations
Mitra, T., & Gilbert, E. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i1.14625 · Cited by 240 · cited 1×
“only 2% of events had 100% agreement among annotators [50].”
BackgroundChallenges for Toxic Comment Classification: An In-Depth Error Analysis
van Aken, B., Risch, J., Krestel, R., & Löser, A. (2018) · Proceedings of the 2nd Workshop on Abusive Language Online (ALW2) · doi:10.18653/v1/w18-5105 · Cited by 235 · cited 1×
“Across tasks such as identifying toxic comments [31, 66], bot accounts [67], and misinformation [72], researchers and platforms [40] increasingly turn to ML to aid their eforts [32].”
BackgroundLe, J., Edmonds, A., Hester, V., & Biewald, L. (2010) · Cited by 232
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Chang, J. C., Amershi, S., & Kamar, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3026044 · Cited by 217 · cited 2×
“One general thrust of such research is to aid the user in providing accurate and useful labels, so that the resulting model is performant [17].”
BackgroundInherent Disagreements in Human Textual Inferences
Pavlick, E., & Kwiatkowski, T. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00293 · Cited by 214 · cited 2×
“Increasingly, researchers argue that these predictions should further embrace disagreement – rather than training on a single label per example, they argue that models should explicitly train on soft labels that represent the distribution of annotations found during the annotation process, and that their predictions s…”
BackgroundHuman Uncertainty Makes Classification More Robust
Peterson, J., Battleday, R., Griffiths, T., & Russakovsky, O. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00971 · Cited by 196 · cited 4×
“Increasingly, researchers argue that these predictions should further embrace disagreement – rather than training on a single label per example, they argue that models should explicitly train on soft labels that represent the distribution of annotations found during the annotation process, and that their predictions s…”
BackgroundMethodInfluentialConvolutional Neural Networks for Toxic Comment Classification
Georgakopoulos, S. V., Tasoulis, S. K., Vrahatis, A. G., & Plagianakos, V. P. (2018) · Proceedings of the 10th Hellenic Conference on Artificial Intelligence · doi:10.1145/3200947.3208069 · Cited by 194 · cited 1×
“Across tasks such as identifying toxic comments [31, 66], bot accounts [67], and misinformation [72], researchers and platforms [40] increasingly turn to ML to aid their efforts [32].”
BackgroundStreet-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · cited 2×
“Despite ostensibly strong reported test scores on these tasks [38, 62], the algorithms are roundly criticized for making obvious mistakes [1].”
BackgroundSelf-citationMeasuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
Roß, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., & Wojatzki, M. (2016) · DuEPublico (University of Duisburg-Essen) · doi:10.17185/duepublico/42132 · Cited by 172 · cited 1×
“It’s not clear that simply providing a defnition would improve agreement: a study of hate speech annotation found that providing annotators a defnition of hate speech did little to improve their agreement [57], with the authors claiming that a “clearer defnition is necessary in order to be able to train a reliable cla…”
BackgroundCrowdsourcing in medical research: concepts and applications
Tucker, J. D., Day, S., Tang, W., & Bayus, B. (2019) · PeerJ · doi:10.7717/peerj.6762 · Cited by 158
Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Kleek, M. G. V., & Shadbolt, N. (2017) · UCL Discovery (University College London) · Cited by 113
User Interaction with Word Prediction: The Effects of Prediction Quality
Trnka, K., McCaw, J., Yarrington, D., McCoy, K. F., & Pennington, C. (2009) · ACM Transactions on Accessible Computing · doi:10.1145/1497302.1497307 · Cited by 93 · cited 1×
“In many cases, common metrics used for technical performance are already aligned with user-facing performance, as strong gesture recognition [70] or text entry prediction rates [64] also represent strong user feedback.”
BackgroundDesigning Ground Truth and the Social Life of Labels
Muller, M., Wolf, C. T., Andres, J., Desmond, M., Joshi, N. N., Ashktorab, Z., Sharma, A., Brimijoin, K., Pan, Q., Duesterwald, E., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445402 · Cited by 83 · cited 1×
“incorrectly ignoring disagreement between people [49, 63].”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2019) · arXiv · doi:10.48550/arxiv.1901.10002 · Cited by 82 · cited 2×
“As described currently, the deconvolution assigns each annotator equal weight in the metric, a one-person-one-vote strategy that will disadvantage minority populations [63].”
BackgroundCrowdsourcing Ground Truth for Medical Relation Extraction
Dumitrache, A., Aroyo, L., & Welty, C. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3152889 · Cited by 81
Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation
Binns, R., Veale, M., Van Kleek, M., & Shadbolt, N. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67256-4_32 · Cited by 81
Squadbox: A Tool to Combat Email Harassment Using Friendsourced Moderation
Mahar, K., Zhang, A. X., & Karger, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174160 · Cited by 80 · cited 1×
“what behavior each interviewee felt constituted harassment [46].”
BackgroundLearning from Crowds by Modeling Common Confusions
Chu, Z., Ma, J., & Wang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16730 · Cited by 73
Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks
Kairam, S., & Heer, J. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2820016 · Cited by 72
Bhuiyan, M. M., Zhang, A. X., Sehat, C. M., & Mitra, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415164 · Cited by 67 · cited 1×
“This dataset aims to support the development of better tools for misinformation [9].”
MethodSentiment Analysis: It’s Complicated!
Kenyon-Dean, K., Ahmed, E. A., Fujimoto, S., Georges-Filteau, J., Glasz, C., Kaur, B., Lalande, A., Bhanderi, S., Belfer, R., Kanagasabai, N., et al. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1171 · Cited by 64 · cited 1×
“analysis [42, 69], we view this fltering technique as problematic because it removes the most challenging items and hides beliefs that the model’s end-users still hold, while ignoring the fact that”
MethodSalminen, J. O., Veronesi, F., Almerekhi, H., Jung, S.-G., & Jansen, B. J. J. (2018) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams.2018.8554954 · Cited by 61 · cited 1×
“There are many potential sources of this disagreement: hate speech defnitions vary signifcantly by country [58], and feminists’ annotations difer from antiracism activists’ annotations which both difer from amateur annotations [69].”
BackgroundResolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work
Schaekermann, M., Goh, J., Larson, K., & Law, E. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274423 · Cited by 54
The Test-Retest Reliability of Qualitative Data
Guttman, L. (1946) · Psychometrika · doi:10.1007/bf02288925 · Cited by 52 · cited 1×
“multiple times as attention checks [34].”
Sprout: Crowd-Powered Task Design for Crowdsourcing
Bragg, J., Mausam, M., & Weld, D. S. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242598 · Cited by 51 · cited 1×
“At the dataset collection stage, crowdsourcing researchers have proposed a number of methods that aim to resolve annotator disagreement either by making task designs clearer or relying on annotators to resolve disagreement among themselves [13, 17, 23, 47, 60].”
BackgroundEfficient Elicitation Approaches to Estimate Collective Crowd Answers
Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S. (., Kim, J., & Lasecki, W. S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359164 · Cited by 46
Capturing Ambiguity in Crowdsourcing Frame Disambiguation
Dumitrache, A., Aroyo, L. M., & Welty, C. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13330 · Cited by 37
Predicting the distribution of emotion perception: capturing inter-rater variability
Zhang, B., Essl, G., & Provost, E. M. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136792 · Cited by 34 · cited 2×
“Increasingly, researchers argue that these predictions should further embrace disagreement – rather than training on a single label per example, they argue that models should explicitly train on soft labels that represent the distribution of annotations found during the annotation process, and that their predictions s…”
BackgroundJigsaw Unintended Bias in Toxicity Classification
Do, Q. (2019) · Cited by 34
Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning
Patel, K., Fogarty, J. A., Landay, J. A., & Harrison, B. L. (2008) · Cited by 32
Self-citationWingIt: Efficient Refinement of Unclear Task Instructions
Manam, V. K. C., & Quinn, A. J. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13338 · Cited by 31 · cited 1×
“At the dataset collection stage, crowdsourcing researchers have proposed a number of methods that aim to resolve annotator disagreement either by making task designs clearer or relying on annotators to resolve disagreement among themselves [13, 17, 23, 47, 60].”
BackgroundRecent Research in Reinterview Procedures
Bailar, B. A. (1968) · Journal of the American Statistical Association · doi:10.1080/01621459.1968.11009221 · Cited by 30
The Role of Source, Headline and Expressive Responding in Political News Evaluation
Jakesch, M., Koren, M., Evtushenko, A., & Naaman, M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3306403 · Cited by 30
Learning to Predict Population-Level Label Distributions
Liu, T., Venkatachalam, A., Bongale, P. S., & Homan, C. M. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317082 · Cited by 30 · cited 2×
“Increasingly, researchers argue that these predictions should further embrace disagreement – rather than training on a single label per example, they argue that models should explicitly train on soft labels that represent the distribution of annotations found during the annotation process, and that their predictions s…”
BackgroundSquadbox: A Tool To Combat Online Harassment Using Friendsourced Moderation
Mahar, K., Zhang, A. X., & Karger, D. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3186498 · Cited by 28
Crowdsourcing Disagreement for Collecting Semantic Annotation
Dumitrache, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-18818-8_43 · Cited by 26 · cited 1×
“At the dataset collection stage, crowdsourcing researchers have proposed a number of methods that aim to resolve annotator disagreement either by making task designs clearer or relying on annotators to resolve disagreement among themselves [13, 17, 23, 47, 60].”
BackgroundThe Design and Analysis of Reinterview: An Overview
Forsman, G., & Schreiner, I. (2004) · Wiley series in probability and statistics · doi:10.1002/9781118150382.ch15 · Cited by 23
Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing
Augustin, A., Venanzi, M., Rogers, A., & Jennings, N. R. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/195 · Cited by 11 · cited 1×
“tions by modeling their various sources [5, 19, 55].”
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
Gebru, T. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3409559 · Cited by 11
Understanding Human and Predictive Moderation of Online Science Discourse
Lucas, E., Alm, C. O., & Bailey, R. (2019) · IEEE Western New York Image and Signal Processing Workshop (WNYISPW) · doi:10.1109/wnyipw.2019.8923109 · Cited by 5 · cited 1×
“Further, moderators disagreed with each other on 29% of removed posts [45].”
Background(2019)
Research works printed but not linked: 4
- Paul M Barrett. 2020. Who Moderates the Social Media Giants?Center for Business (2020).
- Geoffery C. Bowker and Susan Leigh Star. 2000. Sorting Things out: Classification and Its Consequences. MIT Press, Cambridge, MA, USA.
- Matt Gardner Yoav Artzi Victoria Basmova Jonathan Berant Ben Bogin Sihao Chen Pradeep Dasigi Dheeru Dua Yanai Elazar Ananth Gottumukkala 2020. Evaluating nlp models via contrast sets. (2020). arXiv:2004.02709 · arXiv
- Danny Pfeffermann and Calyampudi Radhakrishna Rao. 2009. Sample surveys: design, methods and applications. Elsevier.
Cited by
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
“(4) Uncertain environments: Some decision-making tasks, such as toxicity prediction, lack a “correct” answer [29].”
BackgroundSelf-citationDealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Davani, A. M., Díaz, M., & Prabhakaran, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00449 · Cited by 219
“Gordon et al. (2021) explore multiple annotators judgements to disentangle stable opinions from noise by estimating intra-annotator consistency.”
BackgroundTrauma-Informed Computing: Towards Safer Technology Experiences for All
Chen, J. X., McDonald, A., Zou, Y., Tseng, E., Roundy, K. A., Tamersoy, A., Schaub, F., Ristenpart, T., & Dell, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517475 · Cited by 163
“[64] developed the disagreement deconvolution, a method that aligns ML classification metrics with “the proportion of the overall population that would agree with each classification decision,” thereby providing a more human-centered approach to ML classifiers in line with the collaboration principle.”
MethodJury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 132
“While this majoritarian [55] procedure succeeded for many early machine learning tasks, it now runs aground on tasks where there is substantial disagreement on what the correct label ought to be [37].”
BackgroundInfluentialSelf-citationBuilding Human Values into Recommender Systems: An Interdisciplinary Synthesis
Stray, J., Halevy, A., Assar, P., Hadfield-Menell, D., Boutilier, C., Ashar, A., Bakalar, C., Beattie, L., Ekstrand, M., Leibowicz, C., et al. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3632297 · Cited by 122
“Also, many of these categories are both complex and politically contested (Bozarth et al., 2020; Martinez and Paluck, 2020) and raters often disagree, which complicates the evaluation of model accuracy (Gordon et al., 2021).”
Background
Metadata
- Authors
- Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., & Bernstein, M. S.
- Affiliations
- Stanford University (US)
- Apple (United States) (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-14
- OpenAlex
- W3163078977
- Cited by
- 101
- References
- 67
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 19 · 2023: 27 · 2022: 23 · 2021: 11
- FWCI
- 9.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- classifier evaluation metrics, crowdsourced annotation, misinformation detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.