Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks ranging from online comment toxicity to misinformation detection to medical diagnosis, different groups in society may have irreconcilable disagreements about ground truth labels. Supervised ML today resolves these label disagreements implicitly using majority vote, which overrides minority groups’ labels. We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier’s prediction. For example, a jury learning model for online toxicity might centrally feature women and Black jurors, who are commonly targets of online harassment. To enable jury learning, we contribute a deep learning architecture that models every annotator in a dataset, samples from annotators’ models to populate the jury, then runs inference to classify. Our architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent. A field evaluation finds that practitioners construct diverse juries that alter 14% of classification outcomes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Another line of work has sought to characterize best practices for designers and organizations developing such classifiers [6, 8].”
Background(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
The law of group polarization.
Sunstein, C. R. (2002) · Journal of Political Philosophy · doi:10.1111/1467-9760.00148 · Cited by 1,371
A Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Research works printed but not linked: 61
- 10.1145/2976749.2978318 · DOI · Paper page
- 10.1145/3351095.3372871 · DOI · Paper page
- Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896 [cs.CL] Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896 [cs.CL] · arXiv
- 10.1109/ICSE-SEIP.2019.00042 · DOI · Paper page
- 10.1145/3290605.3300773 · DOI · Paper page
- Paul M Barrett . 2020. Who Moderates the Social Media Giants?Center for Business ( 2020 ). Paul M Barrett. 2020. Who Moderates the Social Media Giants?Center for Business (2020).
- Michael S Bernstein , Margaret Levi , David Magnus , Betsy Rajala , Debra Satz , and Charla Waeiss . 2021 . ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021). Michael S Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz, and Charla Waeiss. 2021. ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021). · arXiv
- 10.1145/3242587.3242598 · DOI
- 10.1145/3025453.3026044 · DOI · Paper page
- John Joon Young Chung , Jean Y Song , Sindhu Kutty , Sungsoo Hong , Juho Kim , and Walter S Lasecki . 2019 . Efficient elicitation approaches to estimate collective crowd answers . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1–25. John Joon Young Chung, Jean Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–25.
- Aida Mostafazadeh Davani Mark Díaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719 [cs.CL] Aida Mostafazadeh Davani Mark Díaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719 [cs.CL] · arXiv
- Alexis De Toqueville . 1835. Democracy in America . New York : A Mentor Book from New American Library ( 1835 ). Alexis De Toqueville. 1835. Democracy in America. New York: A Mentor Book from New American Library (1835).
- Dennis J Devine , Laura D Clayton , Benjamin B Dunford , Rasmy Seying , and Jennifer Pryce . 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 ( 2001 ), 622. Dennis J Devine, Laura D Clayton, Benjamin B Dunford, Rasmy Seying, and Jennifer Pryce. 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 (2001), 622.
- 10.1002/pra2.2018.14505501013 · DOI
- 10.1007/978-3-319-18818-8_43 · DOI
- Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017). Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017). · arXiv
- Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018). Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018). · arXiv
- 10.1145/2702123.2702556 · DOI · Paper page
- 10.1145/3313831.3376293 · DOI · Paper page
- 10.1145/3394486.3409559 · DOI
- Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna Wallach Hal Daumé III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018). Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna Wallach Hal Daumé III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018). · arXiv
- 10.1145/3200947.3208069 · DOI · Paper page
- Tarleton Gillespie . 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media . Yale University Press . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.
- Mary L Gray and Siddharth Suri . 2019. Ghost work: How to stop Silicon Valley from building a new global underclass . Eamon Dolan Books . Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books.
- Moritz Hardt , Eric Price , and Nati Srebro . 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 ( 2016 ), 3315–3323. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016), 3315–3323.
- Khalid Haruna , Maizatul Akmar Ismail , Suhendroyono Suhendroyono , Damiasih Damiasih , Adi Cilik Pierewan , Haruna Chiroma , and Tutut Herawan . 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 ( 2017 ). https://doi.org/10.3390/app7121211 Khalid Haruna, Maizatul Akmar Ismail, Suhendroyono Suhendroyono, Damiasih Damiasih, Adi Cilik Pierewan, Haruna Chiroma, and Tutut Herawan. 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 (2017). https://doi.org/10.3390/app7121211 · DOI
- Tatsunori Hashimoto , Megha Srivastava , Hongseok Namkoong , and Percy Liang . 2018 . Fairness without demographics in repeated loss minimization . In International Conference on Machine Learning. PMLR , 1929–1938. Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning. PMLR, 1929–1938.
- Reid Hastie , Steven D Penrod , and Nancy Pennington . 2013. Inside the jury . Harvard University Press . Reid Hastie, Steven D Penrod, and Nancy Pennington. 2013. Inside the jury. Harvard University Press.
- 10.1145/2818048.2820016 · DOI · Paper page
- Kate Klonick . 2019 . The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression . Yale LJ 129 (2019), 2418 . Kate Klonick. 2019. The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression. Yale LJ 129(2019), 2418.
- 10.1145/3134697 · DOI
- Matthew Lease . 2011 . On quality control and machine learning in crowdsourcing . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Matthew Lease. 2011. On quality control and machine learning in crowdsourcing. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.
- 10.1214/19-AOS1828 · DOI
- David Timothy Lee , Ashish Goel , Tanja Aitamurto , and Helene Landemore . 2014 . Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions . In Second AAAI Conference on Human Computation and Crowdsourcing. David Timothy Lee, Ashish Goel, Tanja Aitamurto, and Helene Landemore. 2014. Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions. In Second AAAI Conference on Human Computation and Crowdsourcing.
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu , Siheon Lee , Alexandros Psomas , 2019 . WeBuildAI: Participatory framework for algorithmic governance . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1–35. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–35.
- Arend Lijphart . 1999. Patterns of democracy . Yale university press . Arend Lijphart. 1999. Patterns of democracy. Yale university press.
- 10.1145/3308560.3317082 · DOI
- V K Chaithanya Manam and Alexander J Quinn . 2018 . Wingit: Efficient refinement of unclear task instructions . In Sixth AAAI Conference on Human Computation and Crowdsourcing. VK Chaithanya Manam and Alexander J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing.
- Erwan Moreau , Carl Vogel , and Marguerite Barry . 2019. A paradigm for democratizing artificial intelligence research . In Innovations in Big Data Mining and Embedded Knowledge . Springer , 137–166. Erwan Moreau, Carl Vogel, and Marguerite Barry. 2019. A paradigm for democratizing artificial intelligence research. In Innovations in Big Data Mining and Embedded Knowledge. Springer, 137–166.
- Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200 [cs.CL] Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200 [cs.CL] · DOI · Paper page
- Kayur Patel James Fogarty James A Landay and Beverly L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563–1566. Kayur Patel James Fogarty James A Landay and Beverly L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563–1566.
- 10.18653/v1/2021.law-1.14 · DOI · Paper page
- 10.1177/019251219201300107 · doi:10.1177/019251219201300107
- Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947 [cs.CL] Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947 [cs.CL] · arXiv
- 10.1145/3359178 · DOI · Paper page
- 10.1145/3313831.3376506 · DOI · Paper page
- 10.1145/3274423 · DOI · Paper page
- 10.1145/3476058 · DOI · Paper page
- 10.1145/3411764.3445742 · DOI
- 10.1145/1401890.1401965 · DOI · Paper page
- 10.2139/ssrn.199668 · DOI · Paper page
- 10.5555/3298023.3298188 · doi:10.5555/3298023.3298188
- 10.7717/peerj.6762 · DOI · Paper page
- Betty van Aken Julian Risch Ralf Krestel and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018). Betty van Aken Julian Risch Ralf Krestel and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018). · DOI · Paper page
- 10.1007/978-3-642-13739-6_25 · DOI · Paper page
- 10.5555/3367471.3367557 · doi:10.5555/3367471.3367557
- 10.1145/3442381.3450078 · DOI · Paper page
- Tian Wang Yuri M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021). Tian Wang Yuri M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021). · arXiv
- 10.1145/3357236.3395528 · DOI · Paper page
- 10.1145/3379337.3415858 · DOI · Paper page
- 10.1145/3136755.3136792 · DOI
Cited by
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256
Karinshak, E., Liu, S. X., Park, J. S., & Hancock, J. T. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579592 · Cited by 167
Self-citationLi, Z., Zhu, H., Lu, Z., & Yin, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.647 · Cited by 139
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
Plank, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.731 · Cited by 121
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
“…proposed to leverage non-aggregated data to train models adapted to different users, in what they call “humancentered approach”; Sudre et al. (2019), Gordon et al. (2022) and Guan et al. (2018) proposed multi-task approaches to deal with observer variability and dissenting voices, showing how…”
BackgroundLaMP: When Large Language Models Meet Personalization
Salemi, A., Mysore, S., Bendersky, M., & Zamani, H. (2023) · arXiv · doi:10.48550/arxiv.2304.11406 · Cited by 109
“[36] examines the application of personalization methods to modeling annotators in a classification task reliant on modeling social norms – making an important connection between personalization and an emerging body of work on accommodating human label variation in NLP research [14, 35, 40].”
BackgroundMethodLaMP: When Large Language Models Meet Personalization
Salemi, A., Mysore, S., Bendersky, M., & Zamani, H. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.399 · Cited by 109
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 103
Challenges and Strategies in Cross-Cultural NLP
Hershcovich, D., Frank, S., Lent, H., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Cabello Piqueras, L., Chalkidis, I., Cui, R., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.482 · Cited by 99
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, E., & Lampe, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533135 · Cited by 92
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“…also be improved by algorithms that explicitly model multiple evaluators [Gordon et al., 2021, Davani et al., 2022, Daniels-Koch and Freedman, 2022, Gordon et al., 2022, Barnett et al., 2023], that tune models to individuals [Kumar et al., 2021], or that use more sophisticated aggregation…”
BackgroundCrowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
The AI life cycle: a holistic approach to creating ethical AI for health decisions
Ng, M. Y., Kapur, S., Blizinsky, K. D., & Hernandez-Boussard, T. (2022) · Nature Medicine · doi:10.1038/s41591-022-01993-y · Cited by 71
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
A hunt for the Snark: Annotator Diversity in Data Practices
Kapania, S., Taylor, A. S., & Wang, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580645 · Cited by 50
“The first step to manage annotator subjectivities would be to acknowledge that data production is fundamentally a collective and interpretive task, and there are likely cases where individual annotators will not conform with the ‘majority perspective’ [35].”
BackgroundTowards Measuring the Representation of Subjective Global Opinions in Language Models
Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16388 · Cited by 48
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.00955 · Cited by 44
A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Salimzadeh, S., He, G., & Gadiraju, U. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3592959 · Cited by 42
“Researchers have shown that explanations can effectively inform mental models of humans and improve their understanding, especially for laypeople [35, 36, 44, 90].”
BackgroundEveryday Uncertainty: How Blind People Use GenAI Tools for Information Access
Tang, X., Abdolrahmani, A., Gergle, D., & Piper, A. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713433 · Cited by 30
Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making
Guerdan, L., Coston, A., Wu, Z. S., & Holstein, K. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594036 · Cited by 28
AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War
Humphreys, D. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00928-y · Cited by 26
“Labellers are often selected to align with organisational values, one sug - gestion to counter this bias is of a jury learning model, to integrate dissenting voices in the training stage (Gordon et al., 2022 ).”
Are Large Language Models Consistent over Value-laden Questions?
Moore, J., Deshpande, T., & Yang, D. (2024) · arXiv · doi:10.48550/arxiv.2407.02996 · Cited by 12
The Value of Disagreement in AI Design, Evaluation, and Alignment
Fazelpour, S., & Fleisher, W. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732146 · Cited by 10
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics
Orlikowski, M., Röttger, P., Cimiano, P., & Hovy, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2023.acl-short.88 · Cited by 9
“Since human label variation is relevant to a wide range of NLP tasks, recent research has begun to model individual anno-tator behaviour, rather than predicting aggregated labels (Davani et al., 2022; Gordon et al., 2022).”
BackgroundMethodResultInfluentialThe Role of Expertise in Effectively Moderating Harmful Social Media Content
Abdelkadir, N. A., Yang, T., Kapania, S., Estefanos, M., Gebrekidan, F. B., Zelalem, Z., Ali, M., Berhe, R., Baker, D., Talat, Z., et al. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714010 · Cited by 5
We're Afraid Language Models Aren't Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.14399 · Cited by 3
Cripping AI: Reimagining AI Through Lived Disability Experiences
Tang, X., Lin, T.-A., Li, J., & Wu, S. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806744 · Cited by 2
Metadata
- Authors
- Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S.
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-19
- OpenAlex
- W4225006216
- Cited by
- 132
- References
- 28
- FWCI
- 28.76
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.