CoChat

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

Open access
Gordon, Mitchell L., Lam, Michelle S., Park, Joon Sung, Patel, Kayur, Hancock, Jeff, Hashimoto, Tatsunori, Bernstein, Michael S. · 2022 · CHI Conference on Human Factors in Computing Systems
Cited by 132

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Abstract

Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks ranging from online comment toxicity to misinformation detection to medical diagnosis, different groups in society may have irreconcilable disagreements about ground truth labels. Supervised ML today resolves these label disagreements implicitly using majority vote, which overrides minority groups’ labels. We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier’s prediction. For example, a jury learning model for online toxicity might centrally feature women and Black jurors, who are commonly targets of online harassment. To enable jury learning, we contribute a deep learning architecture that models every annotator in a dataset, samples from annotators’ models to populate the jury, then runs inference to classify. Our architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent. A field evaluation finds that practitioners construct diverse juries that alter 14% of classification outcomes.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3491102.3502004

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 22 references

Research works printed but not linked: 67

  1. 10.1145/2976749.2978318 · DOI · Paper page
  2. 10.1145/3351095.3372871 · DOI · Paper page
  3. Alekh Agarwal , Alina Beygelzimer , Miroslav Dudík , John Langford , and Hanna Wallach . 2018 . A reductions approach to fair classification . In International Conference on Machine Learning. PMLR, 60–69 . Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60–69.
  4. Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896 [cs.CL] Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896 [cs.CL] · arXiv
  5. 10.1109/ICSE-SEIP.2019.00042 · DOI · Paper page
  6. 10.1145/3290605.3300773 · DOI · Paper page
  7. Paul M Barrett . 2020. Who Moderates the Social Media Giants?Center for Business ( 2020 ). Paul M Barrett. 2020. Who Moderates the Social Media Giants?Center for Business (2020).
  8. Michael S Bernstein , Margaret Levi , David Magnus , Betsy Rajala , Debra Satz , and Charla Waeiss . 2021 . ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021). Michael S Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz, and Charla Waeiss. 2021. ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021). · arXiv
  9. 10.1145/3242587.3242598 · DOI
  10. Robyn Caplan and Tarleton Gillespie . 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 ( 2020 ), 2056305120936636. Robyn Caplan and Tarleton Gillespie. 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 (2020), 2056305120936636.
  11. 10.1145/3025453.3026044 · DOI · Paper page
  12. John Joon Young Chung , Jean Y Song , Sindhu Kutty , Sungsoo Hong , Juho Kim , and Walter S Lasecki . 2019 . Efficient elicitation approaches to estimate collective crowd answers . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1–25. John Joon Young Chung, Jean Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–25.
  13. Sasha Costanza-Chock . 2020. Design justice: Community-led practices to build the worlds we need . The MIT Press . Sasha Costanza-Chock. 2020. Design justice: Community-led practices to build the worlds we need. The MIT Press.
  14. Aida Mostafazadeh Davani Mark Díaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719 [cs.CL] Aida Mostafazadeh Davani Mark Díaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719 [cs.CL] · arXiv
  15. Alexis De Toqueville . 1835. Democracy in America . New York : A Mentor Book from New American Library ( 1835 ). Alexis De Toqueville. 1835. Democracy in America. New York: A Mentor Book from New American Library (1835).
  16. Dennis J Devine , Laura D Clayton , Benjamin B Dunford , Rasmy Seying , and Jennifer Pryce . 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 ( 2001 ), 622. Dennis J Devine, Laura D Clayton, Benjamin B Dunford, Rasmy Seying, and Jennifer Pryce. 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 (2001), 622.
  17. 10.1002/pra2.2018.14505501013 · DOI
  18. 10.1007/978-3-319-18818-8_43 · DOI
  19. Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017). Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017). · arXiv
  20. Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018). Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018). · arXiv
  21. 10.1145/2702123.2702556 · DOI · Paper page
  22. 10.1145/3313831.3376293 · DOI · Paper page
  23. 10.1145/3394486.3409559 · DOI
  24. Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna Wallach Hal Daumé III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018). Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna Wallach Hal Daumé III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018). · arXiv
  25. 10.1145/3200947.3208069 · DOI · Paper page
  26. Tarleton Gillespie . 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media . Yale University Press . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.
  27. Mary L Gray and Siddharth Suri . 2019. Ghost work: How to stop Silicon Valley from building a new global underclass . Eamon Dolan Books . Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books.
  28. Moritz Hardt , Eric Price , and Nati Srebro . 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 ( 2016 ), 3315–3323. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016), 3315–3323.
  29. Khalid Haruna , Maizatul Akmar Ismail , Suhendroyono Suhendroyono , Damiasih Damiasih , Adi Cilik Pierewan , Haruna Chiroma , and Tutut Herawan . 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 ( 2017 ). https://doi.org/10.3390/app7121211 Khalid Haruna, Maizatul Akmar Ismail, Suhendroyono Suhendroyono, Damiasih Damiasih, Adi Cilik Pierewan, Haruna Chiroma, and Tutut Herawan. 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 (2017). https://doi.org/10.3390/app7121211 · DOI
  30. Tatsunori Hashimoto , Megha Srivastava , Hongseok Namkoong , and Percy Liang . 2018 . Fairness without demographics in repeated loss minimization . In International Conference on Machine Learning. PMLR , 1929–1938. Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning. PMLR, 1929–1938.
  31. Reid Hastie , Steven D Penrod , and Nancy Pennington . 2013. Inside the jury . Harvard University Press . Reid Hastie, Steven D Penrod, and Nancy Pennington. 2013. Inside the jury. Harvard University Press.
  32. 10.1145/2818048.2820016 · DOI · Paper page
  33. Kate Klonick . 2019 . The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression . Yale LJ 129 (2019), 2418 . Kate Klonick. 2019. The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression. Yale LJ 129(2019), 2418.
  34. 10.1145/3134697 · DOI
  35. Deepak Kumar Patrick Gage Kelley Sunny Consolvo Joshua Mason Elie Bursztein Zakir Durumeric Kurt Thomas and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. arXiv preprint arXiv:2106.04511(2021). Deepak Kumar Patrick Gage Kelley Sunny Consolvo Joshua Mason Elie Bursztein Zakir Durumeric Kurt Thomas and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. arXiv preprint arXiv:2106.04511(2021). · arXiv
  36. Matthew Lease . 2011 . On quality control and machine learning in crowdsourcing . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Matthew Lease. 2011. On quality control and machine learning in crowdsourcing. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.
  37. 10.1214/19-AOS1828 · DOI
  38. David Timothy Lee , Ashish Goel , Tanja Aitamurto , and Helene Landemore . 2014 . Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions . In Second AAAI Conference on Human Computation and Crowdsourcing. David Timothy Lee, Ashish Goel, Tanja Aitamurto, and Helene Landemore. 2014. Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions. In Second AAAI Conference on Human Computation and Crowdsourcing.
  39. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu , Siheon Lee , Alexandros Psomas , 2019 . WeBuildAI: Participatory framework for algorithmic governance . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1–35. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–35.
  40. Toby Jia-Jun Li , Lindsay Popowski , Tom Mitchell , and Brad A Myers . 2021 . Screen2Vec: Semantic Embedding of GUI Screens and GUI Components . In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15 . Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
  41. Arend Lijphart . 1999. Patterns of democracy . Yale university press . Arend Lijphart. 1999. Patterns of democracy. Yale university press.
  42. 10.1145/3308560.3317082 · DOI
  43. V K Chaithanya Manam and Alexander J Quinn . 2018 . Wingit: Efficient refinement of unclear task instructions . In Sixth AAAI Conference on Human Computation and Crowdsourcing. VK Chaithanya Manam and Alexander J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing.
  44. Erwan Moreau , Carl Vogel , and Marguerite Barry . 2019. A paradigm for democratizing artificial intelligence research . In Innovations in Big Data Mining and Embedded Knowledge . Springer , 137–166. Erwan Moreau, Carl Vogel, and Marguerite Barry. 2019. A paradigm for democratizing artificial intelligence research. In Innovations in Big Data Mining and Embedded Knowledge. Springer, 137–166.
  45. Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200 [cs.CL] Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200 [cs.CL] · DOI · Paper page
  46. Kayur Patel James Fogarty James A Landay and Beverly L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563–1566. Kayur Patel James Fogarty James A Landay and Beverly L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563–1566.
  47. 10.1109/ICCV.2019.00971 · DOI · Paper page
  48. 10.18653/v1/2021.law-1.14 · DOI · Paper page
  49. 10.1177/019251219201300107 · doi:10.1177/019251219201300107
  50. Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947 [cs.CL] Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947 [cs.CL] · arXiv
  51. 10.1145/3359178 · DOI · Paper page
  52. 10.1145/3313831.3376506 · DOI · Paper page
  53. 10.1145/3274423 · DOI · Paper page
  54. 10.1145/3476058 · DOI · Paper page
  55. 10.1145/3411764.3445742 · DOI
  56. 10.1145/1401890.1401965 · DOI · Paper page
  57. 10.2139/ssrn.199668 · DOI · Paper page
  58. 10.5555/3298023.3298188 · doi:10.5555/3298023.3298188
  59. 10.7717/peerj.6762 · DOI · Paper page
  60. Betty van Aken Julian Risch Ralf Krestel and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018). Betty van Aken Julian Risch Ralf Krestel and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018). · DOI · Paper page
  61. 10.1007/978-3-642-13739-6_25 · DOI · Paper page
  62. 10.5555/3367471.3367557 · doi:10.5555/3367471.3367557
  63. 10.1145/3442381.3450078 · DOI · Paper page
  64. Tian Wang Yuri M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021). Tian Wang Yuri M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021). · arXiv
  65. 10.1145/3357236.3395528 · DOI · Paper page
  66. 10.1145/3379337.3415858 · DOI · Paper page
  67. 10.1145/3136755.3136792 · DOI

Cited by

See all 132

Metadata

Authors
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S.
Venue
CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2022-04-27
Pages
1-19
OpenAlex
W4225006216
Cited by
132
References
22
FWCI
28.76
Type
conference paper
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Jury Learning: Integrating Dissenting Voices into Machine Learning Models. https://app.cochat.ai/papers/10.1145/3491102.3502004
The paper
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S. (2022). Jury Learning: Integrating Dissenting Voices into Machine Learning Models. CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3502004

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.