CoChat
Sign Up
Cited by 50Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)AI and HR Technologies

A hunt for the Snark: Annotator Diversity in Data Practices

Open access
Kapania, Shivani, Taylor, Alex S, Wang, Ding·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3580645
Open this paper in CoChat — free accountOpen PDF

Abstract

Diversity in datasets is a key component to building responsible AI/ML. Despite this recognition, we know little about the diversity among the annotators involved in data production. We investigated the approaches to annotator diversity through 16 semi-structured interviews and a survey with 44 AI/ML practitioners. While practitioners described nuanced understandings of annotator diversity, they rarely designed dataset production to account for diversity in the annotation process. The lack of action was explained through operational barriers: from the lack of visibility in the annotator hiring process, to the conceptual difficulty in incorporating worker diversity. We argue that such operational barriers and the widespread resistance to accommodating annotator diversity surface a prevailing logic in data practices—where neutrality, objectivity and ‘representationalist thinking’ dominate. By understanding this logic to be part of a regime of existence, we explore alternative ways of accounting for annotator subjectivity and diversity in data practices.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3580645

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies of industry-scale data collection revealed that the pursuit of label precision is often driven by market-oriented values such as objectivity, standardization, and corporate interests.

    “studies of industryscale data collection revealed that the pursuit of label precision is often driven by market-oriented values such as objectivity, standardization, and corporate interests [49, 51, 71, 112, 113].”

    From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [49]

    Sections 4.2.1 and 5 describe practitioners pursuing objective annotations and standardised metrics; section 4.3.3 documents cost and competitive pressures on annotation work. The paper does not establish that market-oriented values drive a pursuit of label precision specifically, so that causal framing goes beyond its findings. The available passages have no page numbers. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 73 · Cited by 50

Research works printed but not linked: 82

  1. Jaimeen Ahn and Alice Oh. 2021. Mitigating Language-Dependent Ethnic Bias in BERT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 533-549. https://doi.org/10.18653/v1/2021. emnlp-main.42 · DOI · Paper page
  2. Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 1100-1105. https://doi.org/10.1145/3308560. 3317083 · DOI
  3. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine 36, 1 (2015), 15-24.
  4. Karen Barad. 2003. Posthumanist performativity: Toward an understanding of how matter comes to matter. Signs: Journal of women in culture and society 28, 3 (2003), 801-831.
  5. Alice M Brawley and Cynthia LS Pury. 2016. Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior 54 (2016), 531-546.
  6. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accu- racy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77-91. https://proceedings.mlr.press/v81/buolamwini18a. html
  7. Graham Button and Wes Sharrock. 1997. The Production of Order and the Order of Production: Possibilities for Distributed Organisations, Work and Technology in the Print Industry. Springer Netherlands, Dordrecht, 1-16. https://doi.org/10. 1007/978-94-015-7372-6_1 · DOI
  8. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Seman- tics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. https://doi.org/10.1126/science.aal4230 arXiv:http://science.sciencemag.org/content/356/6334/183.full.pdf · DOI · Paper page
  9. Scott Allen Cambo and Darren Gergle. 2022. Model Positionality and Compu- tational Reflexivity: Promoting Reflexivity in Data Science. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 572, 19 pages. https://doi.org/10.1145/3491102.3501998 · DOI · Paper page
  10. L. Elisa Celis, Amit Deshpande, Tarun Kathuria, and Nisheeth K. Vishnoi. 2016. How to be Fair and Diverse? arXiv e-prints, Article arXiv:1610.07183 (Oct. 2016), 5 pages. https://doi.org/10.48550/arXiv.1610.07183 arXiv:1610.07183 [cs.LG] · DOI
  11. Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces. Springer New York, New York, NY, 377-392. https://doi.org/10.1007/978-1-4614-8806-4_30 · DOI
  12. Patricia Hill Collins and Sirma Bilge. 2020. Intersectionality. John Wiley & Sons.
  13. Andrew Crabtree, Mark Rouncefield, and Peter Tolmie. 2012. Doing design ethnography. Springer, London.
  14. Andy Crabtree, Peter Tolmie, and Mark Rouncefield. 2013. "How many bloody examples do you want?": fieldwork and generalisation. In Proceedings of the 13th European Conference on Computer Supported Cooperative Work ECSCW 2013,
  15. Olav W. Bertelsen, Luigina Ciolfi, Maria Antonietta Grasso, and George Angelos Papadopoulos (Eds.). Springer Verlag, London, 1-20. https://doi.org/10.1007/ 978-1-4471-5346-7_1 · DOI
  16. Kimberlé Crenshaw. 2013. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In Feminist legal theories. Routledge, New York, NY, USA, 23-51.
  17. Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Comput- ing Surveys (CSUR) 51, 1 (2018), 1-40.
  18. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics 10 (2022), 92-110.
  19. Jenny L Davis, Apryl Williams, and Michael W Yang. 2021. Algorithmic repara- tion. Big Data & Society 8, 2 (2021), 20539517211044808.
  20. Anne A H de Hond, Marieke M van Buchem, and Tina Hernandez- Boussard. 2022. Picture a data scientist: a call to action for increasing diversity, equity, and inclusion in the age of AI. Journal of the Ameri- can Medical Informatics Association 29, 12 (09 2022), 2178-2181. https:// doi.org/10.1093/jamia/ocac156 arXiv:https://academic.oup.com/jamia/article- pdf/29/12/2178/47027843/ocac156.pdf · DOI
  21. Wesley Hanwen Deng, Manish Nagireddy, Michelle Seng Ah Lee, Jatinder Singh, Zhiwei Steven Wu, Kenneth Holstein, and Haiyi Zhu. 2022. Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 473-484. https://doi.org/10.1145/3531146.3533113 · DOI · Paper page
  22. Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar. 2020. On Measuring and Mitigating Biased Inferences of Word Embeddings. Proceedings of the AAAI Conference on Artificial Intelligence 34, 05 (Apr. 2020), 7659-7666. https: //doi.org/10.1609/aaai.v34i05.6267 · DOI · Paper page
  23. C. Disalvo. 2015. Adversarial Design. MIT Press, Cambridge, MA, USA. https: //books.google.co.in/books?id=gdcGEAAAQBAJ
  24. Lynn Dombrowski, Ellie Harmon, and Sarah Fox. 2016. Social Justice-Oriented Interaction Design: Outlining Key Design Strategies and Commitments. In Pro- ceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS '16). Association for Computing Machinery, New York, NY, USA, 656-671. https://doi.org/10.1145/2901790.2901861 · DOI
  25. Ryan Drapeau, Lydia Chilton, Jonathan Bragg, and Daniel Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 4, 1, 32-41. https://doi.org/10.1609/hcomp.v4i1.13270 · DOI
  26. Marina Drosou, HV Jagadish, Evaggelia Pitoura, and Julia Stoyanovich. 2017. Diversity in big data: A review. Big data 5, 2 (2017), 73-84.
  27. Sina Fazelpour and Maria De-Arteaga. 2022. Diversity in sociotechnical machine learning systems. Big Data & Society 9, 1 (2022), 20539517221082027.
  28. L. Gitelman. 2013. Raw Data Is an Oxymoron. MIT Press, Cambridge, MA, USA. https://books.google.co.in/books?id=oARaHF4D6h0C
  29. Barney G Glaser. 1998. Doing grounded theory: Issues and discussions. Vol. 254. Sociology Press, Mill Valley, CA.
  30. Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge, London.
  31. Charles Goodwin. 1995. Seeing in depth. Social studies of science 25, 2 (1995), 237-274.
  32. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 115, 19 pages. https://doi.org/10.1145/3491102.3502004 · DOI
  33. Nitesh Goyal, Ian D. Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 363 (nov 2022), 28 pages. https://doi.org/10.1145/3555088 · DOI · Paper page
  34. Ian Hacking et al. 1983. Representing and intervening: Introductory topics in the philosophy of natural science. Cambridge university press.
  35. Ange-Marie Hancock. 2007. When multiplication doesn't equal quick addition: Examining intersectionality as a research paradigm. Perspectives on politics 5, 1 (2007), 63-79.
  36. Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Vol. 14. Feminist Studies, Inc., 575-599. http://www.jstor.org/stable/3178066
  37. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Unintended Machine Learning Biases as Social Barriers for Persons with Disabilitiess. SIGACCESS Access. Comput. 125, Article 9 (mar 2020), 1 pages. https://doi.org/10.1145/3386296.3386305 · DOI · Paper page
  38. Lilly Irani. 2019. Justice for data janitors. In Think in Public. Columbia University Press, New York, NY, USA, 23-40.
  39. Azra Ismail and Neha Kumar. 2018. Engaging solidarity in data collection practices for community health. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-24.
  40. Nataliya V Ivankova, John W Creswell, and Sheldon L Stick. 2006. Using mixed- methods sequential explanatory design: From theory to practice. Field methods 18, 1 (2006), 3-20.
  41. Farnaz Jahanbakhsh, Justin Cranshaw, Scott Counts, Walter S. Lasecki, and Kori Inkpen. 2020. An Experimental Study of Bias in Platform Worker Ratings: The Role of Performance Quality and Gender. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https: //doi.org/10.1145/3313831.3376860 · DOI
  42. Florian Jaton. 2021. Assessing biases, relaxing moralism: On ground-truthing practices in machine learning design and application. Big Data & Society 8, 1 (2021), 20539517211013569.
  43. Eun Seo Jo and Timnit Gebru. 2020. Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Machinery, New York, NY, USA, 306-316. https://doi.org/10.1145/3351095.3372829 · DOI · Paper page
  44. Sanjay Kairam and Jeffrey Heer. 2016. Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW '16). Association for Computing Machin- ery, New York, NY, USA, 1637-1648. https://doi.org/10.1145/2818048.2820016 · DOI
  45. Matthew Lease. 2011. On Quality Control and Machine Learning in Crowd- sourcing. In Proceedings of the 11th AAAI Conference on Human Computation (AAAIWS'11-11). AAAI Press, Palo Alto, CA, USA, 97-102.
  46. Michelle Seng Ah Lee and Jat Singh. 2021. The Landscape and Gaps in Open Source Fairness Toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 699, 13 pages. https: //doi.org/10.1145/3411764.3445261 · DOI · Paper page
  47. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. UNQOVERing Stereotyping Biases via Underspecified Questions. In Find- ings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3475-3489. https://doi.org/10.18653/v1/ 2020.findings-emnlp.311 · DOI
  48. Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting Stance in Media On Global Warming. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3296-3315. https://doi.org/10.18653/v1/2020.findings-emnlp.296 · DOI
  49. Adam Marcus, David Karger, Samuel Madden, Robert Miller, and Sewoong Oh. 2012. Counting with the crowd. Proceedings of the VLDB Endowment 6, 2 (2012), 109-120.
  50. David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW '14). Association for Computing Machinery, New York, NY, USA, 224-235. https: //doi.org/10.1145/2531602.2531663 · DOI
  51. Milagros Miceli and Julian Posada. 2022. The Data-Production Dispositif. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 460 (nov 2022), 37 pages. https: //doi.org/10.1145/3555561 · DOI
  52. Milagros Miceli, Julian Posada, and Tianling Yang. 2022. Studying up machine learning data: Why talk about bias when we mean power? Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1-14.
  53. Milagros Miceli, Martin Schuessler, and Tianling Yang. 2020. Between Subjectiv- ity and Imposition: Power Dynamics in Data Annotation for Computer Vision. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 115 (oct 2020), 25 pages. https://doi.org/10.1145/3415186 · DOI · Paper page
  54. Milagros Miceli, Tianling Yang, Laurens Naudts, Martin Schuessler, Diana Ser- banescu, and Alex Hanna. 2021. Documenting Computer Vision Datasets: An Invitation to Reflexive Data Practices. In Proceedings of the 2021 ACM Con- ference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21). Association for Computing Machinery, New York, NY, USA, 161-172. https://doi.org/10.1145/3442188.3445880 · DOI
  55. Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-15. https://doi.org/10.1145/3290605.3300356 · DOI
  56. Michael Muller, Christine T. Wolf, Josh Andres, Michael Desmond, Naren- dra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan. 2021. Designing Ground Truth and the Social Life of Labels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Associ- ation for Computing Machinery, New York, NY, USA, Article 94, 16 pages. https://doi.org/10.1145/3411764.3445402 · DOI · Paper page
  57. Joaquin Navajas, Tamara Niella, Gerry Garbulsky, Bahador Bahrami, and Mar- iano Sigman. 2018. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behaviour 2, 2 (2018), 126-132.
  58. Jacki O'Neill, Stefania Castellani, Frederic Roulland, Nicolas Hairon, Cornell Juliano, and Liwei Dai. 2011. From Ethnographic Study to Mixed Reality: A Remote Collaborative Troubleshooting System. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (Hangzhou, China) (CSCW '11). Association for Computing Machinery, New York, NY, USA, 225-234. https: //doi.org/10.1145/1958824.1958859 · DOI
  59. Samir Passi and Steven Jackson. 2017. Data Vision: Learning to See Through Algorithmic Abstraction. In Proceedings of the 2017 ACM Conference on Com- puter Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 2436-2447. https://doi.org/10.1145/2998181.2998331 · DOI
  60. Kathleen H. Pine and Max Liboiron. 2015. The Politics of Measurement and Action. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 3147-3156. https://doi.org/10. 1145/2702123.2702298 · DOI
  61. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On Releasing Annotator-Level Labels and Information in Datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop. Association for Computational Linguistics, Punta Cana, Dominican Republic, 133-138. https://doi.org/10.18653/v1/2021. law-1.14 · DOI
  62. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 1776-1826. https://doi.org/10.1145/3531146.3533231 · DOI
  63. David Randall, Richard Harper, and Mark Rouncefield. 2007. Fieldwork for design: theory and practice. Springer Science & Business Media, London.
  64. Negar Rostamzadeh, Diana Mincu, Subhrajit Roy, Andrew Smart, Lauren Wilcox, Mahima Pushkarna, Jessica Schrouff, Razvan Amironesei, Nyalleng Moorosi, and Katherine Heller. 2022. Healthsheet: Development of a Transparency Artifact for Health Datasets. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 1943-1961. https://doi.org/10.1145/3531146. 3533239 · DOI
  65. Nithya Sambasivan and Rajesh Veeraraghavan. 2022. The Deskilling of Do- main Expertise in AI Development. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Asso- ciation for Computing Machinery, New York, NY, USA, Article 587, 14 pages. https://doi.org/10.1145/3491102.3517578 · DOI
  66. Mike Schaekermann, Graeme Beaton, Minahz Habib, Andrew Lim, Kate Larson, and Edith Law. 2019. Understanding expert disagreement in medical data analysis through structured adjudication. Proceedings of the ACM on Human- Computer Interaction 3, CSCW (2019), 1-23.
  67. Mike Schaekermann, Carrie J. Cai, Abigail E. Huang, and Rory Sayres. 2020. Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376290 · DOI · Paper page
  68. Mike Schaekermann, Joslin Goh, Kate Larson, and Edith Law. 2018. Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-19.
  69. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Las Vegas, Nevada, USA) (KDD '08). Association for Computing Machinery, New York, NY, USA, 614-622. https://doi.org/10.1145/ 1401890.1401965 · DOI
  70. Yaron Singer and Manas Mittal. 2013. Pricing Mechanisms for Crowdsourcing Markets. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW '13). Association for Computing Machinery, New York, NY, USA, 1157-1166. https://doi.org/10.1145/2488388.2488489 · DOI
  71. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and Fast -But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, 254-263. https://aclanthology.org/D08-1027
  72. Harini Suresh and John Guttag. 2021. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. In Equity and Access in Al- gorithms, Mechanisms, and Optimization. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3465416.3483305 · DOI · Paper page
  73. Alex S. Taylor, Siân Lindley, Tim Regan, David Sweeney, Vasillis Vlachokyriakos, Lillie Grainger, and Jessica Lingel. 2015. Data-in-Place: Thinking through the Relations Between Data and Community. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 2863-2872. https://doi.org/10.1145/2702123.2702558 · DOI
  74. Geneviève Teil. 2012. No such thing as terroir? Objectivities and the regimes of existence of objects. Science, Technology, & Human Values 37, 5 (2012), 478-505.
  75. Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for Toxic Comment Classification: An In-Depth Error Analysis. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Compu- tational Linguistics, Brussels, Belgium, 33-42. https://doi.org/10.18653/v1/W18- 5105 · DOI
  76. Janet Vertesi and Paul Dourish. 2011. The Value of Data: Considering the Context of Production in Data Economies. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (Hangzhou, China) (CSCW '11). Association for Computing Machinery, New York, NY, USA, 533-542. https: //doi.org/10.1145/1958824.1958906 · DOI
  77. Zeerak Waseem. 2016. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the First Work- shop on NLP and Computational Social Science. Association for Computational Linguistics, Austin, Texas, 138-142. https://doi.org/10.18653/v1/W16-5618 · DOI
  78. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns. TACL 6 (2018), 605-617. https://transacl.org/ojs/index.php/tacl/article/view/1484
  79. Peter Welinder and Pietro Perona. 2010. Online crowdsourcing: Rating annota- tors and obtaining cost-effective labels. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition -Workshops. IEEE, San Francisco, CA, USA, 25-32. https://doi.org/10.1109/CVPRW.2010.5543189 · DOI
  80. Etienne Wenger et al. 1998. Communities of practice: Learning as a social system. Systems thinker 9, 5 (1998), 2-3.
  81. Allison Woodruff, Sarah E. Fox, Steven Rousso-Schindler, and Jeffrey Warshaw. 2018. A Qualitative Exploration of Perceptions of Algorithmic Fairness. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3173574.3174230 · DOI · Paper page
  82. Xinyi Zhou and Reza Zafarani. 2020. A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. ACM Comput. Surveys 53, 5, Article 109 (sep 2020), 40 pages. https://doi.org/10.1145/3395046 · DOI · Paper page

Authors
Kapania, S., Taylor, A. S., & Wang, D.
Affiliations
  • Google (United States) (US)
  • University of London (GB)
  • Google (Singapore) (SG)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-15
DOI
10.1145/3544548.3580645
OpenAlex
W4366588458
Cited by
50
References
73
Citations by year
2026: 9 · 2025: 21 · 2024: 16 · 2023: 4
FWCI
12.53
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
Keywords
data annotation, responsible AI, data diversity, operational barriers, data practices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review