CoChat
Sign Up
Cited by 72Ethics and Social Impacts of AIMobile Crowdsensing and CrowdsourcingAI and HR Technologies

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

Open access
Díaz, Mark, Kivlichan, Ian, Rosen, Rachel, Baker, Dylan, Amironesei, Razvan, Prabhakaran, Vinodkumar, Denton, Remi·2022·2022 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3531146.3534647
Open this paper in CoChat — free accountOpen PDF

Abstract

Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators’ lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3531146.3534647

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Díaz et al.'s documentation framework for crowdsourced dataset annotation asks for transparency regarding working conditions, such as communication channels and compensation standards.

    “The documentation framework for crowdsourced dataset annotation proposed by Díaz et al. [40] asks for transparency regarding working conditions, such as the availability of communication channels and compensation standards.”

    From 'The plan is just survival': Data Work in Kenya and the Regime of Entrapment (2026) · cites this paper as [40]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Ali Alkhatib, Michael S. Bernstein, and Margaret Levi. 2017. Examining Crowd Work and Gig Work Through The Historical Lens of Piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 4599-4616. https: //doi.org/10.1145/3025453.3025974 · DOI · Paper page
  2. Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions. In Companion Proceedings of The 2019 World Wide Web Conference (WWW '19). Association for Computing Machinery, New York, NY, USA, 1100-1105. https://doi.org/10.1145/3308560.3317083 · DOI
  3. Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013. ACM (2013).
  4. Lora Aroyo and Chris Welty. 2015. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine 36, 1 (Mar. 2015), 15-24. https: //doi.org/10.1609/aimag.v36i1.2564 · DOI
  5. Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comp. Lab. L. & Pol'y J. 37 (2015), 543.
  6. Maja Bott and Gregor Young. 2012. The role of crowdsourcing for better gov- ernance in international development. Praxis: The Fletcher Journal of Human Security 27, 1 (2012), 47-70.
  7. Kasia S. Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. 2020. The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence. (2020). http://securedata.lol/ NeurIPS Workshop on Dataset Curation and Security.
  8. Emily L. Denton, A. Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, and M. Scheuerman. 2020. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. ICML Workshop on Participatory Approaches to Machine Learning (2020).
  9. Anna Filippova, Connor Gilroy, Ridhi Kashyap, Antje Kirchner, Allison C. Mor- gan, Kivan Polimis, Adaner Usmani, and Tong Wang. 2019. Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Sur- vey Data. Socius 5 (2019), 2378023118820157.
  10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.
  11. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). · arXiv
  12. R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. Garbage in, Garbage out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20). Association for Computing Machinery, New York, NY, USA, 325-336. https://doi.org/10.1145/3351095.3372862 · DOI
  13. Christine Gerber. 2021. Community building on crowdwork platforms: Autonomy and control of online workers? Competition & Change 25, 2 (2021), 190-211.
  14. Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021). Association for Computational Linguistics, Online, 313-328. https://aclanthology. org/2021.wnut-1.35
  15. Benjamin V Hanrahan, Anita Chen, JiaHua Ma, Ning F Ma, Anna Squicciarini, and Saiph Savage. 2021. The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical Turk. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-23.
  16. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers' earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1-14.
  17. Anna Lauren Hoffmann. 2021. Terms of inclusion: Data, discourse, vio- lence. New Media & Society 23, 12 (2021), 3539-3556. https://doi.org/10.1177/ 1461444820958725 arXiv:https://doi.org/10.1177/1461444820958725 · DOI
  18. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielin- ski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018). · arXiv
  19. Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards Ac- countability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Ac- countability, and Transparency (FAccT '21). Association for Computing Machinery, 560-575.
  20. Lilly C Irani and M Six Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems. 611-620.
  21. Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P Bigham. 2018. Striving to earn more: a survey of work strategies and tool use among crowd workers. In Sixth AAAI Conference on Human Computation and Crowdsourcing.
  22. David Kocsis and Gert Jan De Vreede. 2016. Towards a taxonomy of ethical con- siderations in crowdsourcing (22nd Americas Conference on Information Systems: Surfing the IT Innovation Wave, AMCIS 2016).
  23. David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Be- ing a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). Association for Computing Ma- chinery, New York, NY, USA, 224-235. https://doi.org/10.1145/2531602.2531663 · DOI
  24. Milagros Miceli, Martin Schuessler, and Tianling Yang. 2020. Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 115 (Oct. 2020), 25 pages. https://doi.org/10.1145/3415186 · DOI · Paper page
  25. Cecilia Ovesdotter Alm. 2011. Subjective Natural Language Problems: Motiva- tions, Applications, Characterizations, and Implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, 107-112. https://aclanthology.org/P11-2019
  26. Lisa Posch, Arnim Bleier, Fabian Flöck, and Markus Strohmaier. 2018. Character- izing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint arXiv:1812.05948 (2018). · arXiv
  27. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On Releasing Annotator-Level Labels and Information in Datasets. In Proceed- ings of the 15th Linguistic Annotation Workshop. Association for Computational Linguistics, Virtual.
  28. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In Pro- ceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency.
  29. Alexander J. Quinn and Benjamin B. Bederson. 2011. Human Computation: A Survey and Taxonomy of a Growing Field. Association for Computing Machinery, New York, NY, USA, 1403-1412. https://doi.org/10.1145/1978942.1979148 · DOI
  30. Jorge Ramírez, Burcu Sayin, Marcos Baez, Fabio Casati, Luca Cernuzzi, Boualem Benatallah, and Gianluca Demartini. 2021. On the State of Reporting in Crowd- sourcing Experiments and a Checklist to Aid Current Practices. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 387 (oct 2021), 34 pages. https: //doi.org/10.1145/3479531 · DOI · Paper page
  31. Sarah T Roberts. 2016. Digital refuse: Canadian garbage, commercial content moderation and the global circulation of social media's waste. Wi: Journal of Mobile Media 10, 1 (2016), 1-18.
  32. Reka Marta Sabou, Kalina Bontcheva, Leon Derczynski, and A. Scharl. 2014. Cor- pus Annotation through Crowdsourcing:Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Eval- uation (LREC'14). European Language Resources Association (ELRA), Reykjavik, Iceland.
  33. Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and Clickhappier. 2015. We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers. Association for Computing Machinery, New York, NY, USA, 1621-1630. https://doi.org/10.1145/2702123.2702508 · DOI
  34. Morgan Klaus Scheuerman, Emily Denton, and Alex Hanna. 2021. Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development. Computer Supported Cooperative Work (CSCW) (2021).
  35. Daniel Schlagwein, Dubravka Cecez-Kecmanovic, and Benjamin Hanckel. 2019. Ethical norms and issues in crowdsourcing practices: A Habermasian analysis. Information Systems Journal 29, 4 (2019), 811-837. https://doi.org/10.1111/isj. 12227 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/isj.12227 · DOI
  36. Louise Seamster and Raphaël Charron-Chénier. 2017. Predatory Inclu- sion and Education Debt: Rethinking the Racial Wealth Gap. Social Currents 4, 3 (2017), 199-207. https://doi.org/10.1177/2329496516686620 arXiv:https://doi.org/10.1177/2329496516686620 · DOI
  37. Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19). Association for Computing Machinery, New York, NY, USA, 59-68. https://doi.org/10.1145/3287560.3287598 · DOI · Paper page
  38. Shilad Sen, Margaret E. Giesel, Rebecca Gold, Benjamin Hillmann, Matt Lesicko, Samuel Naden, Jesse Russell, Zixiao (Ken) Wang, and Brent Hecht. 2015. Turk- ers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). Association for Computing Ma- chinery, New York, NY, USA, 826-838. https://doi.org/10.1145/2675133.2675285 · DOI · Paper page
  39. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing. arXiv preprint arXiv:2104.10097 (2021). · arXiv
  40. Ramya Malur Srinivasan, Emily Denton, Jordan Jennifer Famularo, Negar Ros- tamzadeh, Fernando Diaz, and Beth Coleman. 2021. Artsheets for Art Datasets. In Proceedings of Neural Information Processing Systems (NeurIPS), Datasets & Benchmarks Track. https://openreview.net/pdf?id=K7ke_GZ_6N
  41. Carlos Toxtli, Siddharth Suri, and Saiph Savage. 2021. Quantifying the Invisible Labor in Crowd Work. Proc. ACM Hum.-Comput. Interact. 5, CSCW2 (2021).
  42. Donna Vakharia and Matthew Lease. 2015. Beyond Mechanical Turk: An analysis of paid crowd work platforms. Proceedings of the iConference (2015), 1-17.
  43. Zeerak Waseem. 2016. Are You a Racist or Am I Seeing Things? Annotator Influ- ence on Hate Speech Detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science. Association for Computational Linguistics, Austin, Texas, 138-142. https://doi.org/10.18653/v1/W16-5618 · DOI

Cited by

Sorted by citations

Authors
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R.
Affiliations
  • Google (United States) (US)
  • Google, USA
ORCID
Venue
2022 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2022-06-20
Pages
2342-2351
DOI
10.1145/3531146.3534647
OpenAlex
W4283168936
Cited by
72
References
3
Citations by year
2026: 9 · 2025: 21 · 2024: 16 · 2023: 23 · 2022: 3
FWCI
15.65
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
Keywords
ethical considerations, crowdsourcing platforms, collective identity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review