Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
Open accessAbstract
In response to algorithmic unfairness embedded in sociotechnical systems, significant attention has been focused on the contents of machine learning datasets which have revealed biases towards white, cisgender, male, and Western data subjects. In contrast, comparatively less attention has been paid to the histories, values, and norms embedded in such datasets. In this work, we outline a research program - a genealogy of machine learning data - for investigating how and why these datasets have been created, what and whose values influence the choices of data to collect, the contextual and contingent conditions of their creation. We describe the ways in which benchmark datasets in machine learning operate as infrastructure and pose four research questions for these datasets. This interrogation forces us to "bring the people back in" by aiding us in understanding the labor embedded in dataset construction, and thereby presenting new avenues of contestation for other researchers encountering the data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Universal benchmarks often express the perspectives of dominant groups in the name of objectivity and neutrality and thus perpetuate further harm against marginalized groups.
“and “universal” benchmarks often express the perspectives of dominant groups in the name of objectivity and neutrality and thus perpetuate further harm against marginalized groups (Denton et al. 2020).”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Denton et al. 2020
The Abstract (p. 1) describes dataset biases toward white, cisgender, male, and Western subjects; Data Infrastructure (pp. 2–3) says authoritative benchmarks can stand in for more complex tasks and that dataset practices become taken for granted. The Conclusion (p. 4) says powerful actors’ norms and values shape data. These passages support the concern about dominant perspectives and harm, but do not establish that benchmarks claim universality or invoke objectivity and neutrality to justify those perspectives. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Politics and Poetics of Infrastructure
Larkin, B. (2013) · Annual Review of Anthropology · doi:10.1146/annurev-anthro-092412-155522 · Cited by 4,095 · see citing works
Tufekci, Z. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14517 · Cited by 780 · see citing works
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · see citing works
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.487 · Cited by 274 · see citing works
Self-citationScheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 25
- Barabas, C., Doyle, C., Rubinovitz, J., and Dinakar, K. Studying up: Reorienting the study of algorithmic fairness around issues of power. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , FAT* 20, pp. 167176, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/3351095.3372859. URL https://doi.org/10.1145/3351095.3372859 https://doi.org/10.1145/3351095.3372859 · DOI
- Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code . John Wiley & Sons, 2019.
- Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., and Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS16, pp. 43564364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
- Bowker, G., Baker, K., Millerand, F., and Ribes, D. Toward Information Infrastructure Studies: Ways of Knowing in a Networked Environment , pp. 97-117. 06 2010. doi: 10.1007/978-1-4020-9789-8 5. · DOI
- Bowker, G. C. and Star, S. L. Sorting Things Out . MIT Press, 2000.
- Dotan, R. and Milli, S. Value-laden disciplinary shifts in machine learning. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 2020.
- Eubanks, V. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor . St. Martins Press, Inc., USA, 2018. ISBN 1250074312.
- Forsythe, D. E. Ethics and politics of studying up in technoscience. Anthropology of Work Review , 20 (1):6-11, 1999. doi: 10.1525/awr.1999.20.1.6. URL · DOI
- Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J., Wallach, H., Daume, I., and Crawford, K. Datasheets for datasets. 03 2018. URL https://arxiv.org/abs/1803.09010 . · arXiv
- Geiger, R. S., Yu, K., Yang, Y., Dai, M., Qiu, J., Tang, R., and Huang, J. Garbage in, garbage out? do machine learning application papers in social computing report where human-labeled training data comes from? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 2020.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning . MIT Press, 2016. http://www.deeplearningbook.org .
- Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, pp. 3323-3331, USA, 2016. Curran Associates Inc. ISBN 978-1-5108-3881-9. URL http://dl.acm.org/citation.cfm?id=3157382.3157469
- Irani, L. C. and Silberman, M. S. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems , pp. 611-620, 2013.
- Jo, E. S. and Gebru, T. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , FAT* 20, 2020.
- Kohler, R. E. Lords of the fly: Drosophila genetics and the experimental life . University of Chicago Press, 1994.
- Fujimura, J. H. The molecular biological bandwagon in cancer research: Where social worlds meet. Social Problems , 35(3):261-283, 1988.
- Latour, B. Science in action: How to follow scientists and engineers through society . Harvard university press, 1987.
- Latour, B. and Woolgar, S. Laboratory life: The construction of scientific facts . Princeton University Press, 1979.
- Mulligan, D. K., Kluttz, D., and Kohli, N. Shaping our tools: Contestability as a means to promote responsible algorithmic decision making in the professions. Available at SSRN 3311894 , 2019.
- Noble, S. U. Algorithms of oppression: How search engines reinforce racism . nyu Press, 2018.
- Sachs, S. The algorithm at work? explanation and repair in the enactment of similarity in art data. Information, Communication & Society , pp. 1-17, 2019.
- Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., and Milland, K. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd annual ACM conference on human factors in computing systems , pp. 1621-1630, 2015.
- Scheuerman, M. K., Wade, K., Lustig, C., and Brubaker, J. R. How weve taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proc. ACM Hum.-Comput. Interact. , 4(CSCW1), 2020. doi: 10.1145/3392866. URL https://doi.org/10.1145/3392866 . · DOI · Paper page
- Seaver, N. Captivating algorithms: Recommender systems as traps. Journal of Material Culture , 24(4):421-436, 2019.
- Trewin, S. Ai fairness for people with disabilities: Point of view. ArXiv , abs/1811.10670, 2018. · arXiv
Cited by
Sorted by citations
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“Curating training data can also help to make LMs fairer [45, 84, 94].”
BackgroundBias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Self-citationPredictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Rethinking Fairness: An Interdisciplinary Survey of Critiques of Hegemonic ML Fairness Approaches
Weinberg, L. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13196 · Cited by 127 · see citing works
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · see citing works
Self-citationLanguage Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31 · see citing works
- Authors
- Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K.
- Affiliations
- University of San Francisco (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2007.07399
- OpenAlex
- W3043315258
- Cited by
- 100
- References
- 5
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 19 · 2023: 18 · 2022: 21 · 2021: 18 · 2020: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Feminist Epistemology and Gender Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic fairness, data bias, socio-technical systems, data infrastructure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
