The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Open accessAbstract
Data is a crucial component of machine learning. The field is reliant on data to train, validate, and test models. With increased technical capabilities, machine learning research has boomed in both academic and industry settings, and one major focus has been on computer vision. Computer vision is a popular domain of machine learning increasingly pertinent to real-world applications, from facial recognition in policing to object detection for autonomous vehicles. Given computer vision's propensity to shape machine learning research and impact human life, we seek to understand disciplinary practices around dataset documentation - how data is collected, curated, annotated, and packaged into datasets for computer vision researchers and practitioners to use for model tuning and development. Specifically, we examine what dataset documentation communicates about the underlying values of vision data and the larger practices and goals of computer vision as a field. To conduct this study, we collected a corpus of about 500 computer vision datasets, from which we sampled 114 dataset publications across different vision tasks. Through both a structured and thematic content analysis, we document a number of values around accepted data practices, what makes desirable data, and the treatment of humans in the dataset construction process. We discuss how computer vision datasets authors value efficiency at the expense of care; universality at the expense of contextuality; impartiality at the expense of positionality; and model work at the expense of data work. Many of the silenced values we identify sit in opposition with social computing practices. We conclude with suggestions on how to better incorporate silenced values into the dataset creation and curation process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize ' efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . '.”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Scheuerman et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Very recent research - most published within the last year - has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Situating knowledges: positionality, reflexivities and other tactics
Rose, G. (1997) · Progress in Human Geography · doi:10.1191/030913297673302122 · Cited by 2,390
Race and the Education of Desire: Foucault’s History of Sexuality and the Colonial Order of Things
Stoler, A. L. (1995) · Duke University Press eBooks · doi:10.1215/9780822377719 · Cited by 2,194
(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
Computer-supported cooperative work: history and focus
Grudin, J. (1994) · Computer · doi:10.1109/2.291294 · Cited by 1,639
The economic benefits of publicly funded basic research: a critical review
Salter, A. J., & Martin, B. R. (2001) · Research Policy · doi:10.1016/s0048-7333(00)00091-3 · Cited by 1,358
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9302.001.0001 · Cited by 1,160
Feminist HCI: taking stock and outlining an agenda for design
Bardzell, S. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753521 · Cited by 1,123
Research works printed but not linked: 41
- Shazia Afzal, Rajmohan C, Manish Kesarwani, Sameep Mehta, and Hima Patel. 2020. Data Readiness Report. arXiv:2010.07213 [cs] (Oct. 2020). arXiv:2010.07213 [cs] · arXiv
- Philip E. Agre. 1998. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. Psychology Press.
- Carolyn Ashurst Markus Anderljung Carina Prunkl Jan Leike Yarin Gal Toby Shevlane and Allan Dafoe. 2020. A Guide to Writing the NeurIPS Impact Statement. (2020).
- Christine L Borgman. 2016. Big Data, Little Data, No Data: Scholarship in the Networked World. MIT press.
- Geoffrey C Bowker. 2005. Memory Practices in the Sciences. Mit Press Cambridge, MA.
- 10.5555/380681 · doi:10.5555/380681
- Ruth Breeze. 2011. Disciplinary Values in Legal Discourse: A Corpus Study. Ibérica, Revista de la Asociación Europea de Lenguas para Fines Específicos 21 (2011), 93--115.
- Tom Broens, Dick Quartel, and Marten van Sinderen. 2007. Capturing Context Requirements. In Smart Sensing and Context, Gerd Kortuem, Joe Finney, Rodger Lea, and Vasughi Sundramoorthy (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 223--238.
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In FAT*. 77--91.
- Torkil Clemmensen and Kerstin Roese. 2010. An Overview of a Decade of Journal Publications about Culture and Human-Computer Interaction (HCI). In Human Work Interaction Design: Usability in Social, Cultural and Organizational Contexts, Dinesh Katre, Rikke Orngreen, Pradeep Yammiyavar, and Torkil Clemmensen (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 98--112.
- Harry Collins. 1992. Changing Order: Replication and Induction in Scientific Practice. University of Chicago Press.
- Geoff Cooper and John Bowers. 1995. Representing the user: Notes on the disciplinary rhetoric of human-computer interaction. Cambridge Series on Human Computer Interaction (1995), 48--66.
- Kate Crawford and Trevor Paglen. 2019. Excavating AI: The Politics of Images in Machine Learning Training Sets. Excavating AI (2019).
- Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
- Jacob Dexe, Ulrik Franke, Anneli Avatare Nöu, and Alexander Rad. 2020. Towards Increased Transparency with Value Sensitive Design. In Artificial Intelligence in HCI, Helmut Degen and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 3--15.
- 10.1016/0306--4573(94)90020--5 · doi:10.1016/0306--4573(94)90020--5
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2020. Datasheets for Datasets. arXiv:1803.09010 [cs] (March 2020). arXiv:1803.09010 [cs] · arXiv
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
- Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt.
- Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist studies 14, 3 (1988), 575--599.
- Mehtab Khan and Alex Hanna. 2020. The Legality of Computer Vision Datasets. Under review (2020).
- Colin Koopman. 2019. How We Became Our Data: A Genealogy of the Informational Person. The University of Chicago Press, Chicago.
- Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology. Sage publications.
- Bruno Latour. 1987. Science in action: How to follow scientists and engineers through society. Harvard university press.
- Bruno Latour and Steve Woolgar. 1986. Laboratory life: The construction of scientific facts. Princeton University Press.
- Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza-Ruiz. 2018. Challenges of language technologies for the indigenous languages of the Americas. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 55--69. https://www.aclweb.org/anthology/C18--1006
- 10.1007/978--3-030--15939--9 · doi:10.1007/978--3-030--15939--9
- Madhumita Murgia. 2019. Who's Using Your Face? The Ugly Truth about Facial Recognition. Financial Times (Sept. 2019).
- Michelle Murphy. 2017. The Economization of Life. Duke University Press, Durham ; London.
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Remi Denton, and Alex Hanna. 2020. Data and Its (Dis)Contents: A Survey of Dataset Development and Use in Machine Learning Research. arXiv:2012.05345 [cs] (Dec. 2020). arXiv:2012.05345 [cs] · arXiv
- Andrew Pickering. 2010. The Mangle of Practice: Time, Agency, and Science. University of Chicago Press.
- Inioluwa Deborah Raji, Morgan Klaus Scheuerman, and Razvan Amironesei. 2021. ?You Can't Sit With Us": Exclusionary Pedagogy in AI Ethics Education. In FAccT.
- Jemima Repo. 2015. The Biopolitics of Gender. Oxford University Press.
- 10.1016/S0048--7333(00)00091--3 · doi:10.1016/s0048--7333(00)00091--3
- James C. Scott. 2008. Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed (nachdr. ed.). Yale Univ. Press, New Haven, Conn.
- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc., 2503--2511.
- Evan Selinger and Woodrow Hartzog. 2019. What Happens When Employers Can Read Your Facial Expressions? The New York Times (Oct. 2019).
- 10.1007/978--1--4614--5583--7_441 · doi:10.1007/978--1--4614--5583--7_441
- Lewis Raven Wallace. 2019. The View from Somewhere: Undoing the Myth of Journalistic Objectivity. University of Chicago Press.
- Langdon Winner. 2020. The Whale and the Reactor: A Search for Limits in an Age of High Technology (second edition ed.). University of Chicago Press, Chicago.
- 10.18653/v1/D17--1323 · doi:10.18653/v1/d17--1323
Cited by
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Embedded in AI.. Papers within this theme advocate that HCAI can achieve value alignment by embodying human values, rather than just supporting them, through stakeholder engagement in the design process [ 62, 157, 165, 228, 232, 250].”
BackgroundMuse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157
“Such datasets have various potentially problematic issues such as consent and subject awareness (Paullada et al., 2021; Dulhanty, 2020; Scheuerman et al., 2021).”
BackgroundExploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., & Denton, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534647 · Cited by 72
Self-citationThe BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 93
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
“ance on crowdworkers, was seen as a boon to machine learning.(1,3,6,7) The enormous scale of such datasets has been mythologized as beneficial to the perceived generality of trained systems,(7) but they continue to be impacted by the limitations and biases that impact all datasets.”
BackgroundSelf-citationData Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
“Very recent research — most published within the last year — has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
BackgroundCan We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35
CounterFAccTual: How FAccT Undermines Its Organizing Principles
Gansky, B., & McDonald, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533241 · Cited by 34
“This is not to say that there are no FAccT stakes within the lab, nor that the lab is so easy to delineate by personnel or geography (e.g. [3, 21, 31, 55, 59]).”
BackgroundWe, the Data: Human Rights in the Digital Age
Wong, W. H. (2023) · doi:10.7551/mitpress/14703.001.0001 · Cited by 31
Who is Helping Whom? Student Concerns about AI-Teacher Collaboration in Higher Education Classrooms
Han, B., Coghlan, S., Buchanan, G., & McKay, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711104 · Cited by 31
Glinka, K., & Müller-Birn, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610054 · Cited by 26
“This aspect is already acknowledged within the CSCW and HCI community by calling for a more reflexive data practice (see [67]) or, more generally, a reflexive documentation practice (see [59]).”
BackgroundA Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication
Luccioni, A. S., Corry, F., Sridharan, H., Ananny, M., Schultz, J., & Crawford, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533086 · Cited by 22
From Bias to Repair: Error as a Site of Collaboration and Negotiation in Applied Data Science Work
Lin, C. K., & Jackson, S. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579607 · Cited by 22
Tanksley, T., Smith, A. D. R., Sharma, S., & Huff, E. W. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713510 · Cited by 19
Ecosystem Graphs: The Social Footprint of Foundation Models
Bommasani, R., Soylu, D., Liao, T., Creel, K., & Liang, P. (2023) · doi:10.21203/rs.3.rs-2961271/v1 · Cited by 14
Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp
Hong, R., Agnew, W., Kohno, T., & Morgenstern, J. (2024) · Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3689904.3694702 · Cited by 11
The Making of Performative Accuracy in AI Training: Precision Labor and Its Consequences
Zhang, B. Z., Yang, T., Miceli, M., Haimson, O. L., & Thomas, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713112 · Cited by 7
“Accuracy, together with efficiency, universality, impartiality, and scale, is among the prevailing preferences and values in ML communities associated with “success” and “progress” [5, 78].”
Rothschild, A., Booker, J., Davoll, C., Hill, J., Ivey, V., DiSalvo, C., Rydal Shapiro, B., & DiSalvo, B. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3516384 · Cited by 5
The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes
Hall, S. M., Dalal, S., Sefala, R., Yuehgoh, F., Alaagib, A., Hamzaoui, I., Ishida, S., Magomere, J., Crais, L., Salama, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.05961
Metadata
- Authors
- Scheuerman, M. K., Hanna, A., & Denton, R.
- Affiliations
- University of Colorado Boulder (US)
- Google (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-37
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3189849087
- Cited by
- 216
- References
- 79
- Citations by year
- 2026: 17 · 2025: 44 · 2024: 43 · 2023: 64 · 2022: 45 · 2021: 3
- FWCI
- 48.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- computer vision datasets, dataset documentation, disciplinary values, data curation, data annotation, data collection practices, data work, positionality, universality, efficiency, social computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.