The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Open accessAbstract
Data is a crucial component of machine learning. The field is reliant on data to train, validate, and test models. With increased technical capabilities, machine learning research has boomed in both academic and industry settings, and one major focus has been on computer vision. Computer vision is a popular domain of machine learning increasingly pertinent to real-world applications, from facial recognition in policing to object detection for autonomous vehicles. Given computer vision's propensity to shape machine learning research and impact human life, we seek to understand disciplinary practices around dataset documentation - how data is collected, curated, annotated, and packaged into datasets for computer vision researchers and practitioners to use for model tuning and development. Specifically, we examine what dataset documentation communicates about the underlying values of vision data and the larger practices and goals of computer vision as a field. To conduct this study, we collected a corpus of about 500 computer vision datasets, from which we sampled 114 dataset publications across different vision tasks. Through both a structured and thematic content analysis, we document a number of values around accepted data practices, what makes desirable data, and the treatment of humans in the dataset construction process. We discuss how computer vision datasets authors value efficiency at the expense of care; universality at the expense of contextuality; impartiality at the expense of positionality; and model work at the expense of data work. Many of the silenced values we identify sit in opposition with social computing practices. We conclude with suggestions on how to better incorporate silenced values into the dataset creation and curation process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize ' efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . '.”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Scheuerman et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Very recent research - most published within the last year - has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Situating knowledges: positionality, reflexivities and other tactics
Rose, G. (1997) · Progress in Human Geography · doi:10.1191/030913297673302122 · Cited by 2,390
Race and the Education of Desire: Foucault’s History of Sexuality and the Colonial Order of Things
Stoler, A. L. (1995) · Duke University Press eBooks · doi:10.1215/9780822377719 · Cited by 2,194
(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
Computer-supported cooperative work: history and focus
Grudin, J. (1994) · Computer · doi:10.1109/2.291294 · Cited by 1,639
The economic benefits of publicly funded basic research: a critical review
Salter, A. J., & Martin, B. R. (2001) · Research Policy · doi:10.1016/s0048-7333(00)00091-3 · Cited by 1,358
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9302.001.0001 · Cited by 1,160
Feminist HCI: taking stock and outlining an agenda for design
Bardzell, S. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753521 · Cited by 1,123
Research works printed but not linked: 41
- Shazia Afzal, Rajmohan C, Manish Kesarwani, Sameep Mehta, and Hima Patel. 2020. Data Readiness Report. arXiv:2010.07213 [cs] (Oct. 2020). arXiv:2010.07213 [cs] · arXiv
- Philip E. Agre. 1998. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. Psychology Press.
- Carolyn Ashurst Markus Anderljung Carina Prunkl Jan Leike Yarin Gal Toby Shevlane and Allan Dafoe. 2020. A Guide to Writing the NeurIPS Impact Statement. (2020).
- Christine L Borgman. 2016. Big Data, Little Data, No Data: Scholarship in the Networked World. MIT press.
- Geoffrey C Bowker. 2005. Memory Practices in the Sciences. Mit Press Cambridge, MA.
- 10.5555/380681 · doi:10.5555/380681
- Ruth Breeze. 2011. Disciplinary Values in Legal Discourse: A Corpus Study. Ibérica, Revista de la Asociación Europea de Lenguas para Fines Específicos 21 (2011), 93--115.
- Tom Broens, Dick Quartel, and Marten van Sinderen. 2007. Capturing Context Requirements. In Smart Sensing and Context, Gerd Kortuem, Joe Finney, Rodger Lea, and Vasughi Sundramoorthy (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 223--238.
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In FAT*. 77--91.
- Torkil Clemmensen and Kerstin Roese. 2010. An Overview of a Decade of Journal Publications about Culture and Human-Computer Interaction (HCI). In Human Work Interaction Design: Usability in Social, Cultural and Organizational Contexts, Dinesh Katre, Rikke Orngreen, Pradeep Yammiyavar, and Torkil Clemmensen (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 98--112.
- Harry Collins. 1992. Changing Order: Replication and Induction in Scientific Practice. University of Chicago Press.
- Geoff Cooper and John Bowers. 1995. Representing the user: Notes on the disciplinary rhetoric of human-computer interaction. Cambridge Series on Human Computer Interaction (1995), 48--66.
- Kate Crawford and Trevor Paglen. 2019. Excavating AI: The Politics of Images in Machine Learning Training Sets. Excavating AI (2019).
- Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
- Jacob Dexe, Ulrik Franke, Anneli Avatare Nöu, and Alexander Rad. 2020. Towards Increased Transparency with Value Sensitive Design. In Artificial Intelligence in HCI, Helmut Degen and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 3--15.
- 10.1016/0306--4573(94)90020--5 · doi:10.1016/0306--4573(94)90020--5
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2020. Datasheets for Datasets. arXiv:1803.09010 [cs] (March 2020). arXiv:1803.09010 [cs] · arXiv
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
- Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt.
- Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist studies 14, 3 (1988), 575--599.
- Mehtab Khan and Alex Hanna. 2020. The Legality of Computer Vision Datasets. Under review (2020).
- Colin Koopman. 2019. How We Became Our Data: A Genealogy of the Informational Person. The University of Chicago Press, Chicago.
- Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology. Sage publications.
- Bruno Latour. 1987. Science in action: How to follow scientists and engineers through society. Harvard university press.
- Bruno Latour and Steve Woolgar. 1986. Laboratory life: The construction of scientific facts. Princeton University Press.
- Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza-Ruiz. 2018. Challenges of language technologies for the indigenous languages of the Americas. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 55--69. https://www.aclweb.org/anthology/C18--1006
- 10.1007/978--3-030--15939--9 · doi:10.1007/978--3-030--15939--9
- Madhumita Murgia. 2019. Who's Using Your Face? The Ugly Truth about Facial Recognition. Financial Times (Sept. 2019).
- Michelle Murphy. 2017. The Economization of Life. Duke University Press, Durham ; London.
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Remi Denton, and Alex Hanna. 2020. Data and Its (Dis)Contents: A Survey of Dataset Development and Use in Machine Learning Research. arXiv:2012.05345 [cs] (Dec. 2020). arXiv:2012.05345 [cs] · arXiv
- Andrew Pickering. 2010. The Mangle of Practice: Time, Agency, and Science. University of Chicago Press.
- Inioluwa Deborah Raji, Morgan Klaus Scheuerman, and Razvan Amironesei. 2021. ?You Can't Sit With Us": Exclusionary Pedagogy in AI Ethics Education. In FAccT.
- Jemima Repo. 2015. The Biopolitics of Gender. Oxford University Press.
- 10.1016/S0048--7333(00)00091--3 · doi:10.1016/s0048--7333(00)00091--3
- James C. Scott. 2008. Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed (nachdr. ed.). Yale Univ. Press, New Haven, Conn.
- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc., 2503--2511.
- Evan Selinger and Woodrow Hartzog. 2019. What Happens When Employers Can Read Your Facial Expressions? The New York Times (Oct. 2019).
- 10.1007/978--1--4614--5583--7_441 · doi:10.1007/978--1--4614--5583--7_441
- Lewis Raven Wallace. 2019. The View from Somewhere: Undoing the Myth of Journalistic Objectivity. University of Chicago Press.
- Langdon Winner. 2020. The Whale and the Reactor: A Search for Limits in an Age of High Technology (second edition ed.). University of Chicago Press, Chicago.
- 10.18653/v1/D17--1323 · doi:10.18653/v1/d17--1323
Cited by
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Embedded in AI.. Papers within this theme advocate that HCAI can achieve value alignment by embodying human values, rather than just supporting them, through stakeholder engagement in the design process [ 62, 157, 165, 228, 232, 250].”
BackgroundMuse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157
“Such datasets have various potentially problematic issues such as consent and subject awareness (Paullada et al., 2021; Dulhanty, 2020; Scheuerman et al., 2021).”
Background
Metadata
- Authors
- Scheuerman, M. K., Hanna, A., & Denton, R.
- Affiliations
- University of Colorado Boulder (US)
- Google (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-37
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3189849087
- Cited by
- 216
- References
- 79
- Citations by year
- 2026: 17 · 2025: 44 · 2024: 43 · 2023: 64 · 2022: 45 · 2021: 3
- FWCI
- 48.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- computer vision datasets, dataset documentation, disciplinary values, data curation, data annotation, data collection practices, data work, positionality, universality, efficiency, social computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.