The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
Open accessAbstract
Data is a crucial component of machine learning. The field is reliant on data to train, validate, and test models. With increased technical capabilities, machine learning research has boomed in both academic and industry settings, and one major focus has been on computer vision. Computer vision is a popular domain of machine learning increasingly pertinent to real-world applications, from facial recognition in policing to object detection for autonomous vehicles. Given computer vision's propensity to shape machine learning research and impact human life, we seek to understand disciplinary practices around dataset documentation - how data is collected, curated, annotated, and packaged into datasets for computer vision researchers and practitioners to use for model tuning and development. Specifically, we examine what dataset documentation communicates about the underlying values of vision data and the larger practices and goals of computer vision as a field. To conduct this study, we collected a corpus of about 500 computer vision datasets, from which we sampled 114 dataset publications across different vision tasks. Through both a structured and thematic content analysis, we document a number of values around accepted data practices, what makes desirable data, and the treatment of humans in the dataset construction process. We discuss how computer vision datasets authors value efficiency at the expense of care; universality at the expense of contextuality; impartiality at the expense of positionality; and model work at the expense of data work. Many of the silenced values we identify sit in opposition with social computing practices. We conclude with suggestions on how to better incorporate silenced values into the dataset creation and curation process.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Scheuerman et al. (2021) outline how data efforts in computer vision tend to prioritize ' efficiency [over] care; universality [over] contextuality; impartiality [over] positionality . . . '.”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Scheuerman et al. (2021)
The cited source says what the claim attributes to it.
In the citing paper
“Very recent research - most published within the last year - has begun to analyze dataset values [18, 40], question assumptions around dataset use [100], unpack what is represented in ML datasets [43, 88, 124] and establish basics of how an organized dataset lifecycle might proceed [74].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Situating knowledges: positionality, reflexivities and other tactics
Rose, G. (1997) · Progress in Human Geography · doi:10.1191/030913297673302122 · Cited by 2,390
Race and the Education of Desire: Foucault’s History of Sexuality and the Colonial Order of Things
Stoler, A. L. (1995) · Duke University Press eBooks · doi:10.1215/9780822377719 · Cited by 2,194
(2020) · doi:10.7551/mitpress/12255.001.0001 · Cited by 1,706
Computer-supported cooperative work: history and focus
Grudin, J. (1994) · Computer · doi:10.1109/2.291294 · Cited by 1,639
The economic benefits of publicly funded basic research: a critical review
Salter, A. J., & Martin, B. R. (2001) · Research Policy · doi:10.1016/s0048-7333(00)00091-3 · Cited by 1,358
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9302.001.0001 · Cited by 1,160
Feminist HCI: taking stock and outlining an agenda for design
Bardzell, S. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753521 · Cited by 1,123
Sengers, P., Boehner, K., David, S., & Kaye, J. '. (2005) · Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility · doi:10.1145/1094562.1094569 · Cited by 1,030
Friedman, B. (1996) · Interactions · doi:10.1145/242485.242493 · Cited by 966
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 4×
“have all put forth data reporting frameworks for improving dataset documentation [10, 43, 57].”
BackgroundMethodInfluentialMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Sustainable interaction design: invention & disposal, renewal & reuse
Blevis, E. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240705 · Cited by 771
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
The relationship of action research to human-computer interaction
Hayes, G. R. (2011) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1993060.1993065 · Cited by 452
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404
Do categories have politics?: The language/action perspective reconsidered
Suchman, L. (1993) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/bf00749015 · Cited by 400
Ogbonnaya-Ogburu, I. F., Smith, A. D. R., To, A., & Toyama, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376392 · Cited by 396
Where are human subjects in big data research? the emerging ethics divide
Metcalf, J., & Crawford, K. (2016) · Big Data & Society · doi:10.1177/2053951716650211 · Cited by 394
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Is paper safer? The role of paper flight strips in air traffic control
MacKay, W. E. (1999) · ACM Transactions on Computer-Human Interaction · doi:10.1145/331490.331491 · Cited by 356
Undefined By Data: A Survey of Big Data Definitions
Ward, J. S., & Barker, A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.5821 · Cited by 356
DeVito, M. A., Gergle, D., & Birnholtz, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025659 · Cited by 311
Be(com)ing a reflexive researcher: a developmental approach to research methodology
Attia, M., & Edge, J. (2017) · Open Review of Educational Research · doi:10.1080/23265507.2017.1300068 · Cited by 305
Knowledge, Human Interests, and Objectivity in Feminist Epistemology
Anderson, E., & University of Arkansas Press (1995) · Philosophical Topics · doi:10.5840/philtopics199523213 · Cited by 303
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295 · cited 1×
“…design frameworks which have explicitly embraced specific politics and values [8, 24, 65, 81] Particularly with its focus on real-world applications, the artifacts produced by machine learning researchers and practitioners — for example, in the form of datasets — are value-laden [51, 98, 107].”
Self-citationHutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279
Self-citationLitman, L., Robinson, J., & Rosenzweig, C. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275
The notion of data and its quality dimensions
Fox, C., Levitin, A., & Redman, T. (1994) · Information Processing & Management · doi:10.1016/0306-4573(94)90020-5 · Cited by 265
Large image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257 · cited 3×
“A recent audit of ImageNet found the dataset contained significant gender biases, and even the inclusion of non-consensual pornographic imagery, depicting predominantly women [89].”
Scheuerman, M. K., Paul, J. M., & Brubaker, J. R. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359246 · Cited by 252
Self-citationLessons from archives: strategies for collecting sociocultural data in machine learning
Jo, E. S., & Gebru, T. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372829 · Cited by 240
For You, or For"You"?: Everyday LGBTQ+ Encounters with TikTok
Simpson, E., & Semaan, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432951 · Cited by 230
On the Reuse of Scientific Data
Pasquetto, I. V., Randles, B. M., & Borgman, C. L. (2017) · Data Science Journal · doi:10.5334/dsj-2017-008 · Cited by 223
Roles for computing in social change
Abebe, R., Barocas, S., Kleinberg, J., Levy, K., Raghavan, M., & Robinson, D. G. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372871 · Cited by 219
"Coming Out Okay": Community Narratives for LGBTQ Identity Recovery Work
Dym, B., Brubaker, J. R., Fiesler, C., & Semaan, B. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359256 · Cited by 210
Scheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392866 · Cited by 193 · cited 6×
“2 Scheuerman et al. raised concerns regarding the use of gender and racial categories in facial analysis datasets, finding that dataset creators tend to present these classifications as natural or immutable characteristics [98].”
InfluentialSelf-citationGil, Y., David, C. H., Demir, I., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 180
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
Chancellor, S., Baumer, E. P. S., & De Choudhury, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359249 · Cited by 172
(2019) · doi:10.1145/3351095.3372862 · Cited by 167
Shankar, S., Halpern, Y., Breck, E. J., Atwood, J. R., Wilson, J., & Sculley, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08536 · Cited by 167
Measuring the social impact of research
Smith, R. W. (2001) · BMJ · doi:10.1136/bmj.323.7312.528 · Cited by 154
Facial recognition is the plutonium of AI
Stark, L. (2019) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3313129 · Cited by 150
The value of data: considering the context of production in data economies
Vertesi, J., & Dourish, P. (2011) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1958824.1958906 · Cited by 134
Big? Smart? Clean? Messy? Data In The Humanities
Schöch, C. (2013) · OpenEdition (OpenEdition) · doi:10.5281/zenodo.8432 · Cited by 131
Documenting Computer Vision Datasets: An Invitation to Reflexive Data Practices
Miceli, M., Yang, T., Naudts, L., Schuessler, M., Serbanescu, D. A., & Hanna, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445880 · Cited by 123
Self-citationThe Rôle of the Organization in the Production of Techno-Scientific Knowledge
Vaughan, D. (1999) · Social Studies of Science · doi:10.1177/030631299029006005 · Cited by 122
Becher, T. (1987) · Studies in Higher Education · doi:10.1080/03075078712331378052 · Cited by 122
On the Ethics of Crowdsourced Research
Williamson, V. S. (2016) · PS Political Science & Politics · doi:10.1017/s104909651500116x · Cited by 106
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100
Self-citationFellbaum, C. (2012) · The Encyclopedia of Applied Linguistics · doi:10.1002/9781405198431.wbeal1285 · Cited by 96
Large image datasets: A pyrrhic win for computer vision?
Prabhu, V. U., & Birhane, A. (2020) · arXiv · doi:10.48550/arxiv.2006.16923 · Cited by 91
Pittman, M., & Sheehan, K. B. (2016) · Journal of Media Ethics · doi:10.1080/23736992.2016.1228811 · Cited by 73
Good systems, bad data?: Interpretations of AI hype and failures
Slota, S. C., Fleischmann, K. R., Greenberg, S. R., Verma, N., Cummings, B., Li, L., & Shenefiel, C. (2020) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.275 · Cited by 60
Situational When: Designing for Time Across Cultures
Taylor, J. L., Soro, A., Roe, P., Hong, A. L., & Brereton, M. F. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025936 · Cited by 55
García, P. M., & Cifor, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359292 · Cited by 53
Opportunities and Challenges in Involving Users in Project-Based HCI Education
Roldan, W., Gao, X., Hishikawa, A. M., Ku, T., Li, Z., Zhang, E., Froehlich, J. E., & Yip, J. C. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376530 · Cited by 52
Garcia (2020) · doi:10.1145/3406865.3419014 · Cited by 51 · cited 1×
“Employing empirical methods, such as surveys, interviews, or ethnography, can ground dataset curation decisions and make datasets more contextually useful for intended stakeholders.”
Relational, Flexible, Everyday: Learning from Ethics in Dementia Research
Hodge, J., Foley, S., Brankaert, R., Kenning, G., Lazar, A., Boger, J. N., & Morrissey, K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376627 · Cited by 50
Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards
Sen, S., Giesel, M. E., Gold, R., Hillmann, B., Lesicko, M., Naden, S., Russell, J., Wang, Z. (., & Hecht, B. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675285 · Cited by 47
Behind the Voices: The Practice and Challenges of Esports Casters
Kempe-Cook, L., Sher, S. T.-H., & Su, N. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300795 · Cited by 39
(2020) · Routledge eBooks · doi:10.4324/9781003135838 · Cited by 36
Muller, M. J., Aragon, C., Guha, S., Kogan, M., Neff, G., Seidelin, C., Shilton, K., & Tanweer, A. (2020) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3406865.3418584 · Cited by 29
Stodden, V., Krafczyk, M. S., & Bhaskar, A. (2018) · International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS) · doi:10.1145/3214239.3214242 · Cited by 27
Teo, T. (2014) · Springer eBooks · doi:10.1007/978-1-4614-5583-7_441 · Cited by 22
Shifting Expectations: Understanding Youth Employees' Handoffs in a 3D Print Shop
Easley, W., Hamidi, F., Lutters, W. G., & Hurst, A. K. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274316 · Cited by 19
Visual Cultural Biases in Food Classification
Zhang, Q., Elsweiler, D., & Trattner, C. (2020) · Foods · doi:10.3390/foods9060823 · Cited by 18
Positionality-aware machine learning: translation tutorial
Kaeser‐Chen, C., Dubois, E., Schüür, F., & Moss, E. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3375666 · Cited by 15
Human-Computer Insurrection: Notes on an Anarchist HCIPreprint
Keyes, O., Hoy, J., & Drouhard, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06167 · Cited by 15
A Paradigm for Democratizing Artificial Intelligence Research
Moreau, E., Vogel, C., & Barry, M. (2019) · Intelligent Systems Reference Library · doi:10.1007/978-3-030-15939-9_8 · Cited by 10
Toward a value‐analytic approach to information standards
Dobreski, B. (2018) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2018.14505501013 · Cited by 3
'Disciplining' Truth and Science: Michel Foucault and the Power of Social Science
Powell, J. L. (2015) · ChesterRep (University of Chester) · Cited by 2
Guide to Social Science Data Preparation and Archiving: Best Practice Throughout the Data Life Cycle
Inter-University Consortium For Political And Social Research (2012) · Deep Blue (University of Michigan) · doi:10.3886/guidetosocialsciencedatapreparationandarchiving · Cited by 2
(2013) · Museum Anthropology Review (Indiana University)
Research works printed but not linked: 41
- Shazia Afzal, Rajmohan C, Manish Kesarwani, Sameep Mehta, and Hima Patel. 2020. Data Readiness Report. arXiv:2010.07213 [cs] (Oct. 2020). arXiv:2010.07213 [cs] · arXiv
- Philip E. Agre. 1998. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. Psychology Press.
- Carolyn Ashurst Markus Anderljung Carina Prunkl Jan Leike Yarin Gal Toby Shevlane and Allan Dafoe. 2020. A Guide to Writing the NeurIPS Impact Statement. (2020).
- Christine L Borgman. 2016. Big Data, Little Data, No Data: Scholarship in the Networked World. MIT press.
- Geoffrey C Bowker. 2005. Memory Practices in the Sciences. Mit Press Cambridge, MA.
- 10.5555/380681 · doi:10.5555/380681
- Ruth Breeze. 2011. Disciplinary Values in Legal Discourse: A Corpus Study. Ibérica, Revista de la Asociación Europea de Lenguas para Fines Específicos 21 (2011), 93--115.
- Tom Broens, Dick Quartel, and Marten van Sinderen. 2007. Capturing Context Requirements. In Smart Sensing and Context, Gerd Kortuem, Joe Finney, Rodger Lea, and Vasughi Sundramoorthy (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 223--238.
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In FAT*. 77--91.
- Torkil Clemmensen and Kerstin Roese. 2010. An Overview of a Decade of Journal Publications about Culture and Human-Computer Interaction (HCI). In Human Work Interaction Design: Usability in Social, Cultural and Organizational Contexts, Dinesh Katre, Rikke Orngreen, Pradeep Yammiyavar, and Torkil Clemmensen (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 98--112.
- Harry Collins. 1992. Changing Order: Replication and Induction in Scientific Practice. University of Chicago Press.
- Geoff Cooper and John Bowers. 1995. Representing the user: Notes on the disciplinary rhetoric of human-computer interaction. Cambridge Series on Human Computer Interaction (1995), 48--66.
- Kate Crawford and Trevor Paglen. 2019. Excavating AI: The Politics of Images in Machine Learning Training Sets. Excavating AI (2019).
- Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
- Jacob Dexe, Ulrik Franke, Anneli Avatare Nöu, and Alexander Rad. 2020. Towards Increased Transparency with Value Sensitive Design. In Artificial Intelligence in HCI, Helmut Degen and Lauren Reinerman-Jones (Eds.). Springer International Publishing, Cham, 3--15.
- 10.1016/0306--4573(94)90020--5 · doi:10.1016/0306--4573(94)90020--5
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2020. Datasheets for Datasets. arXiv:1803.09010 [cs] (March 2020). arXiv:1803.09010 [cs] · arXiv
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
- Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt.
- Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist studies 14, 3 (1988), 575--599.
- Mehtab Khan and Alex Hanna. 2020. The Legality of Computer Vision Datasets. Under review (2020).
- Colin Koopman. 2019. How We Became Our Data: A Genealogy of the Informational Person. The University of Chicago Press, Chicago.
- Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology. Sage publications.
- Bruno Latour. 1987. Science in action: How to follow scientists and engineers through society. Harvard university press.
- Bruno Latour and Steve Woolgar. 1986. Laboratory life: The construction of scientific facts. Princeton University Press.
- Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza-Ruiz. 2018. Challenges of language technologies for the indigenous languages of the Americas. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 55--69. https://www.aclweb.org/anthology/C18--1006
- 10.1007/978--3-030--15939--9 · doi:10.1007/978--3-030--15939--9
- Madhumita Murgia. 2019. Who's Using Your Face? The Ugly Truth about Facial Recognition. Financial Times (Sept. 2019).
- Michelle Murphy. 2017. The Economization of Life. Duke University Press, Durham ; London.
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Remi Denton, and Alex Hanna. 2020. Data and Its (Dis)Contents: A Survey of Dataset Development and Use in Machine Learning Research. arXiv:2012.05345 [cs] (Dec. 2020). arXiv:2012.05345 [cs] · arXiv
- Andrew Pickering. 2010. The Mangle of Practice: Time, Agency, and Science. University of Chicago Press.
- Inioluwa Deborah Raji, Morgan Klaus Scheuerman, and Razvan Amironesei. 2021. ?You Can't Sit With Us": Exclusionary Pedagogy in AI Ethics Education. In FAccT.
- Jemima Repo. 2015. The Biopolitics of Gender. Oxford University Press.
- 10.1016/S0048--7333(00)00091--3 · doi:10.1016/s0048--7333(00)00091--3
- James C. Scott. 2008. Seeing like a State: How Certain Schemes to Improve the Human Condition Have Failed (nachdr. ed.). Yale Univ. Press, New Haven, Conn.
- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc., 2503--2511.
- Evan Selinger and Woodrow Hartzog. 2019. What Happens When Employers Can Read Your Facial Expressions? The New York Times (Oct. 2019).
- 10.1007/978--1--4614--5583--7_441 · doi:10.1007/978--1--4614--5583--7_441
- Lewis Raven Wallace. 2019. The View from Somewhere: Undoing the Myth of Journalistic Objectivity. University of Chicago Press.
- Langdon Winner. 2020. The Whale and the Reactor: A Search for Limits in an Age of High Technology (second edition ed.). University of Chicago Press, Chicago.
- 10.18653/v1/D17--1323 · doi:10.18653/v1/d17--1323
Cited by
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Advances, challenges and opportunities in creating data for trustworthy AI
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00516-1 · Cited by 559
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 633
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Embedded in AI.. Papers within this theme advocate that HCAI can achieve value alignment by embodying human values, rather than just supporting them, through stakeholder engagement in the design process [ 62, 157, 165, 228, 232, 250].”
BackgroundMuse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157
“Such datasets have various potentially problematic issues such as consent and subject awareness (Paullada et al., 2021; Dulhanty, 2020; Scheuerman et al., 2021).”
Background
Metadata
- Authors
- Scheuerman, M. K., Hanna, A., & Denton, R.
- Affiliations
- University of Colorado Boulder (US)
- Google (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-10-13
- Volume
- 5
- Issue
- CSCW2
- Pages
- 1-37
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3189849087
- Cited by
- 216
- References
- 79
- Citations by year
- 2026: 17 · 2025: 44 · 2024: 43 · 2023: 64 · 2022: 45 · 2021: 3
- FWCI
- 48.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Sociology and Political Science › Data Analysis and Archiving
- Keywords
- computer vision datasets, dataset documentation, disciplinary values, data curation, data annotation, data collection practices, data work, positionality, universality, efficiency, social computing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.