Large image datasets: A pyrrhic win for computer vision?
Abstract
In this paper we investigate problematic practices and consequences of large scale vision datasets (LSVDs). We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets. Taking the ImageNet-ILSVRC-2012 dataset as an example, we perform a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class- wise accuracy, human-cardinality-analysis, and the semanticity of the image class information in order to statistically investigate the extent and subtleties of ethical transgressions. We then use the census to help hand-curate a look-up-table of images in the ImageNet-ILSVRC-2012 dataset that fall into the categories of verifiably pornographic: shot in a non-consensual setting (up-skirt), beach voyeuristic, and exposed private parts. We survey the landscape of harm and threats both the society at large and individuals face due to uncritical and ill-considered dataset curation practices. We then propose possible courses of correction and critique their pros and cons. We have duly open-sourced all of the code and the census meta-datasets generated in this endeavor for the computer vision community to build on. By unveiling the severity of the threats, our hope is to motivate the constitution of mandatory Institutional Review Boards (IRB) for large scale dataset curation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such audits have revealed the presence of misogyny, pornography and harmful stereotypes.
“Such audits have revealed the presence of misogyny, pornography and harmful stereotypes [5, 6]”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [5]
Auditing of Stable Diffusion's underlying datasets has revealed legal challenges and deeply problematic content.
“This has enabled auditing of the underlying datasets, which has revealed legal challenges [49] and deeply problematic content [4, 5].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [5]
Current data collection processes prioritize scale over inclusion, relying on web-scraping and synthetic pipelines that reflect developer biases and the uneven availability of online content.
“Current data collection processes prioritize scale over inclusion, relying on web-scraping (e.g., [40, 97]) and synthetic pipelines that reflect developer biases and the uneven availability of online content [13, 14, 57, 96].”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [14]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · see citing works
Sorting Things Out: Classification and Its Consequences
Bowker, G. C., & Star, S. L. (1999) · doi:10.7551/mitpress/6352.001.0001 · Cited by 6,188 · see citing works
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
(2018) · Cited by 3,510 · see citing works
Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2018) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,318 · see citing works
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,110 · see citing works
Algorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · doi:10.2307/j.ctt1pwt9w5 · Cited by 2,709 · see citing works
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372 · see citing works
(2006) · Cited by 2,256 · see citing works
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00982 · Cited by 2,194 · see citing works
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921 · see citing works
D'Ignazio, C., & Klein, L. F. (2020) · doi:10.7551/mitpress/11805.001.0001 · Cited by 1,738 · see citing works
Review of “Race After Technology: Abolitionist Tools for the New Jim Code”
Jipguep-Akhtar, M. C. (2019) · Social Forces · doi:10.1093/sf/soz162 · Cited by 1,482 · see citing works
(2019) · Cited by 1,082 · see citing works
Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks
Rothe, R., Timofte, R., & Gool, L. V. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0940-3 · Cited by 904 · see citing works
Estimating the success of re-identifications in incomplete datasets using generative models
Rocher, L., Hendrickx, J. M., & de Montjoye, Y.-A. (2019) · Nature Communications · doi:10.1038/s41467-019-10933-3 · Cited by 816 · see citing works
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790 · see citing works
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models
Menon, S., Damian, A., Hu, S., Ravi, N., & Rudin, C. D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00251 · Cited by 552 · see citing works
Machine Learning Models that Remember Too Much
Song, C., Ristenpart, T., & Shmatikov, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134077 · Cited by 526 · see citing works
Translating Principles into Practices of Digital Ethics: Five Risks of Being Unethical
Floridi, L. (2019) · Philosophy & Technology · doi:10.1007/s13347-019-00354-x · Cited by 524 · cited 1× · see citing works
“We are up against a system that has veritably mastered ethics shopping , ethics bluewashing , ethics lobbying , ethics dumping , and ethics shirking [33].”
BackgroundRetinaFace: Single-stage Dense Face Localisation in the WildPreprint
Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., & Zafeiriou, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00641 · Cited by 518 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition
Keyes, O. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274357 · Cited by 505 · see citing works
Beyond ‘Revenge Porn’: The Continuum of Image-Based Sexual Abuse
McGlynn, C., Rackley, E., & Houghton, R. (2017) · Feminist Legal Studies · doi:10.1007/s10691-017-9343-2 · Cited by 471 · see citing works
Learning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337 · see citing works
Consumers and Their Animal Companions
Hirschman, E. C. (1994) · Journal of Consumer Research · doi:10.1086/209374 · Cited by 310 · see citing works
Abbott, L. J., & Grady, C. (2011) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2011.6.1.3 · Cited by 292 · see citing works
Predictive Inequity in Object DetectionPreprint
Wilson, B., Hoffman, J., & Morgenstern, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11097 · Cited by 271 · see citing works
Yang, K., Qinami, K., Fei-Fei, L., Deng, J., & Russakovsky, O. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375709 · Cited by 254 · see citing works
Gonen, H., & Goldberg, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03862 · Cited by 242 · see citing works
Weindling, P. J. (2001) · Bulletin of the history of medicine · doi:10.1353/bhm.2001.0049 · Cited by 217 · see citing works
VisDA: A Synthetic-to-Real Benchmark for Visual Domain Adaptation
Peng, X., Usman, B., Kaushik, N., Wang, D., Hoffman, J., & Saenko, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00271 · Cited by 196 · see citing works
Wang, T., Zhu, J.-Y., Torralba, A., & Efros, A. A. (2018) · arXiv · doi:10.48550/arxiv.1811.10959 · Cited by 194 · see citing works
Fairness, Equality, and Power in Algorithmic Decision-Making
Kasy, M., & Abebe, R. (2021) · Conference on Fairness, Accountability and Transparency · doi:10.1145/3442188.3445919 · Cited by 193 · see citing works
Shankar, S., Halpern, Y., Breck, E. J., Atwood, J. R., Wilson, J., & Sculley, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08536 · Cited by 175 · see citing works
The Secretive Company That Might End Privacy as We Know It *
Hill, K. (2022) · Auerbach Publications eBooks · doi:10.1201/9781003278290-26 · Cited by 158 · see citing works
Star, S. L., & Bowker, G. C. (2007) · Ethics and Information Technology · doi:10.1007/s10676-007-9141-7 · Cited by 142 · see citing works
Data Augmentation Using GANsPreprint
Tanaka, F., & Aranha, C. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09135 · Cited by 137 · see citing works
Image Pixelization with Differential Privacy
Fan, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-95729-6_10 · Cited by 130 · see citing works
Why Do People Volunteer? An Experimental Analysis of Preferences for Time Donations
Brown, A. L., Meer, J., & Williams, J. F. (2018) · Management Science · doi:10.1287/mnsc.2017.2951 · Cited by 120 · see citing works
Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., & Zhang, T. (2019) · IEEE Access · doi:10.1109/access.2019.2956775 · Cited by 79 · see citing works
From ImageNet to Image Classification: Contextualizing Progress on BenchmarksPreprint
Tsipras, D., Santurkar, S., Engstrom, L., Ilyas, A., & Mądry, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11295 · Cited by 76 · see citing works
"My Dog's Just Like Me": Dog Ownership as a Gender Display
Ramirez, M. (2006) · Symbolic Interaction · doi:10.1525/si.2006.29.3.373 · Cited by 71 · see citing works
Algorithmic Injustices: Towards a Relational EthicsPreprint
Birhane, A., & Cummins, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07376 · Cited by 66 · see citing works
Self-citationCorbett, S. F. (2011) · Modern Law Review · doi:10.1111/j.1468-2230.2011.00858.x · Cited by 41 · see citing works
Gender and Musical Instrument Stereotypes in Middle School Children
Wrape, E. R., Dittloff, A. L., & Callahan, J. L. (2014) · Update Applications of Research in Music Education · doi:10.1177/8755123314564255 · Cited by 29 · see citing works
Time to discuss consent in digital-data studies
(2019) · Nature · doi:10.1038/d41586-019-02322-z · Cited by 24 · see citing works
Dulhanty, C., & Wong, A. M. (2020) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3375627.3375875 · Cited by 10 · see citing works
Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classificationPreprint
Prabhu, V. U., Han, S., Yap, D. A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08633 · Cited by 5 · see citing works
Self-citationInformed Consent the Global Picture
Blain, C., Mackay, M., & Tanner, J. (2002) · Journal of Perioperative Practice · doi:10.1177/175045890201201101 · Cited by 4 · see citing works
Informed consent for photography in dental practice : communication
Naidoo, S. (2009) · Cited by 2 · see citing works
Creative Commons Licences: A Symptom or a Cause?Preprint
Corbett, S. F. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.2028726 · Cited by 2 · see citing works
Where Have We Been? Where Are We Going?
Tedrow, M. K. (2017) · Routledge eBooks · doi:10.4324/9781315167572-4 · Cited by 2 · see citing works
Carroll, E., & Coates, J. (2011) · QUT ePrints (Queensland University of Technology) · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
On the genealogy of machine learning datasets: A critical history of ImageNet
Denton, E., Hanna, A., Amironesei, R., Smart, A., & Nicole, H. (2021) · Big Data & Society · doi:10.1177/20539517211035955 · Cited by 221 · see citing works
The algorithm audit: Scoring the algorithms that score us
Brown, S., Davidovic, J., & Hasan, A. (2021) · Big Data & Society · doi:10.1177/2053951720983865 · Cited by 204 · see citing works
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219 · see citing works
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021) · arXiv · doi:10.48550/arxiv.2110.01963 · Cited by 160 · see citing works
Self-citationMuse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157 · see citing works
“Many of the commonly used datasets tend to reflect negative social stereotypes and viewpoints (Prabhu & Birhane, 2020).”
BackgroundLiesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Rethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 67 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · see citing works
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
“[107] Vinay Uday Prabhu and Abeba Birhane.”
BackgroundTaxonomizing and Measuring Representational Harms: A Look at Image Tagging
Katzman, J., Wang, A., Scheuerman, M., Blodgett, S. L., Laird, K., Wallach, H., & Barocas, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26670 · Cited by 32 · see citing works
The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models
Birhane, A., Dehdashtian, S., Prabhu, V., & Boddeti, V. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658968 · Cited by 28 · see citing works
Self-citation‘We can see a savage’: a case study of the colonial gaze in generative AI algorithms
Alenichev, A., Shaffer, J. D., Kingori, P., Grietens, K. P., Muldoon, J., & Rocher, L. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02685-0 · Cited by 3 · see citing works
“In AI systems, these hierarchies can be reproduced or exacerbated within the datasets used to train computer vision models as well as within the model outputs (Buolamwini and Gebru 2018; Bhargava and Forsyth 2019; Birhane et al. 2021; Birhane and Prabhu 2021).”
InfluentialEngaging Communities Meaningfully in Defining Disability Representation for AI Image Generation
Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1 · see citing works
“Current data collection processes prioritize scale over inclusion, relying on web-scraping (e.g., [40, 97]) and synthetic pipelines that reflect developer biases and the uneven availability of online content [13, 14, 57, 96].”
- Authors
- Birhane, A., & Prabhu, V. U.
- Affiliations
- University College Dublin (IE)
- Venue
- 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2021-01
- Pages
- 1536-1546
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W3119746452
- Cited by
- 257
- References
- 64
- Citations by year
- 2026: 19 · 2025: 47 · 2024: 54 · 2023: 58 · 2022: 49 · 2021: 29 · 2020: 1
- FWCI
- 42.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- Funders
- Science Foundation Ireland
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
- Keywords
- large-scale image databases, consent, justice, data curation, research ethics committees
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review