CoChat
Sign Up
PreprintCited by 91Ethics and Social Impacts of AIAdvanced Neural Network ApplicationsLaw, AI, and Intellectual Property

Large image datasets: A pyrrhic win for computer vision?

Open access
Prabhu, Vinay Uday, Birhane, Abeba·2020·arXiv
DOI 10.48550/arxiv.2006.16923arXiv 2006.16923
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper we investigate problematic practices and consequences of large scale vision datasets. We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets. Taking the ImageNet-ILSVRC-2012 dataset as an example, we perform a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class-wise accuracy, human-cardinality-analysis, and the semanticity of the image class information in order to statistically investigate the extent and subtleties of ethical transgressions. We then use the census to help hand-curate a look-up-table of images in the ImageNet-ILSVRC-2012 dataset that fall into the categories of verifiably pornographic: shot in a non-consensual setting (up-skirt), beach voyeuristic, and exposed private parts. We survey the landscape of harm and threats both society broadly and individuals face due to uncritical and ill-considered dataset curation practices. We then propose possible courses of correction and critique the pros and cons of these. We have duly open-sourced all of the code and the census meta-datasets generated in this endeavor for the computer vision community to build on. By unveiling the severity of the threats, our hope is to motivate the constitution of mandatory Institutional Review Boards (IRB) for large scale dataset curation processes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.16923

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There has been an increased focus on analyzing other dimensions of data quality and stewardship, with several initiatives aiming to document existing and newly developed resources.

    “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”

    From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [107]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 67

  1. Md Zahangir Alom, Tarek M Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S Awwal, and Vijayan K Asari. The history began from alexnet: A comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164, 2018. · arXiv
  2. Stephanie Baran. Visual patriarchy: Peta advertising and the commodification of sexualized bodies. In Women and Nature?, pages 43-56. Routledge, 2017.
  3. Ruha Benjamin. Race after technology: Abolitionist tools for the new jim code. John Wiley & Sons, 2019.
  4. Abeba Birhane and Fred Cummins. Algorithmic injustices: Towards a relational ethics. arXiv preprint arXiv:1912.07376, 2019. · arXiv
  5. Colin Blain, Margaret Mackay, and Judith Tanner. Informed consent the global picture. British Journal of Perioperative Nursing (United Kingdom), 12(11):402-407, 2002.
  6. Margaret A Boden and Ernest A Edmonds. What is generative art? Digital Creativity, 20(1-2):21-46, 2009.
  7. Geoffrey C Bowker and Susan Leigh Star. Sorting things out: Classification and its consequences. MIT press, 2000.
  8. Alexander L Brown, Jonathan Meer, and J Forrest Williams. Why do people volunteer? an experimental analysis of preferences for time donations. Management Science, 65(4):1455-1468, 2019.
  9. Claudia Bullerjahn, Katharina Heller, and Jan Hoffmann. How masculine is a flute? a replication study on gender stereotypes and preferences for musical instruments among young children. In Proceedings of the 14th International Conference on Music Perception and Cognition, pages 5-9, 2016.
  10. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91, 2018.
  11. Emma Carroll and Jessica Coates. The school girl, the billboard, and virgin: The virgin mobile case and the use of creative commons licensed photographs by commercial entities. Knowledge policy for the 21st century. A legal perspective, pages 181-204, 2011.
  12. François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
  13. Susan Corbett. Creative commons licences: A symptom or a cause? Available at SSRN 2028726, 2009.
  14. Susan Corbett. Creative commons licences, the copyright regime and the online community: Is there a fatal disconnect? The Modern Law Review, 74(4):503-531, 2011.
  15. Judith K Delzell and David A Leppla. Gender association of musical instruments and preferences of fourth-grade students for selected instruments. Journal of research in music education, 40(2):93-103, 1992.
  16. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
  17. Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-stage dense face localisation in the wild. In arxiv, 2019.
  18. Catherine D'Ignazio and Lauren F Klein. Data feminism. MIT Press, 2020.
  19. Fabio Henrique Kiyoiti dos Santos Tanaka and Claus Aranha. Data augmentation using gans. Proceedings of Machine Learning Research XXX, 1:16, 2019.
  20. Chris Dulhanty and Alexander Wong. Auditing imagenet: Towards a model-driven framework for annotating demographic attributes of large-scale image datasets. arXiv preprint arXiv:1905.01347, 2019. · arXiv
  21. Robert Eaglestone. One and the same? ethics, aesthetics, and truth. Poetics Today, 25(4):595-608, 2004.
  22. Editorial. Time to discuss consent in digital-data studies. https://www.nature.com/articles/ d41586-019-02322-z, July 2019. (Accessed on 06/02/2020).
  23. Virginia Eubanks. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press, 2018.
  24. Liyue Fan. Image pixelization with differential privacy. In IFIP Annual Conference on Data and Applications Security and Privacy, pages 148-162. Springer, 2018.
  25. Luciano Floridi. Translating principles into practices of digital ethics: five risks of being unethical. Philosophy & Technology, 32(2):185-193, 2019.
  26. Brendan J Frey and Delbert Dueck. Clustering by passing messages between data points. science, 315(5814):972-976, 2007.
  27. Alisdair A Gillespie. Tackling voyeurism: Is the voyeurism (offences) act 2019 a wasted opportunity? The Modern Law Review, 82(6):1107-1131, 2019.
  28. Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 609-614, 2019.
  29. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. · arXiv
  30. Elizabeth C Hirschman. Consumers and their animal companions. Journal of consumer research, 20(4):616-632, 1994.
  31. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. · arXiv
  32. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
  33. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982, 2018. · arXiv
  34. Clare McGlynn and Erika Rackley. More than revenge porn: image-based sexual abuse and the reform of irish law. Irish probation journal., 14:38-51, 2017.
  35. Clare McGlynn, Erika Rackley, and Ruth Houghton. Beyond revenge porn: The continuum of image-based sexual abuse. Feminist Legal Studies, 25(1):25-46, 2017.
  36. Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. · arXiv
  37. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, In- ioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. arXiv preprint arXiv:1810.03993, 2018. · arXiv
  38. S Naidoo. Informed consent for photography in dental practice: communication. South African Dental Journal, 64(9):404-406, 2009.
  39. Arvind Narayanan and Vitaly Shmatikov. Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008), pages 111-125. IEEE, 2008.
  40. Safiya Umoja Noble. Algorithms of oppression: How search engines reinforce racism. nyu Press, 2018.
  41. Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2021-2026, 2018.
  42. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
  43. Anastasia Powell. Configuring consent: Emerging technologies, unauthorized sexual images and sexual assault. Australian & New Zealand journal of criminology, 43(1):76-90, 2010.
  44. Anastasia Powell, Nicola Henry, and Asher Flynn. Image-based sexual abuse. In Routledge handbook of critical criminology, pages 305-315. Routledge, 2018.
  45. Vinay Uday Prabhu, Sanghyun Han, Dian Ang Yap, Mihail Douhaniaris, Preethi Seshadri, and John Whaley. Fonts-2- handwriting: A seed-augment-train framework for universal digit classification. arXiv preprint arXiv:1905.08633, 2019. · arXiv
  46. Michael Ramirez. "my dog's just like me": Dog ownership as a gender display. Symbolic Interaction, 29(3):373-391, 2006.
  47. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. · arXiv
  48. Navid Rekabsaz, James Henderson, Robert West, and Allan Hanbury. Measuring societal biases in text corpora via first-order co-occurrence. arXiv:1812.10424 [cs, stat], Apr 2020. arXiv: 1812.10424. · arXiv
  49. Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2-4):144-157, 2018.
  50. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
  51. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536, 2017. · arXiv
  52. Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 587-601, 2017.
  53. Susan Leigh Star and Geoffrey C Bowker. Enacting silence: Residual categories as a challenge for ethics, information systems, and communication. Ethics and Information Technology, 9(4):273-280, 2007.
  54. Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the European Conference on Computer Vision (ECCV), pages 498-512, 2018.
  55. Lucy Suchman. Human-machine reconfigurations: Plans and situated actions. Cambridge university press, 2007.
  56. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
  57. Yi Chern Tan and L Elisa Celis. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, pages 13209-13220, 2019.
  58. Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958-1970, 2008.
  59. Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018. · arXiv
  60. Paul Weindling. The origins of informed consent: the international scientific commission on medical war crimes, and the nuremberg code. Bulletin of the History of Medicine, pages 37-71, 2001.
  61. Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097, 2019. · arXiv
  62. Elizabeth R Wrape, Alexandra L Dittloff, and Jennifer L Callahan. Gender and musical instrument stereotypes in middle school children: Have trends changed? Update: Applications of Research in Music Education, 34(3):40-47, 2016.
  63. Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, and Tong Zhang. Tencent ml-images: A large-scale multi-label image database for visual representation learning. IEEE Access, 7:172683-172693, 2019.
  64. Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 547-558, 2020.
  65. Jason Zervoudakes and Judith M Tanur. Gender and musical instruments: Winds of change? Journal of Research in Music Education, pages 58-67, 1994.
  66. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2017.
  67. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.

Cited by

Sorted by citations

Authors
Prabhu, V. U., & Birhane, A.
Affiliations
  • University College Dublin (IE)
  • Lero (IE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.16923
arXiv
2006.16923
OpenAlex
W3040589425
Cited by
91
References
9
Citations by year
2026: 1 · 2025: 25 · 2024: 10 · 2023: 13 · 2022: 9 · 2021: 23 · 2020: 10
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Science Foundation Ireland (13/RC/2094)
  • European Regional Development Fund (13/RC/2094)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
Keywords
large-scale image databases, revenge porn, consent, justice, NSFW content, data curation, research ethics committees, image classification accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review