Multimodal datasets: misogyny, pornography, and malignant stereotypes
Open accessAbstract
We have now entered the era of trillion parameter machine learning models trained on billion-sized datasets scraped from the internet. The rise of these gargantuan datasets has given rise to formidable bodies of critical work that has called for caution while generating these large datasets. These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a source for training large language models, and the entrenched biases in large-scale visio-linguistic models (such as OpenAI's CLIP model) trained on opaque datasets (WebImageText). In the backdrop of these specific calls of caution, we examine the recently released LAION-400M dataset, which is a CLIP-filtered dataset of Image-Alt-text pairs parsed from the Common-Crawl dataset. We found that the dataset contains, troublesome and explicit images and text pairs of rape, pornography, malign stereotypes, racist and ethnic slurs, and other extremely problematic content. We outline numerous implications, concerns and downstream harms regarding the current state of large scale datasets while raising open questions for various stakeholders including the AI community, regulators, policy makers and data subjects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data Providers might have reason to propose their own license, especially to support values that are misrepresented in legal frameworks relevant to them.
“Data Providers might have reason to propose their own license, especially to support values that are misrepresented in legal frameworks relevant to them 19 .”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [19]
Section 5.4 (p. 12) says dataset creators are responsible for their licensing scheme and suggests a more restrictive licence for LAION-400M to limit harm. It does not say that providers should propose their own licences because relevant legal frameworks misrepresent their values; its discussion of law concerns possible copyright and GDPR violations. (LLM, conf=0.83)
Experimental — accuracy not yet measuredMultimodal generative AI models such as CLIP and the CLIP-filtered LAION dataset have harmful biases embedded in them.
“Recent studies expose the harmful biases embedded in multimodal generative AI models such as CLIP (contrastive language-image pretraining; Wolfe et al. 2022) and the CLIP-filtered LAION dataset (Birhane et al. 2021), which are core components of generative AI models (e.g., Dall-E 2 or Stable Diffusion).”
From Generative AI (2024) · cites this paper as Birhane et al. 2021
Such audits have revealed the presence of misogyny, pornography and harmful stereotypes.
“Such audits have revealed the presence of misogyny, pornography and harmful stereotypes [5, 6]”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [6]
There has been an increased focus on analyzing other dimensions of data quality and stewardship, with several initiatives aiming to document existing and newly developed resources.
“There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [20]
Even works that include significant curation efforts for the sake of improving models focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes.
“Even works that do include significant curation efforts for the sake of improving models [57, 113] focus on definitions of quality that prioritize technical performance over the agency of data and algorithm subjects, which can result in widespread data that proliferates misogyny, pornography without consent, and malignant stereotypes [19].”
From Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,632 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Algorithmic injustice: A relational ethics approach
Birhane, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100205 · Cited by 707 · cited 2× · see citing works
“It fails to account for the rights, welfare, and interests of vulnerable individuals and communities, many of whom are likely to suffer worst from the downstream impacts of this dataset and the models trained on it [69].”
BackgroundSelf-citationRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919012117 · Cited by 685 · see citing works
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548 · cited 1× · see citing works
“…of large scale computer vision datasets contains various broad problems including curation biases, inclusion of problematic content in the images, the questionable approaches of associating these images with offensive and non-imageable labels, as well as the gradual erosion of privacy [4, 5].”
BackgroundCollecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
You, Y., Zhang, Z., Hsieh, C., Demmel, J. W., & Keutzer, K. (2018) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3225058.3225069 · Cited by 393 · see citing works
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317 · see citing works
Self-citationThe Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2021) · arXiv · doi:10.48550/arxiv.2106.15590 · Cited by 317 · see citing works
Self-citationExcavating AI: the politics of images in machine learning training sets
Crawford, K., & Paglen, T. (2021) · AI & Society · doi:10.1007/s00146-021-01162-8 · Cited by 278 · see citing works
Large image datasets: A pyrrhic win for computer vision?
Birhane, A., & Prabhu, V. U. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00158 · Cited by 257 · see citing works
Self-citationMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 240 · see citing works
On the genealogy of machine learning datasets: A critical history of ImageNet
Denton, E., Hanna, A., Amironesei, R., Smart, A., & Nicole, H. (2021) · Big Data & Society · doi:10.1177/20539517211035955 · Cited by 221 · see citing works
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463257 · Cited by 214 · see citing works
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182 · see citing works
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00447 · Cited by 179 · see citing works
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.12028 · Cited by 179 · see citing works
Why AI is harder than we think
Mitchell, M. (2021) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3449639.3465421 · Cited by 170 · see citing works
WebInSight: making web images accessible
Bigham, J. P., Kaminsky, R. S., Ladner, R. E., Danielsson, O. M., & Hempton, G. L. (2006) · Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1168987.1169018 · Cited by 161 · cited 1× · see citing works
“These works have demonstrated how even highly reputable high-traffic websites have poor alt-text coverage of about 50% [28].”
BackgroundDetoxifying Language Models Risks Marginalizing Minority VoicesPreprint
Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., & Klein, D. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.06390 · Cited by 155 · see citing works
Bennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., & To, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445498 · Cited by 128 · see citing works
Morris, M. R. (2020) · Communications of the ACM · doi:10.1145/3356727 · Cited by 114 · see citing works
Challenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112 · see citing works
Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
Guinness, D., Cutrell, E., & Morris, M. R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174092 · Cited by 108 · see citing works
Gleason, C., Carrington, P., Cassidy, C., Morris, M., Kitani, K. M., & Bigham, J. (2019) · The Web Conference · doi:10.1145/3308558.3313605 · Cited by 108 · see citing works
A Study of Face Obfuscation in ImageNetPreprint
Yang, K., Yau, J., Fei-Fei, L., Deng, J., & Russakovsky, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06191 · Cited by 103 · see citing works
Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., & Zhang, T. (2019) · IEEE Access · doi:10.1109/access.2019.2956775 · Cited by 79 · see citing works
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Wang, A., Narayanan, A., & Russakovsky, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_43 · Cited by 72 · see citing works
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A., & Viviano, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.24 · Cited by 69 · see citing works
CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language ModelPreprint
Xu, L. J., Zhang, X., & Dong, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01355 · Cited by 54 · see citing works
Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge
Dognin, P., Melnyk, I., Mroueh, Y., Padhi, I., Rigotti, M., Ross, J., Schiff, Y., Young, R. A., & Belgodere, B. M. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13113 · Cited by 54 · see citing works
The art of ALT: toward a more accessible Web
Slatin, J. M. (2001) · Computers & composition/Computers and composition · doi:10.1016/s8755-4615(00)00049-9 · Cited by 44 · see citing works
Computer Vision and Conflicting Values: Describing People with Automated Alt Text
Hanley, M., Barocas, S., Levy, K., Azenkot, S., & Nissenbaum, H. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462620 · Cited by 38 · see citing works
Against Scale: Provocations and Resistances to Scale ThinkingPreprint
Hanna, A., & Park, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08850 · Cited by 34 · see citing works
Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31 · see citing works
Diaper, D., & Worman, L. (2004) · Springer eBooks · doi:10.1007/978-1-4471-3754-2_22 · Cited by 26 · see citing works
(2021) · arXiv (Cornell University) · Cited by 21 · see citing works
How Do We Talk about Other People? Group (Un)Fairness in Natural Language Image Descriptions
Otterbacher, J., Barlas, P., Kleanthous, S., & Kyriakou, K. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5267 · Cited by 20 · see citing works
Some features of alt texts associated with images in Web pages
Craven, T. C. (2006) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 17 · see citing works
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Luccioni, A. S., & Viviano, J. D. (2021) · arXiv · doi:10.48550/arxiv.2105.02732 · Cited by 17 · see citing works
ALT Text and Basic Accessibility
McEwan, T., & Weerts, B. (2007) · Electronic workshops in computing · doi:10.14236/ewic/hci2007.64 · Cited by 15 · see citing works
The Inclusive Images Competition
Atwood, J. R., Halpern, Y., Baljekar, P. N., Breck, E. J., Sculley, D., Ostyakov, P., Nikolenko, S., Ivanov, I., Solovyev, R. A., Wang, W., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_6 · Cited by 12 · see citing works
Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 ChallengePreprint
Dognin, P., Melnyk, I., Mroueh, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11696 · Cited by 8 · see citing works
How Will We Manage the Alt Text
Bramlett, F. (2012) · Cited by 1 · see citing works
Reading Isn't Believing: Adversarial Attacks On Multi-Modal NeuronsPreprint
Noever, D. A., & Noever, S. E. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10480 · Cited by 1 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462
Flagged first, then by citations
Research works printed but not linked: 10
- [4][4] M. K. Scheuerman, E. Denton, and A. Hanna, 'Do datasets have politics? disciplinary values in computer vision dataset development,' arXiv preprint arXiv:2108.04308 , 2021. - · arXiv
- [10][10] V. U. Prabhu, 'The phantom of the corpora: JFT-300M,' in Proceedings of the 2021 Beyond Fairness CVPR Workshop , 2021, available at shorturl.at/djnR0. -
- [12][12] K. Yang, K. Qinami, L. Fei-Fei, J. Deng, and O. Russakovsky, 'Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy,' in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 2020, pp. 547-558. -
- [24][24] H. Petrie, C. Harrison, and S. Dev, 'Describing images on the web: a survey of current practice and prospects for the future,' Proceedings of Human Computer Interaction International (HCII) , vol. 71, no. 2, 2005. -
- [30][30] M. Hanley, S. Barocas, K. Levy, S. Azenkot, and H. Nissenbaum, 'Computer vision and conflicting values: Describing people with automated alt text,' arXiv preprint arXiv:2105.12754 , 2021. - · arXiv
- [37][37] S. Matic, C. Iordanou, G. Smaragdakis, and N. Laoutaris, 'Identifying sensitive urls at webscale,' in Proceedings of the ACM Internet Measurement Conference , 2020, pp. 619-633. -
- [48][48] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, 'Yfcc100m: The new data in multimedia research,' Communications of the ACM , vol. 59, no. 2, pp. 64-73, 2016. -
- [68][68] M. Steiger, T. J. Bharucha, S. Venkatagiri, M. J. Riedl, and M. Lease, 'The psychological well-being of content moderators,' in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI , vol. 21, 2021. -
- [73][73] D. Erhan, Y. Bengio, A. Courville, and P. Vincent, 'Visualizing higher-layer features of a deep network,' University of Montreal , vol. 1341, no. 3, p. 1, 2009. -
- [74][74] R. Q. Quiroga, L. Reddy, G. Kreiman, C. Koch, and I. Fried, 'Invariant visual representation by single neurons in the human brain,' Nature , vol. 435, no. 7045, pp. 1102-1107, 2005.
Cited by
Sorted by citations
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 1,409 · see citing works
“(2022)) and the CLIP-filtered LAION dataset (Birhane et al., 2021), which are core components of generative AI models (e.”
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationSafe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219 · see citing works
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138 · see citing works
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2023) · arXiv · doi:10.48550/arxiv.2302.10149 · Cited by 138 · see citing works
“[7] note that LAION-400M has “troublesome and explicit images and text pairs of rape, pornography, malign stereotypes, racist and ethnic slurs, and other extremely problematic content”.”
BackgroundLiesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
Rethinking open source generative AI: open washing and the EU AI Act
Liesenfeld, A., & Dingemanse, M. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659005 · Cited by 67 · see citing works
“Such audits have revealed the presence of misogyny, pornography and harmful stereotypes [5, 6] and may have made makers of such models more aware of the harmful uses to which their systems can be put.”
BackgroundHoskins, A. (2024) · Memory, Mind & Media · doi:10.1017/mem.2024.16 · Cited by 64 · see citing works
“Datasets of the required scale for LLMs to function efficiently can reproduce (and further hide) the processes of the reproduction of explicit images, violence, rape, pornog-raphy, sexism, racist, and ethnic slurs (Birhane et al. 2021), as well as correcting towards heteronormativity (Zawacki 2023).”
Introduction to large language models (LLMs) for dementia care and research
Treder, M. S., Lee, S., & Tsvetanov, K. A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1385303 · Cited by 63 · see citing works
Epistemic Injustice in Generative AI
Kay, J., Kasirzadeh, A., & Mohamed, S. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31671 · Cited by 76 · see citing works
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 45 · see citing works
“There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
BackgroundWolfe, R., Yang, Y., Howe, B., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594072 · Cited by 42 · see citing works
Hartmann, J., Schwenzow, J., & Witte, M. (2023) · arXiv · doi:10.48550/arxiv.2301.01768 · Cited by 37 · see citing works
“Proceedings of the National Academy of Sciences, 118(8): e2017548118, 2021.”
BackgroundBenjamin, J. J., Lindley, J., Edwards, E., Rubegni, E., Korjakow, T., Grist, D., & Sharkey, R. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660677 · Cited by 32 · see citing works
DeAR: Debiasing Vision-Language Models with Additive Residuals
Seth, A., Hemani, M., & Agarwal, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00659 · Cited by 34 · see citing works
Deepfakes on Demand: The rise of accessible non-consensual deepfake image generators
Hawkins, W., Mittelstadt, B., & Russell, C. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732107 · Cited by 20 · see citing works
“T2I models are trained on vast amounts of training data usually derived from the internet, which can lead to models perpetuating or amplifying stereotypes or biases found in the data [4, 5, 35, 39].”
‘We can see a savage’: a case study of the colonial gaze in generative AI algorithms
Alenichev, A., Shaffer, J. D., Kingori, P., Grietens, K. P., Muldoon, J., & Rocher, L. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02685-0 · Cited by 3 · see citing works
- Authors
- Birhane, A., Prabhu, V. U., & Kahembwe, E.
- Affiliations
- Carnegie Mellon University (US)
- University of Edinburgh (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.01963
- OpenAlex
- W3203737321
- Cited by
- 160
- References
- 59
- Citations by year
- 2026: 6 · 2025: 27 · 2024: 50 · 2023: 57 · 2022: 18 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
- Keywords
- multimodal dataset, data curation, misogyny, pornography, alt text, Common Crawl, large language model training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
