CoChat
Sign Up
Cited by 34Hate Speech and Cyberbullying DetectionMisinformation and Its ImpactsSentiment Analysis and Opinion Mining

Into the LAION’s Den: Investigating Hate in Multimodal Datasets

Open access
Birhane, Abeba, Prabhu, Vinay, Han, Sanghyun, Boddeti, Vishnu, Luccioni, Sasha·2023·Advances in Neural Information Processing Systems 36
DOI 10.52202/075280-0930
Open this paper in CoChat — free accountOpen PDF

Abstract

'Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning to scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision-language datasets such as LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common Crawl, which is known to have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. Our results show that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively using a metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, we found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with a reflection and a discussion of the significance of our results for dataset curation and usage in the AI community. Code and the meta-data assets curated in this paper are publicly available at https://github.com/vinayprabhu/hate_scaling. Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/075280-0930

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many actors in the field opt to err on the side of caution and only disclose elements of their system as required.

    “Naturally, many actors in the field opt to err on the side of caution and only disclose elements of their system as required [4].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [4]

    The Introduction (page not provided) says prominent model papers systematically conceal critical training-data details, supporting the claim that some actors disclose limited information. It does not say they do so out of caution or disclose only what is required. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Releasing datasets containing CSAM material prior to assessment in the name of openness would pose a clear risk.

    “For instance, in the case of dealing with datasets that contain CSAM material, the release of data prior to assessment in the name of openness would pose a clear risk [4, 31].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [4]

    Section 2, “Scaling Datasets: An Overview” (page unavailable), says web-sourced data raises concerns and that rigorous audits are imperative for healthy open-sourcing practices. That supports the general need for assessment, but our copy does not identify CSAM or state that releasing a dataset containing it before assessment poses a clear risk. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • For Llama, there is no documentation of training datasets whatsoever.

    “This means that for Llama, there is no documentation of training datasets whatsoever -a prime example of a strategy described by Birhane et al. as a tactical template of '(non)declaring the training dataset information' [4].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [4]

  • Auditing of Stable Diffusion's underlying datasets has revealed legal challenges and deeply problematic content.

    “This has enabled auditing of the underlying datasets, which has revealed legal challenges [49] and deeply problematic content [4, 5].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [4]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 62

  1. ABDALLA, M., AND ABDALLA, M. The grey hoodie project: Big tobacco, big tech, and the threat on academic integrity. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (2021), pp. 287-297.
  2. ABID, A., FAROOQI, M., AND ZOU, J. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (2021), pp. 298-306.
  3. BARLAS, P., KYRIAKOU, K., GUEST, O., KLEANTHOUS, S., AND OTTERBACHER, J. To" see" is to stereotype: Image tagging algorithms, gender recognition, and the accuracy-fairness trade-off. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1-31.
  4. BENDER, E. M., GEBRU, T., MCMILLAN-MAJOR, A., AND SHMITCHELL, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610-623.
  5. BIANCHI, F., KALLURI, P., DURMUS, E., LADHAK, F., CHENG, M., NOZZA, D., HASHIMOTO, T., JURAFSKY, D., ZOU, J., AND CALISKAN, A. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. arXiv preprint arXiv:2211.03759 (2022). · arXiv
  6. BIRHANE, A., KALLURI, P., CARD, D., AGNEW, W., DOTAN, R., AND BAO, M. The values encoded in machine learning research. arXiv preprint arXiv:2106.15590 (2021). · arXiv
  7. BIRHANE, A., PRABHU, V. U., AND KAHEMBWE, E. Multimodal datasets: misogyny, pornog- raphy, and malignant stereotypes. arXiv preprint arXiv:2110.01963 (2021). · arXiv
  8. CARLINI, N., HAYES, J., NASR, M., JAGIELSKI, M., SEHWAG, V., TRAMER, F., BALLE, B., IPPOLITO, D., AND WALLACE, E. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 (2023). · arXiv
  9. CITRON, D. K. Hate crimes in cyberspace. Harvard University Press, 2014.
  10. COSTANZA-CHOCK, S., RAJI, I. D., AND BUOLAMWINI, J. Who audits the auditors? recom- mendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (2022), pp. 1571-1583.
  11. DAVIDSON, T., BHATTACHARYA, D., AND WEBER, I. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.12516 (2019). · arXiv
  12. DAVIDSON, T., WARMSLEY, D., MACY, M., AND WEBER, I. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media (2017), vol. 11, pp. 512-515.
  13. DENG, J., DONG, W., SOCHER, R., LI, L.-J., LI, K., AND FEI-FEI, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (2009), Ieee, pp. 248-255.
  14. DENTON, E., HANNA, A., AMIRONESEI, R., SMART, A., AND NICOLE, H. On the genealogy of machine learning datasets: A critical history of imagenet. Big Data & Society 8, 2 (2021), 20539517211035955.
  15. DODGE, J., SAP, M., MARASOVIC, A., AGNEW, W., ILHARCO, G., GROENEVELD, D., AND GARDNER, M. Documenting the english colossal clean crawled corpus. arXiv preprint arXiv:2104.08758 (2021). · arXiv
  16. GAO, L., BIDERMAN, S., BLACK, S., GOLDING, L., HOPPE, T., FOSTER, C., PHANG, J., HE, H., THITE, A., NABESHIMA, N., ET AL. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020). · arXiv
  17. GARCIA, N., HIROTA, Y., WU, Y., AND NAKASHIMA, Y. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 6957-6966.
  18. GARG, T., MASUD, S., SURESH, T., AND CHAKRABORTY, T. Handling bias in toxic speech detection: A survey. ACM Computing Surveys (2022).
  19. GEBRU, T., MORGENSTERN, J., VECCHIONE, B., VAUGHAN, J. W., WALLACH, H., III, H. D., AND CRAWFORD, K. Datasheets for datasets. Communications of the ACM 64, 12 (2021), 86-92.
  20. GEHMAN, S., GURURANGAN, S., SAP, M., CHOI, Y., AND SMITH, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 (2020). · arXiv
  21. GICHOYA, J. W., BANERJEE, I., BHIMIREDDY, A. R., BURNS, J. L., CELI, L. A., CHEN, L.-C., CORREA, R., DULLERUD, N., GHASSEMI, M., HUANG, S.-C., ET AL. Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health 4, 6 (2022), e406-e414.
  22. HANNA, A., AND PARK, T. M. Against scale: Provocations and resistances to scale thinking. arXiv preprint arXiv:2010.08850 (2020). · arXiv
  23. HESTNESS, J., NARANG, S., ARDALANI, N., DIAMOS, G., JUN, H., KIANINEJAD, H., PATWARY, M., ALI, M., YANG, Y., AND ZHOU, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 (2017). · arXiv
  24. HOFFMANN, J., BORGEAUD, S., MENSCH, A., BUCHATSKAYA, E., CAI, T., RUTHERFORD, E., CASAS, D. D. L., HENDRICKS, L. A., WELBL, J., CLARK, A., ET AL. Training compute- optimal large language models. arXiv preprint arXiv:2203.15556 (2022). · arXiv
  25. JAHAN, M. S., AND OUSSALAH, M. A systematic review of hate speech automatic detection using natural language processing. arXiv preprint arXiv:2106.00742 (2021). · arXiv
  26. JIA, C., YANG, Y., XIA, Y., CHEN, Y.-T., PAREKH, Z., PHAM, H., LE, Q. V., SUNG, Y., LI, Z., AND DUERIG, T. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 (2021). · arXiv
  27. KAPLAN, J., MCCANDLISH, S., HENIGHAN, T., BROWN, T. B., CHESS, B., CHILD, R., GRAY, S., RADFORD, A., WU, J., AND AMODEI, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). · arXiv
  28. KOCH, B., DENTON, E., HANNA, A., AND FOSTER, J. G. Reduced, reused and recycled: The life of a dataset in machine learning research. arXiv preprint arXiv:2112.01716 (2021). · arXiv
  29. LAURENÇON, H., SAULNIER, L., WANG, T., AKIKI, C., VILLANOVA DEL MORAL, A., LE SCAO, T., VON WERRA, L., MOU, C., GONZÁLEZ PONFERRADA, E., NGUYEN, H., ET AL. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems 35 (2022), 31809-31826.
  30. LEE, R. K.-W., CAO, R., FAN, Z., JIANG, J., AND CHONG, W.-H. Disentangling hate in online memes. In Proceedings of the 29th ACM International Conference on Multimedia (2021), pp. 5138-5147.
  31. LUCCIONI, A. S., AKIKI, C., MITCHELL, M., AND JERNITE, Y. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408 (2023). · arXiv
  32. LUCCIONI, A. S., AND VIVIANO, J. D. What's in the box? an analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:2105.02732 (2021). · arXiv
  33. MACAVANEY, S., YAO, H.-R., YANG, E., RUSSELL, K., GOHARIAN, N., AND FRIEDER, O. Hate speech detection: Challenges and solutions. PloS one 14, 8 (2019), e0221152.
  34. METAXA, D., PARK, J. S., ROBERTSON, R. E., KARAHALIOS, K., WILSON, C., HANCOCK, J., SANDVIG, C., ET AL. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends® in Human-Computer Interaction 14, 4 (2021), 272-344.
  35. PAULLADA, A., RAJI, I. D., BENDER, E. M., DENTON, E., AND HANNA, A. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021), 100336.
  36. PENG, K., MATHUR, A., AND NARAYANAN, A. Mitigating dataset harms requires stewardship: Lessons from 1000 papers. arXiv preprint arXiv:2108.02922 (2021). · arXiv
  37. PÉREZ, J. M., GIUDICI, J. C., AND LUQUE, F. pysentimiento: A python toolkit for sentiment analysis and socialnlp tasks. arXiv preprint arXiv:2106.09462 (2021). · arXiv
  38. PHAM, H., DAI, Z., GHIASI, G., KAWAGUCHI, K., LIU, H., YU, A. W., YU, J., CHEN, Y.-T., LUONG, M.-T., WU, Y., ET AL. Combined scaling for open-vocabulary image classification. arXiv e-prints (2021), arXiv-2111.
  39. POLETTO, F., BASILE, V., SANGUINETTI, M., BOSCO, C., AND PATTI, V. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation 55 (2021), 477-523.
  40. RADFORD, A., KIM, J. W., HALLACY, C., RAMESH, A., GOH, G., AGARWAL, S., SASTRY, G., ASKELL, A., MISHKIN, P., CLARK, J., ET AL. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021). · arXiv
  41. RAFFEL, C., SHAZEER, N., ROBERTS, A., LEE, K., NARANG, S., MATENA, M., ZHOU, Y., LI, W., AND LIU, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485-5551.
  42. RAJI, I. D., AND BUOLAMWINI, J. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 429-435.
  43. RAMESH, A., PAVLOV, M., GOH, G., GRAY, S., VOSS, C., RADFORD, A., CHEN, M., AND SUTSKEVER, I. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 (2021). · arXiv
  44. ROMBACH, R., BLATTMANN, A., LORENZ, D., ESSER, P., AND OMMER, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 10684-10695.
  45. SAHARIA, C., CHAN, W., SAXENA, S., LI, L., WHANG, J., DENTON, E. L., GHASEMIPOUR, K., GONTIJO LOPES, R., KARAGOL AYAN, B., SALIMANS, T., ET AL. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35 (2022), 36479-36494.
  46. SAMBASIVAN, N., KAPANIA, S., HIGHFILL, H., AKRONG, D., PARITOSH, P., AND AROYO, L. M. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021), pp. 1-15.
  47. SCHEUERMAN, M. K., WADE, K., LUSTIG, C., AND BRUBAKER, J. R. How we've taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-computer Interaction 4, CSCW1 (2020), 1-35.
  48. SCHUHMANN, C., BEAUMONT, R., VENCU, R., GORDON, C., WIGHTMAN, R., CHERTI, M., COOMBES, T., KATTA, A., MULLIS, C., WORTSMAN, M., ET AL. Laion-5b: An open large- scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 (2022). · arXiv
  49. SCHUHMANN, C., VENCU, R., BEAUMONT, R., KACZMARCZYK, R., MULLIS, C., KATTA, A., COOMBES, T., JITSEV, J., AND KOMATSUZAKI, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021). · arXiv
  50. SOLAIMAN, I., TALAT, Z., AGNEW, W., AHMAD, L., BAKER, D., BLODGETT, S. L., DAUMÉ III, H., DODGE, J., EVANS, E., HOOKER, S., ET AL. Evaluating the social impact of generative ai systems in systems and society. arXiv preprint arXiv:2306.05949 (2023). · arXiv
  51. SOMEPALLI, G., SINGLA, V., GOLDBLUM, M., GEIPING, J., AND GOLDSTEIN, T. Diffu- sion art or digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.03860 (2022). · arXiv
  52. VAN MILTENBURG, E. Stereotyping and bias in the flickr30k dataset. arXiv preprint arXiv:1605.06083 (2016). · arXiv
  53. VECCHIONE, B., LEVY, K., AND BAROCAS, S. Algorithmic auditing and social justice: Lessons from the history of audit studies. In Equity and Access in Algorithms, Mechanisms, and Optimization. 2021, pp. 1-9.
  54. VELIOGLU, R., AND ROSE, J. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge. arXiv preprint arXiv:2012.12975 (2020). · arXiv
  55. VIDGEN, B., THRUSH, T., WASEEM, Z., AND KIELA, D. Learning from the worst: Dynamically generated datasets to improve online hate detection. arXiv preprint arXiv:2012.15761 (2020). · arXiv
  56. VIDGEN, B., THRUSH, T., WASEEM, Z., AND KIELA, D. Learning from the worst: Dynamically generated datasets to improve online hate detection. In ACL (2021).
  57. VILLALOBOS, P. Scaling laws literature review, 2023. Accessed: 2023-4-22.
  58. WANG, K., YANG, J., AND WU, H. A survey of toxic comment classification methods. arXiv preprint arXiv:2112.06412 (2021). · arXiv
  59. WASEEM, Z., DAVIDSON, T., WARMSLEY, D., AND WEBER, I. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899 (2017). · arXiv
  60. WEBSTER, R., RABIN, J., SIMON, L., AND JURIE, F. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.12733 (2023). · arXiv
  61. YU, J., XU, Y., KOH, J. Y., LUONG, T., BAID, G., WANG, Z., VASUDEVAN, V., KU, A., YANG, Y., AYAN, B. K., ET AL. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 (2022). · arXiv
  62. YUAN, L., CHEN, D., CHEN, Y.-L., CODELLA, N., DAI, X., GAO, J., HU, H., HUANG, X., LI, B., LI, C., ET AL. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 (2021). · arXiv

Cited by

Sorted by citations

Related

Authors
Birhane, A., Prabhu, V., Han, S., Boddeti, V., & Luccioni, S.
ORCID
Venue
Advances in Neural Information Processing Systems 36
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2023
Pages
21268-21284
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/075280-0930
OpenAlex
W7133191268
Cited by
33
References
0
Citations by year
2026: 12 · 2025: 10 · 2024: 11
FWCI
5.85
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2147116)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
hate speech, multimodal dataset, image-text dataset, data bias, harmful content

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review