CoChat
Sign Up
Cited by 28Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionExplainable Artificial Intelligence (XAI)

The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

Open access
Birhane, Abeba, Dehdashtian, Sepehr, Prabhu, Vinay, Boddeti, Vishnu·2024·The 2024 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3630106.3658968
Open this paper in CoChat — free account

Abstract

‘Scale the model, scale the data, scale the GPU farms’ is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts on model performance remain under-explored. This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the downstream impact of dataset scaling on 14 visio-linguistic models (VLMs) trained on the LAION400-M and LAION-2B datasets by measuring racial and gender bias using the Chicago Face Dataset (CFD) as the probe. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images as offensive non-human classes such as chimpanzee, gorilla, and orangutan decreased, but misclassifying the same images as human offensive classes such as criminal increased. Furthermore, of the 14 Vision Transformer-based VLMs we evaluated, the probability of predicting an image of a Black man and a Latino man as criminal increases by 65% and 69%, respectively, when the dataset is scaled from 400M to 2B samples for the larger ViT-L models. Conversely, for the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples. We ground the model audit results in a qualitative and historical analysis, reflect on our findings and their implications for dataset curation practice, and close with a summary of mitigation mechanisms and ways forward. All the meta-datasets curated in this endeavor and the code used are shared at: https://github.com/SepehrDehdashtian/the-dark-side-of-dataset-scaling. Content warning: This article contains racially dehumanising and offensive descriptions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3630106.3658968

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current data collection processes prioritize scale over inclusion, relying on web-scraping and synthetic pipelines that reflect developer biases and the uneven availability of online content.

    “Current data collection processes prioritize scale over inclusion, relying on web-scraping (e.g., [40, 97]) and synthetic pipelines that reflect developer biases and the uneven availability of online content [13, 14, 57, 96].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [13]

    The Introduction (p. 3) describes dataset creation through internet scraping and Common Crawl, while the Discussion (p. 13) says scale is prized for training and describes aggressive crawl-and-scrape strategies. These passages support the scale and web-scraping portions, but they do not establish the claim about synthetic pipelines, developer biases, or uneven availability of online content. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [3][3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
  2. [6][6] April H Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions of words on the internet, people= men. Science Advances 8, 13 (2022), eabm2463. -
  3. [10][10] Marquis Bey. 2016. 'Bring Out Your Dead' Understanding the Historical Persistence of the Criminalization of Black Bodies. Cultural Studies? Critical Methodologies 16, 3 (2016), 271-277. -
  4. [11][11] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2022. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. arXiv preprint arXiv:2211.03759 (2022). - · arXiv
  5. [15][15] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency . PMLR, 77-91. -
  6. [16][16] Carole Cadwalladr and Emma Graham-Harrison. 2018. Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach. The guardian 17, 1 (2018), 22. -
  7. [17][17] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183-186. -
  8. [19][19] Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788 (2023). - · arXiv
  9. [25][25] Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops . 52-59. -
  10. [28][28] Constance L Fry, Thomas C Naugle Jr, Shelley A Cole, Jonathan Gelfond, Geetha Chittoor, Angeline F Mariani, Martin W Goros, Barrett G Haik, and Venkata Saroja Voruganti. 2017. The Latino eyelid: anthropometric analysis of a spectrum of findings. Ophthalmic plastic and reconstructive surgery 33, 6 (2017), 440. -
  11. [36][36] Dominik Hintersdorf, Lukas Struppek, Manuel Brack, Felix Friedrich, Patrick Schramowski, and Kristian Kersting. 2022. Does CLIP Know My Face? arXiv preprint arXiv:2209.07341 (2022). - · arXiv
  12. [40][40] Pratyusha Kalluri et al. 2020. Don't ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583, 7815 (2020), 169-169. -
  13. [42][42] Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2021. Transformer-based direct speech-to-speech translation with transcoder. In 2021 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 958-965. -
  14. [44][44] Kimmo Kärkkäinen and Jungseock Joo. 2019. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913 (2019). - · arXiv
  15. [45][45] Jana Kasperkevic. 2015. Google says sorry for racist auto-tag in photo app. The Guardian 1 (2015), 2015. -
  16. [48][48] Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. FABLES: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261 (2024). - · arXiv
  17. [49][49] Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. arXiv preprint arXiv:2011.00747 (2020). - · arXiv
  18. [51][51] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv preprint arXiv:2201.12086 (2022). - · arXiv
  19. [52][52] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). - · arXiv
  20. [53][53] Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, and Jean Oh. 2024. SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation. arXiv preprint arXiv:2401.08053 (2024). - · arXiv
  21. [54][54] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable Bias: Analyzing Societal Representations in Diffusion Models. arXiv preprint arXiv:2303.11408 (2023). - · arXiv
  22. [55][55] Alexandra Sasha Luccioni, Yacine Jernite, and Emma Strubell. 2023. Power hungry processing: Watts driving the cost of ai deployment? arXiv preprint arXiv:2311.16863 (2023). - · arXiv
  23. [56][56] Debbie S Ma, Joshua Correll, and Bernd Wittenbrink. 2015. The Chicago face database: A free stimulus set of faces and norming data. Behavior research methods 47 (2015), 1122-1135. -
  24. [57][57] Abhishek Mandal, Susan Leavy, and Suzanne Little. 2023. Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models. arXiv preprint arXiv:2304.13855 (2023). - · arXiv
  25. [58][58] Harvey Mannering. 2023. Analysing Gender Bias in Text-to-Image Models using Object Detection. arXiv preprint arXiv:2307.08025 (2023). - · arXiv
  26. [60][60] Dan McQuillan. 2022. Resisting AI: an anti-fascist approach to artificial intelligence . Policy Press. -
  27. [61][61] Peter Mende-Siedlecki, Jennie Qu-Lee, Jingrun Lin, Alexis Drain, and Azaadeh Goharzad. 2020. The Delaware pain database: A set of painful expressions and corresponding norming data. Pain reports 5, 6 (2020). -
  28. [63][63] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. 2023. Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory. In The Twelfth International Conference on Learning Representations . -
  29. [65][65] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035 (2023). - · arXiv
  30. [71][71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021). - · arXiv
  31. [74][74] Angela Saini. 2019. Superior: the return of race science . Beacon Press. -
  32. [79][79] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2021. CLIPort: What and Where Pathways for Robotic Manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL) . -
  33. [82][82] Rory W Spanton and Olivia Guest. 2022. Measuring Trustworthiness or Automating Physiognomy? A Comment on Safra, Chevallier, Gr \ ezes, and Baumard (2020). arXiv preprint arXiv:2202.08674 (2022). - · arXiv
  34. [84][84] Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. 2024. Is Cosine-Similarity of Embeddings Really About Similarity? arXiv preprint arXiv:2403.05440 (2024). - · arXiv
  35. [86][86] Nina Strohminger, Kurt Gray, Vladimir Chituc, Joseph Heffner, Chelsea Schein, and Titus Brooks Heagins. 2016. The MR2: A multi-racial, mega-resolution database of facial stimuli. Behavior research methods 48 (2016), 1197-1204. -
  36. [89][89] Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. 2022. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. arXiv preprint arXiv:2201.07284 (2022). - · arXiv
  37. [90][90] Emiel Van Miltenburg. 2016. Stereotyping and bias in the flickr30k dataset. arXiv preprint arXiv:1605.06083 (2016). - · arXiv
  38. [91][91] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). -
  39. [93][93] Haitao Wang, Jiale Zheng, Ivan E Carvajal-Roca, Linghui Chen, and Mengqiu Bai. 2023. Financial Fraud Detection Based on Deep Learning: Towards Large-Scale Pre-training Transformer Models. In China Conference on Knowledge Graph and Semantic Computing . Springer, 163-177. -
  40. [94][94] Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019. Predictive inequity in object detection. arXiv preprint arXiv:1902.11097 (2019). - · arXiv
  41. [96][96] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019). -
  42. [97][97] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it?. In The Eleventh International Conference on Learning Representations . -

Cited by

Sorted by citations

Related

Authors
Birhane, A., Dehdashtian, S., Prabhu, V., & Boddeti, V.
Affiliations
  • Trinity College Dublin (IE)
  • Michigan State University (US)
  • HAL51 Inc, USA
ORCID
Venue
The 2024 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2024-06-03
Pages
1229-1244
DOI
10.1145/3630106.3658968
OpenAlex
W4396822621
Cited by
28
References
47
Citations by year
2026: 12 · 2025: 15 · 2024: 1
FWCI
12.78
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Science Foundation Ireland via the ADAPT Centre of Digital Content Technology funded under the European Regional Development Fund (ERDF) (13/RC/2106_P2)
  • National Science Foundation (2147116)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
dataset scaling, racial bias, gender bias, multimodal vision-language models, Common Crawl, CLIP, data curation, ViT

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review