The DOI matches its DataCite record; 93 of the 93 references it cites were found in a registry or an index.
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Open accessAbstract
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For instance, the Hateful Memes dataset has shorter text sequences (i.e., 128) compared to other datasets, in which prompts may become more influential to the final performance if the prompt length is too long. This may cause ambiguity for the model to learn task-specific features, e.g., input-level prompting performs slightly worse on Hateful Memes, especially for the text-missing case in Table 1”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Next, we show that input-level prompting further improves the performance in most settings, except for one case on the Hateful Memes dataset, in which this sensitivity analysis will be discussed in the following section.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Hateful Memes dataset is constructed to fail models that rely on a single modality by adding challenging samples ('benign confounders') to the dataset.”
From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [17]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–107 of 107
Multimodal Religiously Hateful Social Media Memes Classification Based on Textual and Image Data
Hamza, A., Javed, A. R., Iqbal, F., Yasin, A., Srivastava, G., Połap, D., Gadekallu, T. R., & Jalil, Z. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3623396 · Cited by 24
Caption Enriched Samples for Improving Hateful Memes Detection
Blaier, E., Malkiel, I., & Wolf, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.738 · Cited by 22
MemeFier: Dual-stage Modality Fusion for Image Meme Classification
Koutlis, C., Schinas, M., & Papadopoulos, S. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592254 · Cited by 26
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02508 · Cited by 17
Multimodal Fusion Refiner Networks
Sankaran, S., Yang, D., & Lim, S.-N. (2021) · arXiv · doi:10.48550/arxiv.2104.03435 · Cited by 10
Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning
Zhao, Y., Xi, W., Fu, X., & Zhao, J. (2025) · Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2025/274 · Cited by 6
Dai, R., Jian, A., Zhang, R., Jia, Y., Mo, L., Zhan, S., & He, T. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116272
Metadata
- Authors
- Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D.
- Affiliations
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 33
- Pages
- 2611-2624
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.04790
- OpenAlex
- W3023989664
- Cited by
- 352
- References
- 93
- Citations by year
- 2026: 3 · 2025: 47 · 2024: 78 · 2023: 82 · 2022: 73 · 2021: 53 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, multimodal classification, benchmark dataset, human performance comparison, binary classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
