The DOI matches its DataCite record; 93 of the 93 references it cites were found in a registry or an index.
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Open accessAbstract
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For instance, the Hateful Memes dataset has shorter text sequences (i.e., 128) compared to other datasets, in which prompts may become more influential to the final performance if the prompt length is too long. This may cause ambiguity for the model to learn task-specific features, e.g., input-level prompting performs slightly worse on Hateful Memes, especially for the text-missing case in Table 1”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Next, we show that input-level prompting further improves the performance in most settings, except for one case on the Hateful Memes dataset, in which this sensitivity analysis will be discussed in the following section.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Hateful Memes dataset is constructed to fail models that rely on a single modality by adding challenging samples ('benign confounders') to the dataset.”
From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [17]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Cited by
Sorted by citations
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“VisDial [20], a visual dialogue benchmark; HatefulMemes [54] a vision and text classification benchmark; NextQA [129] which specially focuses on causality and temporal relation; STAR [128], a multiple-choice question answering task; and RareAct [73], a benchmark measuring compositionality in action recognition.”
BackgroundMethodInfluentialFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678
Self-citationRegulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
“AI Research has made progress in this area recently [64-66].”
BackgroundDirections in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B., & Derczynski, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243300 · Cited by 369
A systematic review of hate speech automatic detection using natural language processing
Jahan, M. S., & Oussalah, M. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126232 · Cited by 345
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 296
Self-citationHateCheck: Functional Tests for Hate Speech Detection Models
Röttger, P., Vidgen, B., Nguyen, D., Waseem, Z., Margetts, H. Z., & Pierrehumbert, J. B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.4 · Cited by 287
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 438
Are Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198
Self-citationSemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!
Sharma, C., Bhageria, D., Scott, W., Pykl, S., Das, A., Chakraborty, T., Pulabaigari, V., & Gambäck, B. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.99 · Cited by 182
TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text
Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., & Hassner, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00869 · Cited by 178
Self-citationLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
“…entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free sentiment classification Accuracy ✓ HatefulMemes [35] OCR-free meme classification ROC AUC ✓ Cross-modal transfer RelativeSize [36] Commonsense reasoning (object size) Accuracy ✓ MemoryColor [37]…”
BackgroundMultimodal Prompting with Missing Modalities for Visual Recognition
Lee, Y.-L., Tsai, Y.-H., Chiu, W.-C., & Lee, C.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01435 · Cited by 156
A Literature Review of Textual Hate Speech Detection Methods and Datasets
Alkomah, F., & Ma, X. (2022) · Information · doi:10.3390/info13060273 · Cited by 167
MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
Pramanick, S., Sharma, S., Dimitrov, D., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.379 · Cited by 191
A Survey on Stance Detection for Mis- and Disinformation Identification
Hardalov, M., Arora, A., Nakov, P., & Augenstein, I. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.94 · Cited by 170
Multimodal Fake News Detection
Segura-Bédmar, I., & Alonso-Bartolome, S. (2022) · Information · doi:10.3390/info13060284 · Cited by 162
Detecting Harmful Memes and Their Targets
Pramanick, S., Dimitrov, D., Mukherjee, R., Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.246 · Cited by 132
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 126
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27999 · Cited by 146
Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach
Kocoń, J., Figas, A., Gruza, M., Puchalska, D., Kajdanowicz, T., & Kazienko, P. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102643 · Cited by 103
A Self-Supervised Descriptor for Image Copy Detection
Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., & Douze, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01413 · Cited by 101
Disentangling Hate in Online Memes
Lee, R. K.-W., Cao, R., Fan, Z., Jiang, J., & Chong, W.-H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475625 · Cited by 98
FedMultimodal: A Benchmark for Multimodal Federated Learning
Feng, T., Bose, D., Zhang, T., Hebbar, R., Ramakrishna, A., Gupta, R., Zhang, M., Avestimehr, S., & Narayanan, S. S. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599825 · Cited by 98
SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images
Dimitrov, D., Ali, B. B., Shaar, S., Alam, F., Silvestri, F., Firooz, H., Nakov, P., & Martino, G. D. S. (2021) · IRIS Research product catalog (Sapienza University of Rome) · doi:10.18653/v1/2021.semeval-1.7 · Cited by 88
Self-citationPrompting for Multimodal Hateful Meme Classification
Cao, R., Lee, R. K.-W., Chong, W.-H., & Jiang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.22 · Cited by 116
Introducing CAD: the Contextual Abuse Dataset
Vidgen, B., Nguyen, D., Margetts, H. Z., Rossini, P., & Tromble, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.182 · Cited by 88
Handling Bias in Toxic Speech Detection: A Survey
Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023) · ACM Computing Surveys · doi:10.1145/3580494 · Cited by 102
Detecting Propaganda Techniques in Memes
Dimitrov, D., Ali, B. B., Shaar, S., Alam, F., Silvestri, F., Firooz, H., Nakov, P., & Martino, G. D. S. (2021) · doi:10.18653/v1/2021.acl-long.516 · Cited by 89
Self-citationMultimodal Hate Speech Detection in Memes Using Contrastive Language-Image Pre-Training
Arya, G., Hasan, M. K., Bagwari, A., Safie, N., Islam, S., Ahmed, F. R. A., De, A., Khan, M. A., & Ghazal, T. M. (2024) · IEEE Access · doi:10.1109/access.2024.3361322 · Cited by 79
A literature survey on multimodal and multilingual automatic hate speech identification
Chhabra, A., & Vishwakarma, D. K. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01051-8 · Cited by 78
Multimodal Learning For Hateful Memes Detection
Zhou, Y., Chen, Z., & Yang, H. (2021) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew53276.2021.9455994 · Cited by 81
A Survey on Multimodal Disinformation DetectionPreprint
Alam, F., Cresci, S., Chakraborty, T., Silvestri, F., Dimitrov, D., Martino, G. D. S., Shaar, S., Firooz, H., & Nakov, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12541 · Cited by 119
Self-citationMulti-modal Misinformation Detection: Approaches, Challenges and Opportunities
Abdali, S., Shaham, S., & Krishnamachari, B. (2024) · ACM Computing Surveys · doi:10.1145/3697349 · Cited by 107
Detecting and Understanding Harmful Memes: A Survey
Sharma, S. K., Alam, F., Akhtar, M. S., Dimitrov, D., Martino, G. D. S., Firooz, H., Halevy, A. Y., Silvestri, F., Nakov, P., & Chakraborty, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/781 · Cited by 109
Self-citationThe state of human-centered NLP technology for fact-checking
Das, A., Liu, H., Kovatchev, V., & Lease, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103219 · Cited by 89
Hate speech detection: A comprehensive review of recent works
Gandhi, A., Ahir, P., Adhvaryu, K., Shah, P., Lohiya, R., Cambria, E., Poria, S., & Hussain, A. (2024) · Expert Systems · doi:10.1111/exsy.13562 · Cited by 66
Multitask Vision-Language Prompt Tuning
Shen, S., Yang, S., Zhang, T., Zhai, B., Gonzalez, J. E., Keutzer, K., & Darrell, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00556 · Cited by 85
Supervised learning: From theory to applications
Tiwari, A. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-824054-0.00026-5 · Cited by 76
Kumar, G. K., & Nandakumar, K. (2022) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2022.nlp4pi-1.20 · Cited by 111
Malware‐SMELL: A zero‐shot learning strategy for detecting zero‐day vulnerabilities
Barros, P. H., Chagas, E. T. C., Oliveira, L. B., Queiroz, F. D. S., & Ramos, H. S. (2022) · Computers & Security · doi:10.1016/j.cose.2022.102785 · Cited by 61
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Chen, Z., Liu, G., Zhang, B., Yang, Q., & Wu, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.552 · Cited by 109
SoK: Content Moderation in Social Media, from Guidelines to Enforcement, and Research to Practice
Singhal, M., Ling, C., Paudel, P., Thota, P., Kumarswamy, N., Stringhini, G., & Nilizadeh, S. (2023) · 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp57164.2023.00056 · Cited by 57
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!
Hessel, J., & Lee, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.62 · Cited by 57
Preserving integrity in online social networks
Halevy, A. Y., Canton-Ferrer, C., Ma, H., Özertem, U., Pantel, P., Saeidi, M., Silvestri, F., & Stoyanov, V. (2022) · Communications of the ACM · doi:10.1145/3462671 · Cited by 71
Velioğlu, R., & Rose, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12975 · Cited by 116
Joint multimodal sentiment analysis based on information relevance
Chen, D., Su, W., Wu, P., & Hua, B. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103193 · Cited by 60
Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
Radenović, F., Dubey, A., Kadian, A., Mihaylov, T., Vandenhende, S., Patel, Y., Wen, Y., Ramanathan, V., & Mahajan, D. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00673 · Cited by 56
Tackling Online Abuse: A Survey of Automated Abuse Detection MethodsPreprint
Mishra, P., Yannakoudakis, H., & Shutova, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06024 · Cited by 74
A Multimodal Framework for the Detection of Hateful MemesPreprint
Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., & Yannakoudakis, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12871 · Cited by 116
In Data We Trust: A Critical Analysis of Hate Speech Detection Datasets
Madukwe, K. J., Gao, X., & Xue, B. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.18 · Cited by 49
Ayetiran, E. F., & Özgöbek, Ö. (2024) · Information Systems · doi:10.1016/j.is.2024.102378 · Cited by 54
Zhu, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08290 · Cited by 91
HateMM: A Multi-Modal Dataset for Hate Video Classification
Das, M., Raj, R., Saha, P., Mathew, B., Gupta, M., & Mukherjee, A. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22209 · Cited by 64
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Xu, Z., Shen, Y., & Huang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.641 · Cited by 102
Detecting Hate Speech in Multi-modal MemesPreprint
Das, A., Wahi, J. S., & Li, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14891 · Cited by 88
Few-Shot Image Classification of Crop Diseases Based on Vision–Language Models
Zhou, Y., Yan, H., Ding, K., Cai, T., & Zhang, Y. (2024) · Sensors · doi:10.3390/s24186109 · Cited by 54
Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful MemesPreprint
Muennighoff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07788 · Cited by 84
Findings of the Shared Task on Troll Meme Classification in Tamil
Suryawanshi, S., & Chakravarthi, B. R. (2021) · Cited by 43
A Multimodal Memes Classification: A Survey and Open Research Issues
Afridi, T. H., Alam, A., Khan, M. N., Khan, J., & Lee, Y.-K. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-66840-2_109 · Cited by 47
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
Hu, Y., Stretcu, O., Lu, C.-T., Viswanathan, K., Hata, K., Luo, E., Krishna, R., & Fuxman, A. D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00916 · Cited by 63
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
Li, L. H., You, H., Wang, Z., Zareian, A., Chang, S., & Chang, K.-W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.420 · Cited by 51
Decoding the Underlying Meaning of Multimodal Hateful Memes
Hee, M. S., Chong, W.-H., & Lee, R. K.-W. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/665 · Cited by 49
Bhandari, A., Shah, S. B., Thapa, S., Naseem, U., & Nasim, M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00193 · Cited by 40
Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10277 · Cited by 140
Identifying Creative Harmful Memes via Prompt based Approach
Ji, J., Ren, W., & Naseem, U. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3587427 · Cited by 36
Adam, M., Albaseer, A. M., Baroudi, U. A., & Abdallah, M. M. (2025) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2025.3554537 · Cited by 36
AOMD: An analogy-aware approach to offensive meme detection on social media
Shang, L., Zhang, Y., Zha, Y., Chen, Y., Youn, C., & Wang, D. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102664 · Cited by 37
Are Deep Neural Networks SMARTer Than Second Graders?
Cherian, A., Peng, K., Lohit, S., Smith, K. A., & Tenenbaum, J. B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01043 · Cited by 44
Gasparini, F., Rizzi, G., Saibene, A., & Fersini, E. (2022) · Data in Brief · doi:10.1016/j.dib.2022.108526 · Cited by 34
An Investigation of Large Language Models for Real-World Hate Speech Detection
Guo, K., Hu, A., Mu, J., Shi, Z., Zhao, Z., Vishwamitra, N., & Hu, H. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00237 · Cited by 68
Cross-modal prototype based multimodal federated learning under severely missing modality
Le, H. Q., Thwal, C. M., Qiao, Y., Tun, Y. L., Nguyen, M. N. H., Huh, E., & Zhang, C. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103219 · Cited by 33
Hossain, E., Sharif, O., Hoque, M. M., Dewan, M. A. A., Siddique, N. H., & Hossain, M. A. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.06.010 · Cited by 32
Emotion-Aware Multimodal Fusion for Meme Emotion Detection
Sharma, S., Ramaneswaran, S., Akhtar, M. S., & Chakraborty, T. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378698 · Cited by 37
KERMIT: Knowledge-EmpoweRed Model In harmful meme deTection
Grasso, B., Gatta, V. L., Moscato, V., & Sperlí, G. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102269 · Cited by 36
ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation
Hu, X., Zhang, K., Xia, L., Chen, A. Y., Luo, J., Sun, Y., Wang, K., Qiao, N., Zeng, X., Sun, M., et al. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00297 · Cited by 41
DISARM: Detecting the Victims Targeted by Harmful Memes
Sharma, S., Akhtar, M. S., Nakov, P., & Chakraborty, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.118 · Cited by 42
A Review of Deep Learning Techniques for Multimodal Fake News and Harmful Languages Detection
Ayetiran, E. F., & Özgöbek, Ö. (2024) · IEEE Access · doi:10.1109/access.2024.3406258 · Cited by 33
Intra- and Inter-Modal Curriculum for Multimodal Learning
Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 28
Detecting Hateful Memes Using a Multimodal Deep EnsemblePreprint
Sandulescu, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13235 · Cited by 59
Hydra: Multi-head low-rank adaptation for parameter efficient fine-tuning
Kim, S., Yang, H.-M., Kim, Y., Hong, Y., & Park, E. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106414 · Cited by 36
Lin, W.-C., Chen, A., Song, X. B., Weiskopf, N. G., Chiang, M. F., & Hribar, M. R. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad213 · Cited by 27
FedMEKT: Distillation-based embedding knowledge transfer for multimodal federated learning
Le, H. Q., Nguyen, M. N. H., Thwal, C. M., Qiao, Y., Zhang, C., & Hong, C. S. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107017 · Cited by 32
ADIMA: Abuse Detection In Multilingual Audio
Gupta, V., Sharon, R., Sawhney, R., & Mukherjee, D. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746718 · Cited by 26
Recognizing misogynous memes: Biased models and tricky archetypes
Rizzi, G., Gasparini, F., Saibene, A., Rosso, P., & Fersini, E. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103474 · Cited by 30
What Makes Multimodal In-Context Learning Work?
Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., & Piwowarski, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00161 · Cited by 35
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
Lam, M. S., Ma, Z., Li, A., Freitas, I., Wang, D., Landay, J. A., & Bernstein, M. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581290 · Cited by 31
Wu, F., Gao, B., Pan, X., Li, L., Ma, Y., Liu, S., & Liu, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103772 · Cited by 25
PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
Prakash, N., Wang, H., Hoang, N. K., Hee, M. S., & Lee, R. K.-W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3613836 · Cited by 28
A transformer based multi task learning approach to multimodal hate speech detection
Kapil, P., & Ekbal, A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100133 · Cited by 26
Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset
Kirk, H. R., Jun, Y., Rauba, P., Wachtel, G., Li, R., Bai, X., Broestl, N., Doff-Sotta, M., Shtedritski, A., & Asano, Y. M. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.4 · Cited by 26
Findings of the WOAH 5 Shared Task on Fine Grained Hateful Memes Detection
Mathias, L., Nie, S., Davani, A. M., Kiela, D., Prabhakaran, V., Vidgen, B., & Talat, Z. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.21 · Cited by 27
Self-citationRacist or Sexist Meme? Classifying Memes beyond Hateful
Zia, H. B., Castro, I., & Gareth Tyson, A. (2021) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2021.woah-1.23 · Cited by 23
Lin, H., Luo, Z., Ma, J., & Chen, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.611 · Cited by 40
Multimodal detection of hateful memes by applying a vision-language pre-training model
Chen, Y., & Pan, F. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274300 · Cited by 25
Understanding and Detecting Hateful Content Using Contrastive Learning
González-Pizarro, F., & Zannettou, S. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22143 · Cited by 25
Metadata
- Authors
- Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D.
- Affiliations
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 33
- Pages
- 2611-2624
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.04790
- OpenAlex
- W3023989664
- Cited by
- 352
- References
- 93
- Citations by year
- 2026: 3 · 2025: 47 · 2024: 78 · 2023: 82 · 2022: 73 · 2021: 53 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, multimodal classification, benchmark dataset, human performance comparison, binary classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
