The DOI matches its DataCite record; 93 of the 93 references it cites were found in a registry or an index.
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Open accessAbstract
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For instance, the Hateful Memes dataset has shorter text sequences (i.e., 128) compared to other datasets, in which prompts may become more influential to the final performance if the prompt length is too long. This may cause ambiguity for the model to learn task-specific features, e.g., input-level prompting performs slightly worse on Hateful Memes, especially for the text-missing case in Table 1”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Next, we show that input-level prompting further improves the performance in most settings, except for one case on the Hateful Memes dataset, in which this sensitivity analysis will be discussed in the following section.”
From Multimodal Prompting with Missing Modalities for Visual Recognition (2023) · cites this paper as [12]
In the citing paper
“Hateful Memes dataset is constructed to fail models that rely on a single modality by adding challenging samples ('benign confounders') to the dataset.”
From Are Multimodal Transformers Robust to Missing Modality? (2022) · cites this paper as [17]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter
Talat, Z., & Hovy, D. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-2013 · Cited by 1,919
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00686 · Cited by 1,797
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,715
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
A Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · SocialNLP@EACL · doi:10.18653/v1/w17-1101 · Cited by 1,478
A Survey on Automatic Detection of Hate Speech in Text
Fortuna, P., & Nunes, S. (2019) · ACM Computing Surveys · doi:10.1145/3232676 · Cited by 1,241
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 951
Self-citationThe Risk of Racial Bias in Hate Speech Detection
Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1163 · Cited by 817
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674
A survey of multimodal sentiment analysis
Soleymani, M., García, D., Jou, B., Schuller, B. W., Chang, S., & Pantić, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.08.003 · Cited by 647
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 634
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 4×
“In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
BackgroundInfluentialGated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 610
Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter
Talat, Z. (2016) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/w16-5618 · Cited by 607
Fake News Detection on Social Media: A Data Mining PerspectivePreprint
Shu, K., Sliva, A. L., Wang, S., Tang, J., & Liu, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01967 · Cited by 585
Understanding Abuse: A Typology of Abusive Language Detection Subtasks
Waseem, Z., Davidson, T., Warmsley, D., & Weber, I. G. (2017) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w17-3012 · Cited by 486
Learning the Difference that Makes a Difference with Counterfactually-Augmented DataPreprint
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12434 · Cited by 434
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · Cited by 432
Self-citationRacial Bias in Hate Speech and Abusive Language Detection Datasets
Davidson, T., Bhattacharya, D., & Weber, I. (2019) · Proceedings of the Third Workshop on Abusive Language Online · doi:10.18653/v1/w19-3504 · Cited by 414
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue
de Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., & Courville, A. C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.475 · Cited by 385
Benchmarking Aggression Identification in Social Media.
Kumar, R., Ojha, A. K., Malmasi, S., & Zampieri, M. (2018) · International Conference on Computational Linguistics · Cited by 360
Supervised Multimodal Bitransformers for Classifying Images and TextPreprint
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 345
Self-citationVisual Attention Model for Name Tagging in Multimodal Social Media
Lu, D., Neves, L., Carvalho, V., Zhang, N., & Ji, H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1185 · Cited by 344
Visual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 319
Visual Entailment: A Novel Task for Fine-Grained Image UnderstandingPreprint
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 311
Adapting BERT for Target-Oriented Multimodal Sentiment Classification
Yu, J., & Jiang, J. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/751 · Cited by 306
Analyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297
Ma, Y., Xiang, Z., Du, Q., & Fan, W. (2017) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.12.008 · Cited by 293
Exploring Hate Speech Detection in Multimodal Publications
Gómez, R., Gibert, J., Gómez, L., & Karatzas, D. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093414 · Cited by 284
TextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_44 · Cited by 283
Self-citationChallenges in discriminating profanity from hate speech
Malmasi, S., & Zampieri, M. (2017) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2017.1409284 · Cited by 273
Simple Baseline for Visual Question AnsweringPreprint
Zhou, B., Tian, Y., Sukhbaatar, S., Szlam, A. D., & Fergus, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02167 · Cited by 266
Huang, T.-H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1147 · Cited by 254
The New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247
Revisiting Visual Question Answering Baselines
Jabri, A., Joulin, A., & van der Maaten, L. J. P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_44 · Cited by 238
A Shared Task on Multimodal Machine Translation and Crosslingual Image Description
Specia, L., Frank, S., Sima’an, K., & Elliott, D. (2016) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w16-2346 · Cited by 228
Recipe recognition with large multimodal food dataset
Wang, X., Kumar, D., Thome, N., Cord, M., & Precioso, F. (2015) · IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew.2015.7169757 · Cited by 211
A Large Labeled Corpus for Online Harassment Research
Golbeck, J., Ashktorab, Z., Banjo, R. O., Berlinger, A., Bhagwan, S., Buntain, C., Cheakalos, P., Geller, A. A., Gergory, Q., Gnanasekaran, R. K., et al. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3091509 · Cited by 203
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182
Exploring Nearest Neighbor Approaches for Image CaptioningPreprint
Devlin, J., Gupta, S., Girshick, R., Mitchell, M., & Zitnick, C. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04467 · Cited by 175
Automatic Understanding of Image and Video Advertisements
Hussain, Z., Zhang, M., Zhang, X., Ye, K., Thomas, C., Agha, Z., Ong, N., & Kovashka, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.123 · Cited by 160
Probing the Need for Visual Context in Multimodal Machine Translation
Çağlayan, O., Madhyastha, P., Specia, L., & Barrault, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1422 · Cited by 157
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
Content-driven detection of cyberbullying on the instagram social network
Zhong, H., Li, H., Squicciarini, A., Rajtmajer, S. M., Griffin, C. H., Miller, D. J., & Caragea, C. (2016) · International Joint Conference on Artificial Intelligence · Cited by 133
Multi30K: Multilingual English-German Image Descriptions
Elliott, D., Frank, S., Sima’an, K., & Specia, L. (2016) · Workshop on Vision and Language (VL) · doi:10.18653/v1/w16-3210 · Cited by 129
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Kruk, J., Lubin, J., Sikka, K., Lin, X., Jurafsky, D., & Divakaran, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1469 · Cited by 123
Grounding Distributional Semantics in the Visual World
Baroni, M. (2015) · Language and Linguistics Compass · doi:10.1111/lnc3.12170 · Cited by 104
Toward Multimodal Cyberbullying Detection
Singh, V. K., Ghosh, S., & Jose, C. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053169 · Cited by 96
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification
Yang, F., Peng, X., Ghosh, G., Shilon, R., Ma, H., Moore, E., & Predović, G. (2019) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w19-3502 · Cited by 94
Adversarial Evaluation of Multimodal Machine Translation
Elliott, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1329 · Cited by 90
Hate Speech in Pixels: Detection of Offensive Memes towards Automatic ModerationPreprint
Sàbat, B. O., Canton-Ferrer, C., & Giró-i-Nieto, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02334 · Cited by 89
Visual StorytellingPreprint
Ting-Hao, Huang, Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.03968 · Cited by 89
Prediction of Cyberbullying Incidents on the Instagram Social NetworkPreprint
Hosseinmardi, H., Mattson, S. A., Rafiq, R. I., Han, R. Y., Lv, Q., & Mishra, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.06257 · Cited by 81
On the Adequacy of Untuned Warmup for Adaptive Optimization
Ma, J., & Yarats, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17069 · Cited by 78
Expressing an image stream with a sequence of natural sentences
Park, C. C., & Kim, G. (2015) · Seoul National University Open Repository (Seoul National University) · Cited by 75
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65
Characterizing and Detecting Hateful Users on TwitterPreprint
Ribeiro, M. H., Calais, P., Santos, Y. A., Almeida, V. A. F., & Meira, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08977 · Cited by 57
Multimodal Classification Fusion in Real-World Scenarios
Gallo, I., Calefati, A., & Nawaz, S. (2017) · doi:10.1109/icdar.2017.326 · Cited by 55
Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior
Founta, A.-M., Djouvas, C., Chatzakou, D., Ilias Leontiadis, A., Blackburn, J., Stringhini, G., Vakali, A. I., Sirivianos, M., & Kourtellis, N. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1443347 · Cited by 54
Are we pretraining it right? Digging deeper into visio-linguistic pretrainingPreprint
Singh, A., Goswami, V., & Parikh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08744 · Cited by 47
Self-citationEqual But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and TextPreprint
Zhang, M., Hwa, R., & Kovashka, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.08205 · Cited by 46
Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence ResearchPreprint
Kiela, D., Bulat, L., Verő, A. L., & Clark, S. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07432 · Cited by 43
Self-citationAn empirical study on the effectiveness of images in Multimodal Neural Machine Translation
Delbrouck, J.-B., & Dupont, S. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1095 · Cited by 39
CrisisMMD: Multimodal Twitter Datasets from Natural Disasters
Alam, F., Ofli, F., & Imran, M. (2018) · National Conference on Artificial Intelligence · doi:10.5072/fk2/0yu5rd · Cited by 36
Detecting Hate Speech in Social MediaPreprint
Malmasi, S., & Zampieri, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06427 · Cited by 33
Interpretable Multi-Modal Hate Speech DetectionPreprint
Vijayaraghavan, P., Larochelle, H., & Roy, D. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01616 · Cited by 25
Multimodal News Article Analysis
Ramisa, A. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/737 · Cited by 7
Flagged first, then by citations
Cited by
Sorted by citations
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“VisDial [20], a visual dialogue benchmark; HatefulMemes [54] a vision and text classification benchmark; NextQA [129] which specially focuses on causality and temporal relation; STAR [128], a multiple-choice question answering task; and RareAct [73], a benchmark measuring compositionality in action recognition.”
BackgroundMethodInfluentialFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678
Self-citation
Metadata
- Authors
- Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D.
- Affiliations
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 33
- Pages
- 2611-2624
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.04790
- OpenAlex
- W3023989664
- Cited by
- 352
- References
- 93
- Citations by year
- 2026: 3 · 2025: 47 · 2024: 78 · 2023: 82 · 2022: 73 · 2021: 53 · 2020: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- neural information processing systems
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- hate speech detection, multimodal classification, benchmark dataset, human performance comparison, binary classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
