Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
Open accessAbstract
We propose to solve the natural language inference problem without any supervision from the inference labels via task-agnostic multimodal pretraining.Although recent studies of multimodal self-supervised learning also represent the linguistic and visual context, their encoders for different modalities are coupled.Thus they cannot incorporate visual information when encoding plain text alone.In this paper, we propose Multimodal Aligned Contrastive Decoupled learning (MACD) network.MACD forces the decoupled text encoder to represent the visual information via contrastive learning.Therefore, it embeds visual knowledge even for plain text inference.We conducted comprehensive experiments over plain text inference datasets (i.e.SNLI and STS-B).The unsupervised MACD even outperforms the fully-supervised BiLST-M and BiLSTM+ELMO on STS-B.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work has shown utility of alignment objectives in learning representations for prediction.
“While primarily useful for crossmodal retrieval [104, 187], recent work has also shown its utility in learning representations for prediction [9, 33, 91, 151].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [33]
Contrastive learning has emerged as a popular approach that brings similar concepts close in feature space and different concepts far away.
“In the latter, contrastive learning has emerged as a popular approach that brings similar concepts close in feature space and different concepts far away [33, 91, 151].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works
Bar, M. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1476 · Cited by 1,613 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. 2019. Fusion of detected objects in text for visual question answering. In EMNLP-IJCNLP, pages 2131-2140.
- Moshe Bar. 2007. The proactive brain: using analogies and associations to generate predictions. Trends in cognitive sciences, 11(7):280-289.
- Moshe Bar. 2009. The proactive brain: memory for predictions. Philosophical Transactions of the Roy- al Society B: Biological Sciences, 364(1521):1235- 1243.
- Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. 2018. Optimizing the latent s- pace of generative networks. In ICML, pages 600- 609.
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. In EMNLP.
- Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017, pages 1-14.
- Karol Gregor and Yann LeCun. 2010. Learning fast approximations of sparse coding. In ICML, pages 399-406.
- Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, pages 297-304.
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. CVPR.
- Donald Olding Hebb. 2005. The organization of be- havior: A neuropsychological theory. Psychology Press.
- Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In ICLR.
- Donald Homa, Sharon Sterling, and Lawrence Trepel. 1981. Limitations of exemplar-based generalization and the abstraction of categorical information. Jour- nal of Experimental Psychology: Human Learning and Memory, 7(6):418.
- Douwe Kiela, Alexis Conneau, Allan Jabri, and Max- imilian Nickel. 2018. Learning visually grounded sentence representations. In NAACL-HLT, pages 408-418.
- WB Levy and O Steward. 1983. Temporal contigui- ty requirements for long-term associative potentia- tion/depression in the hippocampus. Neuroscience, 8(4):791-797.
- Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui H- sieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557. · arXiv
- Zhizhong Li and Derek Hoiem. 2017. Learning with- out forgetting. TPAMI, 40(12):2935-2947.
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740- 755.
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining ap- proach. CoRR, abs/1907.11692. · arXiv
- Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi- olinguistic representations for vision-and-language tasks. In NeurIPS, pages 13-23.
- Ishan Misra and Laurens van der Maaten. 2020. Self- supervised learning of pretext-invariant representa- tions. CVPR.
- Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive pre- dictive coding. arXiv preprint arXiv:1807.03748. · arXiv
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In NAACL.
- Virginia R de Sa. 1994. Learning classification with unlabeled data. In NeurIPS, pages 112-119.
- Chen Sun, Austin Myers, Carl Vondrick, Kevin Mur- phy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learn- ing. In ICCV, pages 7464-7473.
- Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In AAAI.
- Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP-IJCNLP, pages 5103-5114.
- Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive multiview coding. arXiv preprint arX- iv:1906.05849.
- Amos Tversky and Daniel Kahneman. 1973. Availabil- ity: A heuristic for judging frequency and probabili- ty. Cognitive psychology, 5(2):207-232.
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP, page 353.
- Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. TACL, 2:67-78.
Cited by
Sorted by citations
COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation
Wen, K., Xia, J., Huang, Y., Li, L., Xu, J., & Shao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00221 · Cited by 37 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
- Authors
- Cui, W., Zheng, G., & Wang, W.
- Affiliations
- Shanghai University of Finance and Economics (CN)
- Fudan University (CN)
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 5511-5520
- OpenAlex
- W3099303748
- Cited by
- 17
- References
- 7
- Citations by year
- 2024: 2 · 2023: 2 · 2022: 7 · 2021: 6
- FWCI
- 1.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61732004)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- natural language inference, unsupervised learning, multimodal contrastive learning, multimodal pre-training, semantic textual similarity, SNLI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
