The DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index.
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Abstract
In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content and one matching CNN modeling the joint representation of image and sentence. The matching CNN composes different semantic fragments from words and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. More specifically, our proposed m-CNNs significantly outperform the state-of-the-art approaches for bidirectional image and sentence retrieval on the Flickr8K and Flickr30K datasets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“An adaptive multimodal mapping between two visual modalities (e.g., images and sentences) typically requires strong representations of the individual modalities [213].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [213]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,646
Flagged first, then by citations
2 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- hinton, 2012, Improving neural networks by proventing co-adaptation of feature detecters, arXiv 1207 0580, 6
- kiros, 2014, Multimodal neural language model, ICML, 2
- sermanet, 2014, Overfeat: Intergrated recognition, localization and detection using convolutional networks, arXiv 1312 6229, 2
- girshick, 2014, Rich feature hierachies for accurate object detection and semantic segmentation, CVPR, 8
- ordonez, 2011, Im2txt: Describing images using 1 million captioned photogrphs, NIPS, 1
Cited by
Sorted by citations
Items 301–334 of 334
Online social image ranking in diversified preferences
Zhao, X., Pei, L., Li, T., & Zhang, Z. (2020) · EURASIP Journal on Image and Video Processing · doi:10.1186/s13640-020-00540-4 · Cited by 1
Semantic generation mechanism of news images based on Concept-Net
Chen, Z., Xu, H., Gao, S., & Wang, Y. (2021) · 2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA) · doi:10.1109/icpeca51329.2021.9362691 · Cited by 1
Tree-based Text-Vision BERT for Video Search in Baidu Video Advertising
Yu, T., Liu, J., Yang, Y., Li, Y., Fei, H., & Li, P. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020922 · Cited by 2
Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval FrameworkPreprint
Das, D., Ghouse, N. M., Verma, S. N., & Li, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.02950 · Cited by 1
Upgrading the Newsroom: An Automated Image Selection System for News ArticlesPreprint
Liu, F., Lebret, R., Orel, D., Sordet, P., & Aberer, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11449 · Cited by 1
Goal-driven text descriptions for imagesPreprint
Luo, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12575 · Cited by 4
A Hierarchical Correlation Model for Multi-modal Sentiment Analysis on Social Media
Lin, D., Li, L., Cao, D., & Li, S. (2017) · Chinese Conference on Computer Supported Cooperative Work and Social Computing (ChineseCSCW) · doi:10.1145/3127404.3127419
Grounding natural language phrases in images and video
Plummer, B. A. (2018) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
An Effective Visual Attention Mechanism
Wu, Z., Huang, G., & Cheng, L. (2019) · International Conference on Computer, Network, Communication and Information Systems (CNCI) · doi:10.2991/cnci-19.2019.47
Effendi, J., Tjandra, A., Sakti, S., & Nakamura, S. (2019) · arXiv (Cornell University)
Web text-aided image classification
Wang, D. (2019) · doi:10.32657/10220/47855
Scenario Referring Expression Comprehension via Attributes of Vision and Language
Wei, S., Wang, J., Sun, Y., Jin, G., Liang, J., & Liu, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-31726-3_37
Multi-hop Interactive Cross-Modal Retrieval
Ning, X., Yang, X., & Xu, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37734-2_55
Cross-Modality Video Segment Retrieval with Ensemble Learning
Yu, X., Zhang, Y., & Zhang, R. (2020) · Springer eBooks · doi:10.1007/978-3-030-30671-7_5
An Abnormal State Forecasting Method for Substation Equipment based on Improved LSTM
Li, T., Yu, S., Song, C., Yu, T., Zeng, P., & Shen, L. (2019) · IEEE Sustainable Power and Energy Conference (iSPEC) · doi:10.1109/ispec48194.2019.8975247
Local-to-Global Semantic Supervised Learning for Image Captioning
Wang, J., Duan, Y., Tao, X., & Lü, J. (2020) · International Conference on Communications · doi:10.1109/icc40277.2020.9149264
Detection Mechanism of News-text matching based on Knowledge Graph
Gao, S., Xu, H., Chen, Z., & Wang, Y. (2021) · 2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA) · doi:10.1109/icpeca51329.2021.9362571
Semi-supervised learning for visual relation annotation
Tajrobehkar, M. (2022) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/161913
Man, G., Sun, X., & Wu, W. (2023) · Applied Sciences · doi:10.3390/app13074217
Wiki-match: A multi-model pipeline for image-caption matching task on Wikipedia dataset
Chen, Y., Lei, S., & Sun, Z. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/2/20220539
Xu, M., & Wiese, L. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-3559-4_22
MVItem: A Benchmark for Multi-View Cross-Modal Item Retrieval
Li, B., Zhu, J., Dai, L., Jing, H., Huang, Z., & Sui, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3447872
Nan, H., Miao, Z., Wang, K., Li, W., Chen, H., Wu, X., Pan, X., Qiu, W., & Zhang, H. (2024) · doi:10.1145/3698385.3699877
Enhanced Cross-Modal Retrieval with Low-Rank Multi-Head Self-Attention
Liu, H., Qi, J., & Zhu, Q. (2024) · International Conference on Computer Engineering and Intelligent Control (ICCEIC) · doi:10.1109/icceic64099.2024.10775439
DeepCombine: Multimodal Person Recognition
Trojer, S., Bhateja, V., Peer, P., & Emeršič, Ž. (2025) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro65660.2025.11131792
Self-Paced Attribute Prototype Contrastive Learning for Cross-Modal Materials Perception
Liu, X., Yu, Z., Yang, K., Zeng, H., Jiang, J., & Chen, C. L. P. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3668507
Guo, J., Zhao, L., Liang, Q., Du, X., Xu, Y., & Li, X. (2026) · Applied Sciences · doi:10.3390/app16063018
Learning distributed representations of sentences using neural networks
Conneau, A. (2019) · theses.fr (ABES) · doi:10.70675/b3ecf08ez353ez4532zb829z7c06c26a7e07
Cancer Detection in Full Field Optical Coherence Tomography Images
Mandache, D. (2022) · theses.fr (ABES) · doi:10.70675/a8e12287zf665z4a1fz822fze7174f880867
Deep Inside Visual-Semantic Embeddings
Engilberge, M. (2020) · theses.fr (ABES) · doi:10.70675/0f3a6196zbd83z4b2dz9064z42e952bdfed6
Galaxy redshift estimation from multi-band images with deep learning : strengths and challenges
Ouahmed, M. R. A. (2023) · theses.fr (ABES) · doi:10.70675/687e0665z3cb0z4fadza33bz5a64b0fe417c
Metadata
- Authors
- Ma, L., Lu, Z., Shang, L., & Li, H.
- Affiliations
- Huawei Technologies (China) (CN)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 2623-2631
- OpenAlex
- W1916445035
- Cited by
- 334
- References
- 40
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 15 · 2023: 23 · 2022: 29 · 2021: 31 · 2020: 59 · 2019: 67 · 2018: 41 · 2017: 34 · 2016: 18 · 2015: 3
- FWCI
- 13.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multimodal convolutional neural network, image retrieval, sentence retrieval, cross-modal matching, word composition, Flickr8k, Flickr30k
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
