Multimodal Convolutional Neural Networks for Matching Image and Sentence
Abstract
In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content and one matching CNN modeling the joint representation of image and sentence. The matching CNN composes different semantic fragments from words and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. More specifically, our proposed m-CNNs significantly outperform the state-of-the-art approaches for bidirectional image and sentence retrieval on the Flickr8K and Flickr30K datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Adaptive multimodal mapping between two visual modalities typically requires strong representations of the individual modalities.
“An adaptive multimodal mapping between two visual modalities (e.g., images and sentences) typically requires strong representations of the individual modalities [213].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [213]
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389824 · Cited by 11,971 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,637 · see citing works
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · see citing works
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_23 · Cited by 3,034 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · see citing works
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · see citing works
Convolutional Neural Network Architectures for Matching Natural Language SentencesPreprint
Hu, B., Lu, Z., Li, H., & Chen, Q. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03244 · Cited by 1,168 · see citing works
Self-citationLong-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)Preprint
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 1,039 · see citing works
Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping
Caruana, R., Lawrence, S., & Giles, C. L. (2000) · Cited by 906 · see citing works
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel‐Hamid, O., Mohamed, A., Jiang, H., & Penn, G. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288864 · Cited by 892 · see citing works
Deep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · see citing works
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · see citing works
Wsabie: Scaling up to large vocabulary image annotation
Weston, J., Samy Bengio, A., & Usunier, N. (2011) · Cited by 659 · see citing works
Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · Cited by 593 · see citing works
Explain Images with Multimodal Recurrent Neural NetworksPreprint
Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.1090 · Cited by 423 · see citing works
Recognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · doi:10.1109/cvpr.2011.5995711 · Cited by 421 · see citing works
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_35 · Cited by 286 · see citing works
Learning Representations for Multimodal Data with Deep Belief Nets
Srivastava, N., & Salakhutdinov, R. (2012) · Cited by 181 · see citing works
Learning a Recurrent Visual Representation for Image Caption GenerationPreprint
Chen, X., & Zitnick, C. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5654 · Cited by 170 · see citing works
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 156 · see citing works
Show and Tell: A Neural Image Caption GeneratorPreprint
Vinyals, O., Toshev, A., Samy Bengio, A., & Erhan, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4555 · Cited by 156 · see citing works
Framing image description as a ranking task
HodoshMicah, YoungPeter, & HockenmaierJulia (2013) · doi:10.5555/2566972.2566993 · Cited by 98 · see citing works
Deep Representations and Codes for Image Auto-Annotation
Kiros, R., & Szepesvári, C. (2012) · Cited by 36 · see citing works
A Neural Network to Retrieve Images from Text Queries
Grangier, D., & Bengio, S. (2006) · Lecture notes in computer science · doi:10.1007/11840930_3 · Cited by 7 · see citing works
Flagged first, then by citations
2 of the 40 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795 · see citing works
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.541 · Cited by 788 · see citing works
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779 · see citing works
Dual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.232 · Cited by 749 · see citing works
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3383184 · Cited by 622 · see citing works
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Wang, L., Li, Y., Huang, J. L., & Lazebnik, S. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2797921 · Cited by 538 · see citing works
Deep Cross-Modal Projection Learning for Image-Text Matching
Zhang, Y., & Lu, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_42 · Cited by 513 · see citing works
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., & Han, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01267 · Cited by 451 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00750 · Cited by 409 · see citing works
TediGAN: Text-Guided Diverse Face Image Generation and Manipulation
Xia, W., Yang, Y., Xue, J.-H., & Wu, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00229 · Cited by 376 · see citing works
Stock price prediction using deep learning and frequency decomposition
Rezaei, H., Faaljou, H., & Mansourfar, G. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114332 · Cited by 387 · see citing works
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00645 · Cited by 353 · see citing works
Reconstruction Network for Video Captioning
Wang, B., Ma, L., Zhang, W., & Liu, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00795 · Cited by 350 · see citing works
Self-citationKim, T., & Kim, H. Y. (2019) · PLoS ONE · doi:10.1371/journal.pone.0212320 · Cited by 347 · see citing works
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & van den Hengel, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01000 · Cited by 331 · see citing works
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
Xu, H., He, K., Plummer, B. A., Sigal, L., Sclaroff, S., & Saenko, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019062 · Cited by 318 · see citing works
Context-Aware Attention Network for Image-Text Retrieval
Zhang, Q., Lei, Z., Zhang, Z., & Li, S. Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00359 · Cited by 304 · see citing works
Graph Structured Network for Image-Text Matching
Liu, C., Mao, Z., Zhang, T., Xie, H., Wang, B., & Zhang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01093 · Cited by 305 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 375 · see citing works
Order-Embeddings of Images and Language
Vendrov, I., Kiros, R., Fidler, S., & Urtasun, R. (2015) · arXiv · doi:10.48550/arxiv.1511.06361 · Cited by 296 · see citing works
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 324 · see citing works
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Song, Y., & Soleymani, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00208 · Cited by 287 · see citing works
Hussein, S., Kandel, P., Bolan, C. W., Wallace, M. B., & Bagci, U. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2894349 · Cited by 285 · see citing works
Cross-Modal Attention With Semantic Consistence for Image–Text Matching
Xu, X., Wang, T., Yang, Y., Zuo, L., Shen, F., & Shen, H. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2967597 · Cited by 282 · see citing works
Adversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 258 · see citing works
Learning to Answer Questions from Image Using Convolutional Neural Network
Ma, L., Lu, Z., & Li, H. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10442 · Cited by 260 · see citing works
Self-citationDetecting Sarcasm in Multimodal Social Platforms
Schifanella, R., de Juan, P., Tetreault, J., & Cao, L. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964321 · Cited by 253 · see citing works
A survey on automatic image caption generation
Bai, S., & An, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.080 · Cited by 243 · see citing works
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.767 · Cited by 237 · see citing works
An analytical study of information extraction from unstructured and multidimensional big data
Adnan, K., & Akbar, R. (2019) · Journal Of Big Data · doi:10.1186/s40537-019-0254-8 · Cited by 231 · see citing works
Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
Wang, J., Jiang, W., Ma, L., Liu, W., & Xu, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00751 · Cited by 231 · see citing works
Self-citationNegative-Aware Attention Framework for Image-Text Matching
Zhang, K., Mao, Z., Wang, Q., & Zhang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01521 · Cited by 216 · see citing works
Position Focused Attention Network for Image-Text Matching
Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J. J., Li, B., & Fan, X. N. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/526 · Cited by 208 · see citing works
Self-citationPredicting Visual Features From Text for Image and Video Caption Retrieval
Dong, J., Li, X., & Snoek, C. G. M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2832602 · Cited by 198 · see citing works
Discriminability Objective for Training Descriptive Captions
Luo, R., Shakhnarovich, G., Cohen, S. D., & Price, B. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00728 · Cited by 194 · see citing works
Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
Wang, H., Zhang, Y., Ji, Z., Pang, Y., & Ma, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_2 · Cited by 194 · see citing works
Self-citationControllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network
Wang, B., Ma, L., Zhang, W., Jiang, W., Wang, J., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00273 · Cited by 199 · see citing works
Self-citationLiu, C., Mao, Z., Liu, A.-A., Zhang, T., Wang, B., & Zhang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350869 · Cited by 206 · see citing works
Linking Image and Text with 2-Way Nets
Eisenschtat, A., & Wolf, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.201 · Cited by 171 · see citing works
RNN Fisher Vectors for Action Recognition and Image Annotation
Lev, G., Sadeh, G., Klein, B., & Wolf, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_50 · Cited by 170 · see citing works
Multi-modal Attention Network Learning for Semantic Source Code Retrieval
Wan, Y., Shu, J., Sui, Y., Xu, G., Zhao, Z., Wu, J., & Yu, P. S. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00012 · Cited by 168 · see citing works
Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues
Plummer, B. A., Mallya, A., Cervantes, C. M., Hockenmaier, J., & Lazebnik, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.213 · Cited by 167 · see citing works
Engaging Image Captioning via Personality
Shuster, K., Humeau, S., Hu, H., Bordes, A., & Weston, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01280 · Cited by 160 · see citing works
Visual Sentiment Analysis by Attending on Local Image Regions
You, Q., Jin, H., & Luo, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10501 · Cited by 162 · see citing works
Learning a Recurrent Residual Fusion Network for Multimodal Matching
Liu, Y., Guo, Y., Bakker, E. M., & Lew, M. S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.442 · Cited by 163 · see citing works
Wu, H., Mao, J., Zhang, Y., Jiang, Y., Li, L., Sun, W., & Ma, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00677 · Cited by 162 · see citing works
Towards Context-Aware Interaction Recognition for Visual Relationship Detection
Zhuang, B., Liu, L., Shen, C., & Reid, I. D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.71 · Cited by 161 · see citing works
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.291 · Cited by 147 · see citing works
Composite Correlation Quantization for Efficient Multimodal Retrieval
Long, M., Cao, Y., Wang, J., & Yu, P. S. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911493 · Cited by 146 · see citing works
Comprehension-Guided Referring Expressions
Luo, R., & Shakhnarovich, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.333 · Cited by 147 · see citing works
Cross-Media Learning for Image Sentiment Analysis in the Wild
Vadicamo, L., Carrara, F., Cimino, A., Cresci, S., Dell’Orletta⋄, F., Falchi, F., & Tesconi, M. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.45 · Cited by 144 · see citing works
Zhang, W., Qu, C., Ma, L., Guan, J., & Huang, R. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.01.034 · Cited by 141 · see citing works
Self-citationNeural information retrieval: at the end of the early years
Onal, K. D., Zhang, Y., Altıngövde, İ. S., Rahman, M. M., Karagöz, P., Braylan, A., Dang, B., Chang, H.-L., Kim, H., McNamara, Q., et al. (2017) · Information Retrieval · doi:10.1007/s10791-017-9321-y · Cited by 137 · see citing works
Referring Expression Generation and Comprehension via Attributes
Liu, J., Wang, L., & Yang, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.520 · Cited by 136 · see citing works
Video Summarization via Semantic Attended Networks
Wei, H., Ni, B., Yan, Y., Yu, H., Yang, X., & Yao, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11297 · Cited by 136 · see citing works
Learning Fragment Self-Attention Embeddings for Image-Text Matching
Wu, Y., Wang, S., Song, G., & Huang, Q. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350940 · Cited by 132 · see citing works
Cross-modal Graph Matching Network for Image-text Retrieval
Cheng, Y., Zhu, X., Qian, J., Wen, F., & Liu, P. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3499027 · Cited by 125 · see citing works
Saliency-Guided Attention Network for Image-Sentence Matching
Ji, Z., Wang, H., Han, J., & Pang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00585 · Cited by 119 · see citing works
Dual-Level Representation Enhancement on Characteristic and Context for Image-Text Retrieval
Yang, S. S., Li, Q., Li, W., Li, X., & Liu, A.-A. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3182426 · Cited by 120 · see citing works
Step-Wise Hierarchical Alignment Network for Image-Text Matching
Ji, Z., Chen, K., & Wang, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/106 · Cited by 120 · see citing works
Enhancing Video Summarization via Vision-Language Embedding
Plummer, B. A., Brown, M., & Lazebnik, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.118 · Cited by 112 · see citing works
Deep Adversarial Graph Attention Convolution Network for Text-Based Person Search
Liu, J., Zha, Z.-J., Hong, R., Wang, M., & Zhang, Y. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350991 · Cited by 112 · see citing works
LapsCore: Language-guided Person Search via Color Reasoning
Wu, Y., Yan, Z., Han, X., Li, G., Zou, C., & Cui, S. R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00165 · Cited by 111 · see citing works
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 108 · see citing works
Robust Visual-Textual Sentiment Analysis
You, Q., Cao, L., Jin, H., & Luo, J. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964288 · Cited by 106 · see citing works
Image Understanding using vision and reasoning through Scene Description Graph
Aditya, S., Yang, Y., Baral, C. R., Aloimonos, Y., & Fermuller, C. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.12.004 · Cited by 103 · see citing works
Bi-Directional Spatial-Semantic Attention Networks for Image-Text Matching
Huang, F., Zhang, X., Zhao, Z., & Li, Z. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2882225 · Cited by 104 · see citing works
Topic-Oriented Image Captioning Based on Order-Embedding
Yu, N., Hu, X., Song, B., Yang, J., & Zhang, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2889922 · Cited by 101 · see citing works
Yu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972830 · Cited by 96 · see citing works
Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
Chen, Z., Ma, L., Luo, W., & Wong, K.-Y. K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1183 · Cited by 96 · see citing works
Self-citationLocalizing Natural Language in Videos
Chen, J., Ma, L., Chen, X., Jie, Z., & Luo, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018175 · Cited by 94 · see citing works
Self-citationAlign2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
Datta, S., Sikka, K., Roy, A., Ahuja, K., Parikh, D., & Divakaran, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00269 · Cited by 90 · see citing works
Adaptive Cross-Modal Embeddings for Image-Text Alignment
Wehrmann, J., Kolling, C., & Barros, R. C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6915 · Cited by 88 · see citing works
Syntax-Guided Hierarchical Attention Network for Video Captioning
Deng, J., Li, L., Zhang, B., Wang, S., Zha, Z.-J., & Huang, Q. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3063423 · Cited by 87 · see citing works
Unified Adaptive Relevance Distinguishable Attention Network for Image-Text Matching
Zhang, K., Mao, Z., Liu, A.-A., & Zhang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3141603 · Cited by 89 · see citing works
Leveraging Visual Question Answering for Image-Caption Ranking
Lin, X., & Parikh, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_17 · Cited by 81 · see citing works
Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning
Zhang, W., Wang, B., Ma, L., & Liu, W. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2920899 · Cited by 83 · see citing works
Self-citationMultimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works
Retrieval Topic Recurrent Memory Network for Remote Sensing Image Captioning
Wang, B., Zheng, X., Qu, B., & Lu, X. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2959208 · Cited by 79 · see citing works
Geometry Attention Transformer with position-aware LSTMs for image captioning
Wang, C., Shen, Y., & Ji, L. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117174 · Cited by 74 · see citing works
Cross-Modal Image-Text Retrieval with Semantic Consistency
Chen, H., Ding, G., Lin, Z., Zhao, S., & Han, J. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351055 · Cited by 75 · see citing works
Sun, B., Wu, Y., Zhao, K., He, J. Y. D., Yu, L., Yan, H., & Luo, A. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05587-y · Cited by 77 · see citing works
Deep Transfer Learning for Art Classification Problems
Sabatelli, M., Kestemont, M., Daelemans, W. M. P., & Geurts, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11012-3_48 · Cited by 71 · see citing works
Twitter100k: A Real-World Dataset for Weakly Supervised Cross-Media Retrieval
Hu, Y., Zheng, L., Yang, Y., & Huang, Y. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2760101 · Cited by 71 · see citing works
Visually Grounded Neural Syntax Acquisition
Shi, H., Mao, J., Gimpel, K., & Livescu, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1180 · Cited by 68 · see citing works
ACMM: Aligned Cross-Modal Memory for Few-Shot Image and Sentence Matching
Huang, Y., & Wang, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00587 · Cited by 69 · see citing works
Improving Image Captioning by Leveraging Knowledge Graphs
Zhou, Y., Sun, Y., & Honavar, V. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00036 · Cited by 65 · see citing works
SoDeep: A Sorting Deep Net to Learn Ranking Loss Surrogates
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01105 · Cited by 61 · see citing works
Joint Global and Co-Attentive Representation Learning for Image-Sentence Retrieval
Wang, S., Chen, Y., Zhuo, J., Huang, Q., & Tian, Q. C. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240535 · Cited by 59 · see citing works
Integration of textual cues for fine-grained image captioning using deep CNN and LSTM
Gupta, N., & Jalal, A. S. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04515-z · Cited by 60 · see citing works
Classification of femur fracture in pelvic X-ray images using meta-learned deep neural network
Lee, C., Jang, J., Lee, S., Kim, Y. S., Jo, H. J., & Kim, Y.-S. (2020) · Scientific Reports · doi:10.1038/s41598-020-70660-4 · Cited by 59 · see citing works
Fang, W., Love, P. E. D., Ding, L., Xu, S., Kong, T., & Li, H. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3093166 · Cited by 58 · see citing works
Self-citationDeep Learning for Information Retrieval
Li, H., & Lu, Z. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914800 · Cited by 57 · see citing works
Self-citationA Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech
Cooney, C., Folli, R., & Coyle, D. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3132861 · Cited by 56 · see citing works
GraDual: Graph-based Dual-modal Representation for Image-Text Matching
Long, S., Han, S. C., Wan, X., & Poon, J. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00252 · Cited by 57 · see citing works
Dual-Path Rare Content Enhancement Network for Image and Text Matching
Wang, Y., Su, Y., Li, W., Xiao, J., Li, X., & Liu, A.-A. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3254530 · Cited by 56 · see citing works
Chu, M.-G., Min, B., Kwon, S., Park, G., Kim, S., & Huy, N. X. (2019) · Journal of Petroleum Science and Engineering · doi:10.1016/j.petrol.2019.106805 · Cited by 56 · see citing works
- Authors
- Ma, L., Lu, Z., Shang, L., & Li, H.
- Affiliations
- Huawei Technologies (China) (CN)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 2623-2631
- OpenAlex
- W1916445035
- Cited by
- 334
- References
- 40
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 15 · 2023: 23 · 2022: 29 · 2021: 31 · 2020: 59 · 2019: 67 · 2018: 41 · 2017: 34 · 2016: 18 · 2015: 3
- FWCI
- 13.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multimodal convolutional neural network, image retrieval, sentence retrieval, cross-modal matching, word composition, Flickr8k, Flickr30k
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
