CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Open accessAbstract
Cross-modal retrieval has become a highlighted research topic for retrieval across multimedia data such as image and text. A two-stage learning framework is widely adopted by most existing methods based on deep neural network (DNN):The first learning stageis to generate separate representation for each modality andthe second learning stageis to get the cross-modal common representation. However the existing methods have three limitations: 1) Inthe first learning stagethey only model intramodality correlation but ignore intermodality correlation with rich complementary context. 2) Inthe second learning stagethey only adopt shallow networks with single-loss regularization but ignore the intrinsic relevance of intramodality and intermodality correlation. 3) Only original instances are considered while the complementary fine-grained clues provided by their patches are ignored. For addressing the above problems this paper proposes a cross-modal correlation learning (CCL) approach with multigrained fusion by hierarchical network and the contributions are as follows: 1) Inthe first learning stageCCL exploits multilevel association with joint optimization to preserve the complementary context from intramodality and intermodality correlation simultaneously. 2) Inthe second learning stagea multitask learning strategy is designed to adaptively balance the intramodality semantic category constraints and intermodality pairwise similarity constraints. 3) CCL adopts multigrained modeling which fuses the coarse-grained instances and fine-grained patches to make cross-modal correlation more precise. Comparing with 13 state-of-the-art methods on 6 widely-used cross-modal datasets the experimental results show our CCL approach achieves the best performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Inter- and intra-modal interaction gives perceptual models discriminating power for multisensory stimuli by exploiting synergies between modalities and their intrinsic representations.
“Another crucial advantage of inter- and intra-modal interaction is the discriminating power of the perceptual model for multisensory stimuli by exploiting the potential synergies between modalities and their intrinsic representations [112].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [112]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980 · see citing works
RELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,429 · see citing works
R-FCN: Object Detection via Region-based Fully Convolutional NetworksPreprint
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 5,135 · see citing works
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,297 · see citing works
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025 · see citing works
(2011) · Cited by 2,752 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
Deep Learning Face Representation by Joint Identification-Verification
Sun, Y., Wang, X., & Tang, X. (2014) · arXiv · doi:10.48550/arxiv.1406.4773 · Cited by 1,906 · see citing works
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · Cited by 1,603 · see citing works
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R. G., Lévy, R., & Vasconcelos, N. M. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1873987 · Cited by 1,358 · see citing works
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654902 · Cited by 572 · see citing works
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541 · see citing works
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics
Gong, Y., Ke, Q., Isard, M., & Lazebnik, S. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0658-4 · Cited by 539 · see citing works
Replicated Softmax: an Undirected Topic Model
Hinton, G. E., & Salakhutdinov, R. (2009) · neural information processing systems · Cited by 425 · see citing works
Exponential Family Harmoniums with an Application to Information Retrieval
Welling, M., Rosen‐Zvi, M., & Hinton, G. E. (2004) · Cited by 423 · see citing works
Deep correlation for matching images and text
Yan, F., & Mikolajczyk, K. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298966 · Cited by 407 · see citing works
Joint Feature Selection and Subspace Learning for Cross-Modal Retrieval
Wang, K., He, R., Wang, L., Wang, W., & Tan, T. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505311 · Cited by 318 · see citing works
Learning Cross-Media Joint Representation With Sparse and Semisupervised Regularization
Zhai, X., Peng, Y., & Xiao, J. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2276704 · Cited by 312 · see citing works
Self-citationMulti-Task CNN Model for Attribute Prediction
Abdulnabi, A. H., Wang, G., Lu, J., & Jia, K. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477680 · Cited by 307 · see citing works
Associating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299073 · Cited by 290 · see citing works
Multimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957143 · Cited by 270 · see citing works
Multi-label Cross-Modal Retrieval
Ranjan, V., Rasiwasia, N., & Jawahar, C. V. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.466 · Cited by 244 · see citing works
Learning Consistent Feature Representation for Cross-Modal Multimedia Retrieval
Kang, C., Xiang, S., Liao, S., Xu, C., & Pan, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2390499 · Cited by 222 · see citing works
Shaban, A., & Yousefi, S. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 206 · see citing works
Supervised Coupled Dictionary Learning with Group Structures for Multi-modal Retrieval
Yue, Z., Wang, Y., Wu, F. Q., Zhang⋆, Y., & Lu, W. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8603 · Cited by 196 · see citing works
Learning Representations for Multimodal Data with Deep Belief Nets
Srivastava, N., & Salakhutdinov, R. (2012) · Cited by 181 · see citing works
Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification
Wu, Z., Jiang, Y., Wang, X., Ye, H., & Xue, X. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964328 · Cited by 172 · see citing works
Cross-media shared representation by hierarchical learning with multiple deep networks
Peng, Y., Huang, X., & Qi, J. (2016) · International Joint Conference on Artificial Intelligence · Cited by 168 · see citing works
Self-citationImageNet pre-trained models with batch normalizationPreprint
Simon, M., Rodner, E., & Denzler, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01452 · Cited by 157 · see citing works
Semi-Supervised Cross-Media Feature Learning With Unified Patch Graph Regularization
Peng, Y., Zhai, X., Zhao, Y., & Huang, X. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2400779 · Cited by 153 · see citing works
Self-citationHeterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval
Zhai, X., Peng, Y., & Xiao, J. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8464 · Cited by 134 · see citing works
Self-citationMultimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957142.957143 · Cited by 106 · see citing works
Clip-based similarity measure for query-dependent clip retrieval and video summarization
Peng, Y., & Ngo, C. (2006) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2006.873157 · Cited by 104 · see citing works
Self-citationHe, X., & Peng, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11223 · Cited by 103 · see citing works
Self-citationCross-Modal Retrieval Using Multiordered Discriminative Structured Subspace Learning
Zhang, L., Ma, B., Li, G., et al. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2646219 · Cited by 66 · see citing works
Cross-Modal Correlation Learning by Adaptive Hierarchical Semantic Aggregation
Yan, H., Wang, S., Liu, S., et al. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2535864 · Cited by 50 · see citing works
Coherent bag-of audio words model for efficient large-scale video copy detection
Liu, Y., Zhao, W., Ngo, C., et al. (2010) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1816041.1816057 · Cited by 48 · see citing works
Audio-Visual Speech Synchrony Measure for Talking-Face Identity Verification
Bredin, H., & Chollet, G. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366215 · Cited by 43 · see citing works
Search and Retrieval of Rich Media Objects Supporting Multiple Multimodal Queries
Daras, P., Manolopoulou, S., & Αξενόπουλος, Α. (2011) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2181343 · Cited by 41 · see citing works
Coherent Phrase Model for Efficient Image Near-Duplicate Retrieval
Hu, Y., Cheng, X., Chia, L.-T., Xie, X., Rajan, D., & Tan, A. (2009) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2009.2032676 · Cited by 30 · see citing works
Aggregating Image and Text Quantized Correlated Components
Tran, T. Q. N., Borgne, H. L., & Crucianu, M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.225 · Cited by 24 · see citing works
Bag-of-multimedia-words for image classification
Znaidia, A., Shabou, A., Borgne, H. L., et al. (2012) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Scalable Deep Multimodal Learning for Cross-Modal Retrieval
Hu, P., Zhen, L., Peng, D., & Liu, P. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331213 · Cited by 136 · see citing works
Multimodal data fusion for systems improvement: A review
Gaw, N., Yousefi, S., & Gahrooei, M. R. (2022) · IISE Transactions · doi:10.1080/24725854.2021.1987593 · Cited by 101 · see citing works
Joint feature synthesis and embedding: Adversarial cross-modal retrieval revisited
Xu, X., Lin, K., Yang, Y., Hanjalic, A., & Shen, H. T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045530 · Cited by 88 · see citing works
- Authors
- Peng, Y., Qi, J., Huang, X., & Yuan, Y.
- Affiliations
- Peking University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2018-02
- Volume
- 20
- Issue
- 2
- Pages
- 405-420
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2605649771
- Cited by
- 228
- References
- 50
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 18 · 2023: 17 · 2022: 30 · 2021: 33 · 2020: 50 · 2019: 40 · 2018: 21 · 2017: 3
- FWCI
- 12.78
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61771025, 6171128, 61532005)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- cross-modal retrieval, hierarchical networks, multi-task learning, image-text retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
