Scalable Deep Multimodal Learning for Cross-Modal Retrieval
Abstract
Cross-modal retrieval takes one type of data as the query to retrieve relevant data of another type. Most of existing cross-modal retrieval approaches were proposed to learn a common subspace in a joint manner, where the data from all modalities have to be involved during the whole training process. For these approaches, the optimal parameters of different modality-specific transformations are dependent on each other and the whole model has to be retrained when handling samples from new modalities. In this paper, we present a novel cross-modal retrieval method, called Scalable Deep Multimodal Learning (SDML). It proposes to predefine a common subspace, in which the between-class variation is maximized while the within-class variation is minimized. Then, it trains m modality-specific networks for m modalities (one network for each modality) to transform the multimodal data into the predefined common subspace to achieve multimodal learning. Unlike many of the existing methods, our method can train different modality-specific networks independently and thus be scalable to the number of modalities. To the best of our knowledge, the proposed SDML could be one of the first works to independently project data of an unfixed number of modalities into a predefined common subspace. Comprehensive experimental results on four widely-used benchmark datasets demonstrate that the proposed method is effective and efficient in multimodal learning and outperforms the state-of-the-art methods in cross-modal retrieval.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Router Zloss regularization techniques address expert imbalance in MoE models by fine-tuning the gating mechanism, ensuring a more equitable workload distribution across experts and fostering a stable training environment, thereby enhancing model performance and reducing training time and computational overhead.
“Developments in router Zloss regularization techniques plays a crucial role in addressing expert imbalance in MoE models by fine-tuning the gating mechanism, ensuring a more equitable workload distribution across experts and fostering a stable training environment, thereby enhancing model performance and reducing training time and computational overhead [301], [302].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [302]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,865 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 1× · see citing works
“For video, C3D model [37] pre-trained on Sports-1M [13] is used to extract the 4,096-dimensional outputs from the fc7 layer as the video features.”
MethodRELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,894 · see citing works
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025 · see citing works
Proceedings of the International Joint Conference on Artificial Intelligence 2007
Bundy, A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 2,761 · see citing works
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · Cited by 1,603 · see citing works
On Visual Similarity Based 3D Model Retrieval
Chen, D., Tian, X., Shen, Y., et al. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.00669 · Cited by 1,432 · see citing works
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R. G., Lévy, R., & Vasconcelos, N. M. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1873987 · Cited by 1,358 · see citing works
He, S., & Lau, R. W. H. (2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,236 · see citing works
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779 · see citing works
On Deep Multi-View Representation Learning
Wang, W., Livescu, K., & Bilmes, J. (2015) · Cited by 751 · see citing works
Generalized Multiview Analysis: A discriminative latent space
Sharma, A., Kumar, A., Daumé, H., & Jacobs, D. W. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247923 · Cited by 737 · see citing works
An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation
Lau, J. H., & Baldwin, T. (2016) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w16-1609 · Cited by 672 · see citing works
Multi-View Discriminant Analysis
Kan, M., Shan, S., Zhang, H., Lao, S., & Chen, X. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2435740 · Cited by 574 · see citing works
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654902 · Cited by 572 · see citing works
Deep Cross-Modal Projection Learning for Image-Text Matching
Zhang, Y., & Lu, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_42 · Cited by 513 · see citing works
Deep Supervised Cross-Modal Retrieval
Zhen, L., Hu, P., Wang, X., & Peng, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01064 · Cited by 478 · see citing works
Self-citationA kernel method for canonical correlation analysisPreprint
Akaho, S. (2006) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0609071 · Cited by 459 · see citing works
Triplet-Based Deep Hashing Network for Cross-Modal Retrieval
Deng, C. D., Chen, Z., Liu, X., Gao, X., & Tao, D. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2821921 · Cited by 402 · see citing works
Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch
Sharma, A., & Jacobs, D. W. (2011) · doi:10.1109/cvpr.2011.5995350 · Cited by 384 · see citing works
Structured AutoEncoders for Subspace Clustering
Peng, X., Feng, J., Xiao, S., Yau, W., Zhou, J. T., & Yang, S. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2848470 · Cited by 345 · see citing works
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
Peng, Y., Huang, X., & Zhao, Y. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2705068 · Cited by 342 · see citing works
Learning Cross-Media Joint Representation With Sparse and Semisupervised Regularization
Zhai, X., Peng, Y., & Xiao, J. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2276704 · Cited by 312 · see citing works
Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing data
Nielsen, A. A. (2002) · IEEE Transactions on Image Processing · doi:10.1109/83.988962 · Cited by 300 · see citing works
Peng, Y., & Qi, J. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3284750 · Cited by 290 · see citing works
Multi-label Cross-Modal Retrieval
Ranjan, V., Rasiwasia, N., & Jawahar, C. V. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.466 · Cited by 244 · see citing works
A Comprehensive Survey on Cross-modal RetrievalPreprint
Wang, K., Yin, Q., Wang, W., Wu, S., & Wang, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06215 · Cited by 230 · see citing works
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228 · see citing works
Multi-view Discriminant Analysis
Kan, M., Shan, S., Zhang, H., Lao, S., & Chen, X. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_58 · Cited by 199 · see citing works
Multi-view Deep Network for Cross-View Classification
Kan, M., Shan, S., & Chen, X. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.524 · Cited by 189 · see citing works
Deep subspace clustering with sparsity prior
Peng, X., Xiao, S., Feng, J., et al. (2016) · International Joint Conference on Artificial Intelligence · Cited by 184 · see citing works
Supervised autoencoders: Improving generalization performance with unsupervised regularizers
Le, L., Patterson, A., & White, M. (2018) · neural information processing systems · Cited by 173 · see citing works
Cross-media shared representation by hierarchical learning with multiple deep networks
Peng, Y., Huang, X., & Qi, J. (2016) · International Joint Conference on Artificial Intelligence · Cited by 168 · see citing works
Semi-Supervised Cross-Media Feature Learning With Unified Patch Graph Regularization
Peng, Y., Zhai, X., Zhao, Y., & Huang, X. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2400779 · Cited by 153 · see citing works
Generalized Semi-supervised and Structured Subspace Learning for Cross-Modal Retrieval
Zhang, L., Ma, B., Li, G., et al. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2723841 · Cited by 139 · see citing works
Deep Coupled Metric Learning for Cross-Modal Matching
Liong, V. E., Lu, J., Tan, Y., et al. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2646180 · Cited by 120 · see citing works
Transfer Hashing: From Shallow to Deep
Zhou, J. T., Zhao, H., Peng, X., Fang, M., Qin, Z., & Goh, R. S. M. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2827036 · Cited by 108 · see citing works
Multi-View Linear Discriminant Analysis Network
Hu, P., Peng, D., Sang, Y., et al. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2913511 · Cited by 106 · see citing works
Self-citationCross-modal Bidirectional Translation via Reinforcement Learning
Qi, J., & Peng, Y. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/365 · Cited by 41 · see citing works
Local feature based multi-view discriminant analysis
Hu, P., Peng, D., Guo, J., et al. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.02.008 · Cited by 26 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“…of multimodal learning still confronts ongoing challenges, such as refining the architectures to handle diverse data types more effectively [269], [270], developing comprehensive datasets that accurately represent multifaceted information [269], [271], and establishing benchmarks for evaluating…”
BackgroundSemi-supervised multimodal representation learning through a global workspace
Devillers, B., Maytié, L., & VanRullen, R. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3416701 · Cited by 5 · see citing works
- Authors
- Hu, P., Zhen, L., Peng, D., & Liu, P.
- Affiliations
- Sichuan University (CN)
- Agency for Science, Technology and Research (SG)
- Venue
- Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2019-07-18
- Pages
- 635-644
- OpenAlex
- W2954672622
- Cited by
- 136
- References
- 46
- Citations by year
- 2026: 4 · 2025: 25 · 2024: 30 · 2023: 20 · 2022: 19 · 2021: 24 · 2020: 12 · 2019: 2
- FWCI
- 4.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- National Key Research and Development Project of China (2017YFB1002201)
- National Natural Science Foundation of China (61836006, U1831121)
- SCU-LuZhou Sciences and Technology Coorperation Program (2017CDLZ-G25)
- Sichuan Science and Technology Planning Projects (18PTDJ0085, 2018GZDZX0030, 2019YFH0075)
- Graduate Student's Research and Innovation Fund of Sichuan University (2018YJSY010)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- cross-modal retrieval, multimodal learning, common subspace, scalable learning, intra-class variation, unsupervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
