The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
Abstract
It is known that the inconsistent distributions and representations of different modalities, such as image and text, cause the heterogeneity gap, which makes it very challenging to correlate heterogeneous data and measure their similarities. Recently, generative adversarial networks (GANs) have been proposed and have shown their strong ability to model data distribution and learn discriminative representation. It has also been shown that adversarial learning can be fully exploited to learn discriminative common representations for bridging the heterogeneity gap. Inspired by this, we aim to effectively correlate large-scale heterogeneous data of different modalities with the power of GANs to model cross-modal joint distribution. In this article, we propose Cross-modal Generative Adversarial Networks (CM-GANs) with the following contributions. First, a cross-modal GAN architecture is proposed to model joint distribution over the data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both compete with each other to promote cross-modal correlation learning. Second, the cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form the generative model. They not only exploit the cross-modal correlation for learning the common representations but also preserve reconstruction information for capturing the semantic consistency within each modality. Third, a cross-modal adversarial training mechanism is proposed, which uses two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make the generated common representations more discriminative by the adversarial training process. In summary, our proposed CM-GAN approach can use GANs to perform cross-modal common representation learning by which the heterogeneous data can be effectively correlated. Extensive experiments are conducted to verify the performance of CM-GANs on cross-modal retrieval compared with 13 state-of-the-art methods on 4 cross-modal datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
RELATIONS BETWEEN TWO SETS OF VARIATES
HOTELLING, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,263
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,202
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021
(2011) · Cited by 2,752
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00240 · Cited by 2,466
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
Advances in Neural Information Processing Systems 19 (NIPS 2006)
Simonotto, E., Whalley, H. C., Lawrie, S. M., Murray, L. M., McGonigle, D. J., & Storkey, A. (2006) · Cited by 1,651
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · International Conference on Machine Learning · Cited by 1,602
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Costa Pereira, J., Coviello, E., Doyle, G., Lanckriet, G. R. G., Levy, R., & Vasconcelos, N. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1873987 · Cited by 1,334
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
GraphGAN: Graph Representation Learning With Generative Adversarial Nets
Wang, H., Wang, J., Wang, J., Zhao, M., Zhang, W., Zhang, F., Xie, X., & Guo, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11872 · Cited by 653
Generative Image Modeling Using Style and Structure Adversarial Networks
Wang, X., & Gupta, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_20 · Cited by 574
IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models
Wang, J., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., & Zhang, D. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080786 · Cited by 567
Cross-modal Retrieval with Correspondence Autoencoder
Feng, F., Wang, X., & Li, R. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654902 · Cited by 560
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics
Gong, Y., Ke, Q., Isard, M., & Lazebnik, S. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-013-0658-4 · Cited by 540
IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models
Wang, Y.-X., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., & Zhang, D. (2017) · BIROn (Birkbeck, University of London) · doi:10.48550/arxiv.1705.10513 · Cited by 526
Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval
Li, C., Deng, C., Li, N., Liu, W., Gao, X., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00446 · Cited by 511
The International Joint Conference on Artificial Intelligence (IJCAI)
(1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90052-5 · Cited by 480
Learning Discriminative Binary Codes for Large-scale Cross-modal Retrieval
Xu, X., Shen, F., Yang, Y., Shen, H. T., & Li, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2676345 · Cited by 458
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00750 · Cited by 396
Deep correlation for matching images and text
Yan, F., & Mikolajczyk, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298966 · Cited by 395
Cross-Modal Retrieval With CNN Visual Features: A New Baseline
Wei, Y., Zhao, Y., Lu, C., Wei, S., Liu, L., Zhu, Z., & Yan, S. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2519449 · Cited by 370
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
Peng, Y., Huang, X., & Zhao, Y. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2705068 · Cited by 341
Self-citationJoint Feature Selection and Subspace Learning for Cross-Modal Retrieval
Wang, K., He, R., Wang, L., Wang, W., & Tan, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505311 · Cited by 318
Learning Cross-Media Joint Representation With Sparse and Semisupervised Regularization
Zhai, X., Peng, Y., & Xiao, J. (2014) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2276704 · Cited by 294
Self-citationMultimedia content processing through cross-modal association
Li, D., Dimitrova, N., Li, M., & Sethi, I. K. (2003) · Proceedings of the eleventh ACM international conference on Multimedia · doi:10.1145/957013.957143 · Cited by 265
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228
Self-citationMulti-label Cross-Modal Retrieval
Ranjan, V., Rasiwasia, N., & Jawahar, C. V. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.466 · Cited by 227
Learning Consistent Feature Representation for Cross-Modal Multimedia Retrieval
Kang, C., Xiang, S., Liao, S., Xu, C., & Pan, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2390499 · Cited by 222
International Joint Conference on Artificial Intelligence (IJCAI 2017)
Belle, V. (2017) · Cited by 219
Socher, R., & Fei-Fei, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540112 · Cited by 217
The International Conference on Machine Learning (ICML 2011)
Cho, K., Raiko, T., & Ilin, A. (2011) · Cited by 215
Yang, Y., Zhuang, Y.-T., Wu, F., & Pan, Y.-H. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.917359 · Cited by 213
Learning Representations for Multimodal Data with Deep Belief Nets
Srivastava, N., & Salakhutdinov, R. (2012) · Cited by 180
Cross-media shared representation by hierarchical learning with multiple deep networks
Peng, Y., Huang, X., & Qi, J. (2016) · International Joint Conference on Artificial Intelligence · Cited by 178
Self-citationLearning What and Where to Draw
Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02454 · Cited by 176
Deep Multimodal Learning for Affective Analysis and Retrieval
Pang, L., Zhu, S., & Ngo, C. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2482228 · Cited by 166
Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval
Zhai, X., Peng, Y., & Xiao, J. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8464 · Cited by 127
Self-citationDeep multimodal hashing with orthogonal regularization
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · International Conference on Artificial Intelligence · Cited by 113
Cross-media analysis and reasoning: advances and directions
Peng, Y., Zhu, W., Zhao, Y., Xu, C., Huang, Q., Lu, H., Zheng, Q., Huang, T., & Gao, W. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601787 · Cited by 104
Self-citationUnsupervised cross-modal retrieval through adversarial learning
He, L., Xu, X., Lu, H., Yang, Y., Shen, F., & Shen, H. T. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019549 · Cited by 104
Modality-Dependent Cross-Media Retrieval
Wei, Y., Zhao, Y., Zhu, Z., Wei, S., Xiao, Y., Feng, J., & Yan, S. (2016) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2775109 · Cited by 90
Cross-media semantic representation via bi-directional learning to rank
Wu, F. Q., Lu, X., Zhang, Z. M., et al. (2013) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2502081.2502097 · Cited by 82
Deep Fisher discriminant learning for mobile hand gesture recognition
Li, C., Xie, C., Zhang, B., Chen, C., & Han, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.12.023 · Cited by 72
Correspondence Autoencoders for Cross-Modal Retrieval
Feng, F., Wang, X., Li, R., & Ahmad, I. (2015) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/2808205 · Cited by 32
Learning Label Preserving Binary Codes for Multimedia Retrieval
Li, K., Qi, G.-J., & Hua, K. A. (2017) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3152126 · Cited by 23
Enhanced isomorphic semantic representation for cross-media retrieval
Liu, T., Zhao, Y., Wei, S., Wei, Y., & Liao, L. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019356 · Cited by 7
Exploring many-to-one speech-to-text correlation for web-based language learning
Chen, H.-Y., & Li, S. (2007) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/1236471.1236472 · Cited by 4
Cited by
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073
“cross-modal correlations using Canonical Correlation Analysis [3]; 3) Learning a new shared subspace where all the modalities are simultaneously mapped, using techniques such as adversarial learning [35, 37]. Similar to the third category, we also learn common modality-invariant subspaces. However, we do not use adver…”
MethodDeep Supervised Cross-Modal Retrieval
Zhen, L., Hu, P., Wang, X., & Peng, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01064 · Cited by 478
Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480
Multi-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01095 · Cited by 451
Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
Gu, J., Cai, J., Joty, S., Niu, L., & Wang, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00750 · Cited by 396
Metadata
- Authors
- Peng, Y., & Qi, J.
- Venue
- ACM Transactions on Multimedia Computing, Communications, and Applications
- Published
- 2019-02-07
- Volume
- 15
- Issue
- 1
- Pages
- 1-24
- ISSN
- 1551-6857 (ISSN-L)
- 1551-6865
- OpenAlex
- W2964216321
- Cited by
- 290
- References
- 60
- FWCI
- 18.66
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.