Multi-pathway generative adversarial hashing for unsupervised cross-modal retrieval
Abstract
Cross-modal hashing aims to map heterogeneous cross-modal data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Unsupervised cross-modal hashing is more flexible and applicable than supervised methods, since no intensive labeling work is involved. However, existing unsupervised methods learn the hashing functions by preserving inter- and intra-correlations while ignoring the underlying manifold structure across different modalities, which is extremely helpful in capturing the meaningful nearest neighbors of different modalities for cross-modal retrieval. Furthermore, existing works mainly focus on pairwise relation modeling while ignoring the correlations within multiple modalities. To address the above-mentioned problems, in this paper, we propose a multi-pathwaygenerativeadversarialhashing approach for unsupervised cross-modal retrieval, which makes full use of a generative adversarial network's ability for unsupervised representation learning to exploit the underlying manifold structure of cross-modal data. The main contributions can be summarized as follows: First, we propose a multi-pathwaygenerativeadversarialnetwork to model cross-modal hashing in an unsupervised fashion. In the proposed network, given the data of one modality, the generative model tries to fit the distribution over the manifold structure and selects informative data of other modalities to challenge the discriminative model. The discriminative model learns to distinguish the generated data and the true positive data sampled from the correlation graph to achieve better retrieval accuracy. These two models are trained in an adversarial way to improve each other and promote hashing function learning. Second, we propose a correlation graph-based approach to capture the underlying manifold structure across different modalities so that data of different modalities but within the same manifold can have a smaller Hamming distance to promote retrieval accuracy. Extensive experiments compared with state-of-the-art methods on three widely used datasets verify the effectiveness of our proposed approach.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reformer employed locality-sensitive hashing attention to reduce memory requirements for attention computation.
“Reformer [195] employed locality-sensitive hashing (LSH Attention) to reduce memory requirements for attention computation, while Longformer [196] and Linformer [197] utilized sparse attention and low-rank projections, respectively, to decrease computational complexity, making Transformer more efficient for handling long-text sequences.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [195]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · see citing works
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,498 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2× · see citing works
“4096 dimensional deep features extracted from 19-layer VGGNet, videos by 4096 dimensional deep features extracted from C3D model [58] pre-trained on Sports1M [59],”
MethodCanonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,297 · see citing works
(1999) · Cited by 3,186 · see citing works
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works
Weiss, Y., Torralba, A. B., & Fergus, R. (2008) · neural information processing systems · Cited by 2,389 · see citing works
The MIR flickr retrieval evaluation
Huiskes, M. J., & Lew, M. S. (2008) · doi:10.1145/1460096.1460104 · Cited by 1,866 · see citing works
On Visual Similarity Based 3D Model Retrieval
Chen, D., Tian, X., Shen, Y., et al. (2003) · Computer Graphics Forum · doi:10.1111/1467-8659.00669 · Cited by 1,432 · see citing works
Supervised hashing with kernels
Liu, W., Wang, J., Ji, R., Jiang, Y., & Chang, S. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247912 · Cited by 1,418 · see citing works
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,266 · see citing works
Perceptual Generative Adversarial Networks for Small Object Detection
Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.211 · Cited by 953 · cited 1× · see citing works
“Besides image synthesis, GAN is also applied to video prediction [52] and object detection [53].”
MethodJiang, Q.-Y., & Li, W.-J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.348 · Cited by 913 · see citing works
Simultaneous feature learning and hash coding with deep neural networks
Lai, H., Pan, Y., Liu, Y., & Yan, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298947 · Cited by 884 · see citing works
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779 · see citing works
Large-Scale Supervised Multimodal Hashing with Semantic Correlation Maximization
Zhang, D., & Li, W.-J. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8995 · Cited by 647 · see citing works
Inter-media hashing for large-scale retrieval from heterogeneous data sources
Song, J., Yang, Y., Yang, Y., Huang, Z. H., & Shen, H. T. (2013) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2463676.2465274 · Cited by 631 · see citing works
Semi-supervised hashing for scalable image retrieval
Wang, J., Kumar, S., & Chang, S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539994 · Cited by 623 · see citing works
Learning a Distance Metric from Relative Comparisons
Schultz, M., & Joachims, T. (2003) · Cited by 618 · see citing works
Wang, J., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., & Zhang, D. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080786 · Cited by 569 · see citing works
Semantics-preserving hashing for cross-view retrieval
Lin, Z., Ding, G., Hu, M., & Wang, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299011 · Cited by 563 · see citing works
IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models
Wang, Y.-X., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., & Zhang, D. (2017) · BIROn (Birkbeck, University of London) · doi:10.48550/arxiv.1705.10513 · Cited by 526 · see citing works
Learning to Hash for Indexing Big Data—A Survey
Wang, J., Liu, W., Kumar, S., & Chang, S. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2487976 · Cited by 524 · see citing works
Learning Hash Functions for Cross-View Similarity Search
Kumar, S., & Udupa, R. (2012) · Cited by 490 · see citing works
Latent semantic sparse hashing for cross-modal similarity search
Zhou, J., Ding, G., & Guo, Y. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609610 · Cited by 485 · see citing works
Data fusion through cross-modality metric learning using similarity-sensitive hashing
Bronstein, M. M., Bronstein, A. M., Michel, F., & Nikos Paragios, A. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539928 · Cited by 461 · see citing works
Triplet-Based Deep Hashing Network for Cross-Modal Retrieval
Deng, C. D., Chen, Z., Liu, X., Gao, X., & Tao, D. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2821921 · Cited by 402 · see citing works
Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval
Yang, E., Deng, C. D., Liu, W., Liu, X., Tao, D., & Gao, X. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10719 · Cited by 344 · see citing works
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
Peng, Y., Huang, X., & Zhao, Y. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2705068 · Cited by 342 · see citing works
Self-citationDeep Visual-Semantic Hashing for Cross-Modal Retrieval
Cao, Y., Long, M., Wang, J., Yang, Q., & Yu, P. S. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939812 · Cited by 259 · see citing works
Unsupervised Generative Adversarial Cross-Modal Hashing
Zhang, J., Peng, Y., & Yuan, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11263 · Cited by 224 · see citing works
Self-citationLarge-Scale Cross-Modality Search via Collective Matrix Factorization Hashing
Ding, G., Guo, Y., Zhou, J., & Gao, Y. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2607421 · Cited by 218 · see citing works
Co-Regularized Hashing for Multimodal Data
Zhen, Y., & Yeung, D. (2012) · Cited by 190 · see citing works
Semantic Structure-based Unsupervised Deep Hashing
Yang, E., Deng, C. D., Liu, T., Liu, W., & Tao, D. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/148 · Cited by 173 · see citing works
Jaudio: An Feature Extraction Library.
McEnnis, D., McKay, C., Fujinaga, I., & Depalle, P. (2005) · doi:10.5281/zenodo.1416647 · Cited by 172 · see citing works
Semantic topic multimodal hashing for cross-media retrieval
Wang, D., Gao, X., Xiu, W., et al. (2015) · International Conference on Artificial Intelligence · Cited by 166 · see citing works
Collective Reconstructive Embeddings for Cross-Modal Hashing
Hu, M., Yang, Y., Shen, F., et al. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2890144 · Cited by 161 · see citing works
Shared Predictive Cross-Modal Deep Quantization
Yang, E., Deng, C. D., Li, C., et al. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2793863 · Cited by 157 · see citing works
SSDH: Semi-Supervised Deep Hashing for Large Scale Image Retrieval
Zhang, J., & Peng, Y. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2771332 · Cited by 155 · see citing works
Self-citationComposite Correlation Quantization for Efficient Multimodal Retrieval
Long, M., Cao, Y., Wang, J., & Yu, P. S. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2911493 · Cited by 146 · see citing works
Quantized correlation hashing for fast cross-modal search
Wu, T., Yang, Q., Zheng, W., et al. (2015) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 141 · see citing works
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Y., Xiang, S., Kang, C., Wang, J., & Pan, C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2558463 · Cited by 135 · see citing works
Rastegari, M., Choi, J., Fakhraei, S., et al. (2013) · Cited by 106 · see citing works
Hashing with Angular Reconstructive Embeddings
Hu, M., Yang, Y., Shen, F., et al. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2749147 · Cited by 106 · see citing works
Cross-media analysis and reasoning: advances and directions
Peng, Y., Zhu, W., Zhao, Y., Xu, C., Huang, Q., Lu, H., Zheng, Q., Huang, T., & Gao, W. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601787 · Cited by 104 · see citing works
Self-citationDiscriminative Dictionary Learning With Common Label Alignment for Cross-Modal Retrieval
Deng, C. D., Tang, X., Yan, J., Liu, W., & Gao, X. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2508146 · Cited by 99 · see citing works
Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep Structure
Wang, D., Cui, P., Ou, M., & Zhu, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2455415 · Cited by 98 · see citing works
Correlation Autoencoder Hashing for Supervised Cross-Modal Search
Cao, Y., Long, M., Wang, J., & Zhu, H. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912000 · Cited by 96 · see citing works
Locally Linear Hashing for Extracting Non-linear Manifolds
Irie, G., Li, Z., Wu, X.-M., & Chang, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.272 · Cited by 94 · see citing works
Yang, Y., Duan, Y., Wang, X., et al. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2842190 · Cited by 52 · see citing works
Iterative Multi-View Hashing for Cross Media Indexing
Hu, Y., Jin, Z., Ren, H., et al. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654906 · Cited by 46 · see citing works
Cross-Media Hashing with Neural Networks
Zhuang, Y., Zhou, Y., Wang, W., et al. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2655059 · Cited by 35 · see citing works
Scalable heterogeneous translated hashing
Wei, Y., Song, Y., Zhen, Y., Liu, B., & Yang, Q. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623688 · Cited by 31 · see citing works
TUCH: Turning Cross-view Hashing into Single-view Hashing via Generative Adversarial Nets
Zhao, X. B., Ding, G., Guo, Y., et al. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/491 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“The modality combinations and task types presented here directly impact model performance and generalization, which are further analyzed in the following section.”
- Authors
- Zhang, J., & Peng, Y.
- Affiliations
- Peking University (CN)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2020-01
- Volume
- 22
- Issue
- 1
- Pages
- 174-187
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2953037339
- Cited by
- 103
- References
- 60
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 13 · 2023: 14 · 2022: 24 · 2021: 17 · 2020: 15 · 2019: 1
- FWCI
- 6.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Funders
- National Natural Science Foundation of China (61771025)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- cross-modal hashing, unsupervised hashing, cross-modal retrieval, correlation graph, manifold structure, Hamming space, adversarial learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
