Learning Aligned Cross-Modal Representations from Weakly Aligned Data
Abstract
People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize cross-modal scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for crossmodal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pre-training with a diverse set of tasks improves the performance on downstream tasks.
“Instead, pre-training with a diverse set of tasks [11,83] has been observed to improve the performance on downstream tasks [31,80] and potentially learn a better representation.”
From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [11]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1× · see citing works
“Our approach is a form of curriculum learning [4].”
Learning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505 · see citing works
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · see citing works
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmák, S., & Shawe-Taylor, J. S. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,297 · see citing works
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283 · see citing works
Self-citationOne-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029 · see citing works
Adapting visual category models to new domains
Saenko, K., Kulis, B., Fritz, M., & Darrell, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_16 · Cited by 2,847 · see citing works
(2011) · Cited by 2,752 · see citing works
(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534 · see citing works
Self-citationTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
Self-citationDeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Signature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Stefan Harmeling, B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206594 · Cited by 1,563 · see citing works
Unifying Visual-Semantic Embeddings with Multimodal Neural Language ModelsPreprint
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,386 · see citing works
A new approach to cross-modal multimedia retrieval
Rasiwasia, N., Pereira, J. C., Coviello, E., Doyle, G., Lanckriet, G. R. G., Lévy, R., & Vasconcelos, N. M. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1873987 · Cited by 1,358 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · see citing works
Self-citationSimultaneous Deep Transfer Across Domains and Tasks
Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.463 · Cited by 1,203 · see citing works
Domain adaptation for object recognition: An unsupervised approach
Gopalan, R., Li, R., & Chellappa, R. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126344 · Cited by 1,059 · see citing works
Zero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · see citing works
Zero-Shot Learning by Convex Combination of Semantic EmbeddingsPreprint
Norouzi, M., Mikolov, T., Samy Bengio, A., Singer, Y., Shlens, J., Frome, A., Corrado, G. S., & Dean, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5650 · Cited by 890 · see citing works
Object Detectors Emerge in Deep Scene CNNsPreprint
Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6856 · Cited by 823 · see citing works
Self-citationZero-Shot Learning with Semantic Output Codes
Palatucci, M., Pomerleau, D., Hinton, G. E., & Mitchell, T. M. (2018) · Figshare · doi:10.1184/r1/6476456 · Cited by 752 · see citing works
Geometric context from a single image
Hoiem, D., Efros, A. A., & Hebert, M. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.107 · Cited by 666 · see citing works
Undoing the Damage of Dataset Bias
Khosla, A., Zhou, T., Malisiewicz, T., Efros, A. A., & Torralba, A. B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_12 · Cited by 482 · see citing works
Self-citationSketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors
Eitz, M., Hildebrand, K., Boubekeur, T., & Alexa, M. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.266 · Cited by 411 · see citing works
Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions
Ba, J. L., Swersky, K., Fidler, S., & Salakhutdinov, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.483 · Cited by 401 · see citing works
Owens, A., Isola, P., McDermott, J. H., Torralba, A. B., Adelson, E. H., & Freeman, W. T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.264 · Cited by 352 · see citing works
Self-citationContextual models for object detection using boosted random fields
Torralba, A. B., Murphy, K. P., & Freeman, W. T. (2004) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 303 · see citing works
Self-citationHOGgles: Visualizing Object Detection Features
Vondrick, C., Khosla, A., Malisiewicz, T., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.8 · Cited by 282 · see citing works
Self-citationSketch-based 3D shape retrieval using Convolutional Neural Networks
Wang, F., Kang, L., & Li, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298797 · Cited by 275 · see citing works
Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions
Elhoseiny, M., Saleh, B., & Elgammal, A. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.321 · Cited by 263 · see citing works
Multi-label Cross-Modal Retrieval
Ranjan, V., Rasiwasia, N., & Jawahar, C. V. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.466 · Cited by 244 · see citing works
Object Classification from a Single Example Utilizing Class Relevance Metrics
Fink, M. (2004) · Cited by 206 · see citing works
Cross-Generalization: Learning Novel Classes from a Single Example by Feature Replacement
Bart, E., & Ullman, S. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.117 · Cited by 202 · see citing works
Bringing Semantics into Focus Using Visual Abstraction
Zitnick, C. L., & Parikh, D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.387 · Cited by 197 · see citing works
Learning cross-modality similarity for multinomial data
Jia, Y., Salzmann, M., & Darrell, T. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126524 · Cited by 180 · see citing works
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 156 · see citing works
Cluster Canonical Correlation Analysis
Rasiwasia, N., Mahajan, D. K., Mahadevan, V., & Aggarwal, G. (2014) · Cited by 152 · see citing works
Inverting Convolutional Networks with Convolutional Networks.Preprint
Dosovitskiy, A., & Brox, T. (2015) · arXiv (Cornell University) · Cited by 75 · see citing works
Predicting Object Dynamics in Scenes
Fouhey, D. F., & Zitnick, C. L. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.260 · Cited by 74 · see citing works
What Makes a Good Detector? – Structured Priors for Learning from Few Examples
Gao, T., Stark, M. M., & Koller, D. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_26 · Cited by 25 · see citing works
Learning visual biases from human imaginationPreprint
Vondrick, C., Pirsiavash, H., Oliva, A., & Torralba, A. B. (2014) · arXiv (Cornell University) · doi:10.13016/m2w3st-cf3r · Cited by 14 · see citing works
Self-citationPart level transfer regularization for enhancing exemplar SVMs
Aytar, Y., & Zisserman, A. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.04.004 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
3 of the 51 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
Jégou, S., Drozdzal, M., Vázquez, D., Romero, A., & Bengio, Y. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.156 · Cited by 1,762 · see citing works
Through-Wall Human Pose Estimation Using Radio Signals
Zhao, M., Li, T., Alsheikh, M. A., Tian, Y., Zhao, H., Torralba, A. B., & Katabi, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00768 · Cited by 676 · see citing works
Self-citationSoundNet: Learning Sound Representations from Unlabeled VideoPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 658 · see citing works
Self-citationCross-Domain Weakly-Supervised Object Detection Through Progressive Domain Adaptation
Inoue, N., Furuta, R., Yamasaki, T., & Aizawa, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00525 · Cited by 616 · see citing works
Dual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3383184 · Cited by 622 · see citing works
Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Salvador, A., Hynes, N., Aytar, Y., Marín, J., Ofli, F., Weber, I. G., & Torralba, A. B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.327 · Cited by 416 · see citing works
Self-citationHandling incomplete heterogeneous data using VAEs
Nazábal, A., Olmos, P. M., Zoubin Ghahramani, A., & Valera, I. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107501 · Cited by 369 · see citing works
Multimodal MR Synthesis via Modality-Invariant Latent Representation
Chartsias, A., Joyce, T., Giuffrida, M. V., & Tsaftaris, S. A. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2764326 · Cited by 343 · see citing works
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
Peng, Y., Huang, X., & Zhao, Y. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2705068 · Cited by 342 · see citing works
Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Marín, J., Biswas, A., Ofli, F., Hynes, N., Salvador, A., Aytar, Y., Weber, I. G., & Torralba, A. B. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2927476 · Cited by 350 · see citing works
Self-citationContext Autoencoder for Self-supervised Representation Learning
Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., & Wang, J. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01852-4 · Cited by 330 · see citing works
Domain Adaptation for Visual Applications: A Comprehensive SurveyPreprint
Csurka, G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.05374 · Cited by 324 · see citing works
Multisource Transfer Learning With Convolutional Neural Networks for Lung Pattern Analysis
Christodoulidis, S., Anthimopoulos, M., Ebner, L., Christe, A., & Mougiakakou, S. G. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636929 · Cited by 296 · see citing works
Deep Partial Multi-View Learning
Zhang, C., Cui, Y., Han, Z., Zhou, J. T., Fu, H., & Hu, Q. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3037734 · Cited by 282 · see citing works
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773 · Cited by 272 · see citing works
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works
Colloquium : Machine learning in nuclear physics
Boehnlein, A., Diefenthaler, M., Sato, N., Schram, M., Ziegler, V., Fanelli, C., Hjorth-Jensen, M., Horn, T., Kuchera, M., Lee, D., et al. (2022) · Reviews of Modern Physics · doi:10.1103/revmodphys.94.031003 · Cited by 263 · see citing works
Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227 · see citing works
Bakkouri, I., & Afdel, K. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-07988-1 · Cited by 208 · see citing works
Omnivore: A Single Model for Many Visual Modalities
Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., & Misra, I. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01563 · Cited by 212 · see citing works
Tan, B., Song, Y., Zhong, E., & Yang, Q. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783295 · Cited by 198 · see citing works
RGB-IR Person Re-identification by Cross-Modality Similarity Preservation
Wu, A., Zheng, W., Gong, S., & Lai, J. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01290-1 · Cited by 172 · see citing works
Speech2Face: Learning the Face Behind a Voice
Oh, T.-H., Dekel, T., Kim, C., Mosseri, I., Freeman, W. T., Rubinstein, M., & Matusik, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00772 · Cited by 167 · see citing works
Automatic Understanding of Image and Video Advertisements
Hussain, Z., Zhang, M., Zhang, X., Ye, K., Thomas, C., Agha, Z., Ong, N., & Kovashka, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.123 · Cited by 160 · see citing works
Label Efficient Learning of Transferable Representations across Domains and TasksPreprint
Luo, Z., Zou, Y., Hoffman, J., & Fei-Fei, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00123 · Cited by 172 · see citing works
Deep Learning for Free-Hand Sketch: A Survey
Xu, P., Hospedales, T. M., Yin, Q., Song, Y.-Z., Xiang, T., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148853 · Cited by 180 · see citing works
Self-Supervised Moving Vehicle Tracking With Stereo Sound
Gan, C., Zhao, H., Chen, P., Cox, D., & Torralba, A. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00715 · Cited by 150 · see citing works
Self-citationVenkateswara, H., Chakraborty, S., & Panchanathan, S. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2740460 · Cited by 124 · see citing works
A Deep Multi-Modal CNN for Multi-Instance Multi-Label Image Classification
Song, L., Liu, J., Qian, B., Sun, M., Yang, K., Sun, M., & Abbas, S. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2864920 · Cited by 116 · see citing works
CPM-Nets: Cross Partial Multi-View Networks
Zhang, C., Han, Z., Cui, Y., Fu, H., Zhou, J. T., & Hu, Q. (2019) · neural information processing systems · Cited by 114 · see citing works
See, Hear, and Read: Deep Aligned RepresentationsPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00932 · Cited by 117 · see citing works
Self-citationYu, J., Zhang, W., Lu, Y., Qin, Z., Hu, Y., Tan, J., & Wu, Q. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972830 · Cited by 96 · see citing works
2022 Review of Data-Driven Plasma Science
Anirudh, R., Archibald, R. K., Asif, M. S., Becker, M. M., Benkadda, S., Bremer, P., Budé, R. H. S., Chang, C. S., Chen, L., Churchill, R. M., et al. (2023) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2023.3268170 · Cited by 97 · see citing works
Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition
Zhang, J., Li, W., Ogunbona, P., & Xu, D. (2019) · ACM Computing Surveys · doi:10.1145/3291124 · Cited by 127 · see citing works
Transfer Learning in Natural Language Processing
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.020 · Cited by 94 · see citing works
Zhuang, P., Wang, Y., & Qiao, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240616 · Cited by 84 · see citing works
OmniMAE: Single Model Masked Pretraining on Images and Videos
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01003 · Cited by 78 · see citing works
Cross-Modal Learning for Domain Adaptation in 3D Semantic Segmentation
Jaritz, M., Vu, T.-H., de Charette, R., Wirbel, É., & Pérez, P. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3159589 · Cited by 75 · see citing works
Attention-Based Autism Spectrum Disorder Screening With Privileged Modality
Chen, S., & Zhao, Q. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00127 · Cited by 72 · see citing works
Visually Grounded Neural Syntax Acquisition
Shi, H., Mao, J., Gimpel, K., & Livescu, K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1180 · Cited by 68 · see citing works
Significance of Softmax-based Features in Comparison to Distance Metric Learning-based Features
Horiguchi, S., Ikami, D., & Aizawa, K. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911075 · Cited by 65 · see citing works
Sensor modality fusion with CNNs for UGV autonomous driving in indoor environments
Patel, N., Choromanska, A., Krishnamurthy, P., & Khorrami, F. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8205958 · Cited by 59 · see citing works
Shen, Y., Pokharel, R., Nizolek, T. J., Kumar, A., & Lookman, T. (2019) · Acta Materialia · doi:10.1016/j.actamat.2019.03.026 · Cited by 55 · see citing works
Csurka, G. (2020) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc51798.2020.00013 · Cited by 66 · see citing works
AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos With Deep Learning
Ghose, S., & Prevost, J. J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3005033 · Cited by 51 · see citing works
Semantic analysis on social networks: A survey
Bayrakdar, S., Yücedağ, İ., Şimşek, M., & Doğru, İ. A. (2020) · International Journal of Communication Systems · doi:10.1002/dac.4424 · Cited by 51 · see citing works
Luo, Y., Yan, S., Li, H., Lai, P., & Zheng, Y. (2021) · Photonics Research · doi:10.1364/prj.415590 · Cited by 48 · see citing works
Face-Voice Matching using Cross-modal Embeddings
Horiguchi, S., Kanda, N., & Nagamatsu, K. (2018) · ACM Multimedia · doi:10.1145/3240508.3240601 · Cited by 59 · see citing works
Deep Memory Network for Cross-Modal Retrieval
Song, G., Wang, D., & Tan, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2877122 · Cited by 44 · see citing works
Rehman, S. U., Tu, S., Huang, Y., & Rehman, O. U. (2018) · IEEE Access · doi:10.1109/access.2018.2878868 · Cited by 44 · see citing works
Social network search based on semantic analysis and learning
Kou, F., Du, J., He, Y., & Ye, L. (2016) · CAAI Transactions on Intelligence Technology · doi:10.1016/j.trit.2016.12.001 · Cited by 43 · see citing works
Recognition From Web Data: A Progressive Filtering Approach
Yang, J., Sun, X., Lai, Y., Zheng, L., & Cheng, M. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2855449 · Cited by 40 · see citing works
Improving Night-Time Pedestrian Retrieval With Distribution Alignment and Contextual Distance
Ye, M., Cheng, Y., Lan, X., & Zhu, H. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2946030 · Cited by 40 · see citing works
SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches
Liu, F., Zou, C., Deng, X., Zuo, R., Lai, Y., Ma, C., Liu, Y., & Wang, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58529-7_42 · Cited by 39 · see citing works
PM-GANs: Discriminative Representation Learning for Action Recognition Using Partial-Modalities
Wang, L., Gao, C., Yang, L., Zhao, Y., Zuo, W., & Meng, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_24 · Cited by 37 · see citing works
Learning Visually-Grounded Semantics from Contrastive Adversarial SamplesPreprint
Shi, H., Mao, J., Xiao, T., Jiang, Y., & Sun, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10348 · Cited by 32 · see citing works
Explainability of vision-based autonomous driving systems: Review and challenges.Preprint
Zablocki, É., Ben-younes, H., Pérez, P., & Cord, M. (2021) · arXiv (Cornell University) · Cited by 32 · see citing works
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text
Chowdhury, P. N., Bhunia, A. K., Sain, A., Koley, S., Xiang, T., & Song, Y.-Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01056 · Cited by 30 · see citing works
Audio-Visual Model Distillation Using Acoustic Images
Pérez, A. F., Sanguineti, V., Morerio, P., & Murino, V. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093307 · Cited by 27 · see citing works
Cooperative Learning with Visual AttributesPreprint
Batra, T., & Parikh, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.05512 · Cited by 28 · see citing works
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering
Kim, J., Ma, M., Kim, K., Kim, S.-J., & Yoo, C. D. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852087 · Cited by 25 · see citing works
Scene Graph Reasoning with Prior Visual Relationship for Visual Question AnsweringPreprint
Yang, Z., Qin, Z., Yu, J., & Hu, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.09681 · Cited by 35 · see citing works
Fully Convolutional Pyramidal Networks for Semantic Segmentation
Li, F., Long, Z., He, P., Feng, P., Guo, X., Ren, X., Wei, B., Zhao, M., & Tang, B. (2020) · IEEE Access · doi:10.1109/access.2020.3045280 · Cited by 25 · see citing works
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Koley, S., Bhunia, A. K., Sain, A., Chowdhury, P. N., Xiang, T., & Song, Y.-Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01562 · Cited by 24 · see citing works
Deep Learning for Free-Hand Sketch: A SurveyPreprint
Xu, P., Hospedales, T. M., Yin, Q., Song, Y.-Z., Xiang, T., & Wang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.02600 · Cited by 21 · see citing works
Deep Learning for Free-Hand Sketch: A Survey and A ToolboxPreprint
Xu, P., Hospedales, T. M., Yin, Q., Song, Y.-Z., Xiang, T., & Wang, L. (2020) · arXiv (Cornell University) · Cited by 21 · see citing works
A Review and Computational Analysis of Breast Cancer Using Different Machine Learning Techniques
Nemade, V., Pathak, S. K., Dubey, A. K., & Barhate, D. (2022) · International Journal of Emerging Technology and Advanced Engineering · doi:10.46338/ijetae0322_13 · Cited by 20 · see citing works
Cogni-Net: Cognitive Feature Learning Through Deep Visual Perception
Mukherjee, P., Das, A., Bhunia, A. K., & Roy, P. P. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803717 · Cited by 19 · see citing works
Incorporating Visual Semantics into Sentence Representations within a Grounded Space
Bordes, P., Zablocki, É., Soulier, L., Piwowarski, B., & Gallinari, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1064 · Cited by 19 · see citing works
Chen, S., Zhang, S., Fang, X., Lin, L., Zhao, H., & Yang, Y. (2024) · Nature Communications · doi:10.1038/s41467-024-53116-5 · Cited by 19 · see citing works
Template-Free Try-On Image Synthesis via Semantic-Guided Optimization
Chou, C.-L., Chen, C.-Y., Hsieh, C.-W., Shuai, H.-H., Liu, J., & Cheng, W.-H. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3058379 · Cited by 18 · see citing works
Vijendran, M., Deng, J., Chen, S., Ho, E. S. L., & Shum, H. P. H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11051-3 · Cited by 20 · see citing works
Artistic Object Recognition by Unsupervised Style Adaptation
Thomas, C., & Kovashka, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20893-6_29 · Cited by 18 · see citing works
Exploring Local Detail Perception for Scene Sketch Semantic Segmentation
Ge, C., Sun, H., Song, Y.-Z., Ma, Z., & Liao, J. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3142511 · Cited by 18 · see citing works
FIDDLE: An integrative deep learning framework for functional genomic data inferencePreprint
Eser, U., & Churchman, L. S. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/081380 · Cited by 18 · see citing works
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs
Liu, F., Deng, X., Zou, C., Lai, Y., Chen, K., Zuo, R., Ma, C., Liu, Y., & Wang, H. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3175403 · Cited by 17 · see citing works
Unsupervised Domain Adaptation with Regularized Domain Instance Denoising
Csurka, G., Chidlovskii, B., Clinchant, S., & Michel, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_37 · Cited by 17 · see citing works
Through-Wall Human Pose Estimation by Mutual Information Maximizing Deeply Supervised Nets
Zheng, Z., Pan, J., Zhang, D., Liang, X. Y., Liu, X., & Fang, G. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3294955 · Cited by 16 · see citing works
GuessWhat?! Visual object discovery through multi-modal dialoguePreprint
de Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08481 · Cited by 16 · see citing works
Bakkouri, I., & Afde, K. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08859-w · Cited by 15 · see citing works
Cross-domain Cross-modal Food Transfer
Zhu, B., Ngo, C., & Chen, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413809 · Cited by 14 · see citing works
Sequential Instance Refinement for Cross-Domain Object Detection in Images
Chen, J., Wu, X., Duan, L., & Chen, L. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3066904 · Cited by 14 · see citing works
Multi-modal Learning with Prior Visual Relation Reasoning.Preprint
Yang, Z., Yu, J., Yang, C., Qin, Z., & Hu, Y. (2018) · arXiv (Cornell University) · Cited by 13 · see citing works
Cross-modal learning with prior visual relation knowledge
Yu, J., Zhang, W., Yang, Z., Qin, Z., & Hu, Y. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106150 · Cited by 12 · see citing works
Discrepancy-Based Networks for Unsupervised Domain Adaptation: A Comparative Study
Csurka, G., Baradel, F., Chidlovskii, B., & Clinchant, S. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.312 · Cited by 12 · see citing works
Deep Zero-Shot Learning for Scene Sketch
Xie, Y., Xu, P., & Ma, Z. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803426 · Cited by 11 · see citing works
CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly
Zou, Z., Zhu, X., Zhu, Q., Liu, Y., & Zhu, L. (2024) · IEEE Access · doi:10.1109/access.2024.3370158 · Cited by 11 · see citing works
Multi-Frequency RF Sensor Data Adaptation for Motion Recognition with Multi-Modal Deep Learning
Rahman, M. M., & Gürbüz, S. Z. (2021) · IEEE Radar Conference (RadarConf) · doi:10.1109/radarconf2147009.2021.9455204 · Cited by 10 · see citing works
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.016 · Cited by 10 · see citing works
Adversarial Transfer Learning.Preprint
Wilson, G., & Cook, D. J. (2018) · arXiv (Cornell University) · Cited by 10 · see citing works
Cross-modal retrieval based on shared proxies
Wei, Y., Zheng, L., Qiu, G., & Cai, G. (2024) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00316-2 · Cited by 9 · see citing works
Quantum generative adversarial network with automated noise suppression mechanism based on WGAN-GP
Tian, Y., Tian, C., Fan, Z.-C., Fu, M., & Ma, H. (2025) · EPJ Quantum Technology · doi:10.1140/epjqt/s40507-025-00372-z · Cited by 12 · see citing works
Audio-to-Visual Cross-Modal Generation of Birds
Shim, J. Y., Kim, J., & Kim, J. (2023) · IEEE Access · doi:10.1109/access.2023.3257565 · Cited by 11 · see citing works
Exploiting Convolution Filter Patterns for Transfer Learning
Aygün, M. M., Aytar, Y., & Ekenel, H. K. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.309 · Cited by 9 · see citing works
Self-citationMix and Match Networks: Cross-Modal Alignment for Zero-Pair Image-to-Image Translation
Wang, Y., Herranz, L., & van de Weijer, J. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01340-z · Cited by 9 · see citing works
SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition
Liu, F., Deng, X., Song, J., Lai, Y., Liu, Y., Wang, H., Ma, C., Qin, S., & Wang, H. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3543956 · Cited by 9 · see citing works
Seeing Faces in Things: A Model and Dataset for Pareidolia
Hamilton, M., Stent, S., DuTell, V., Harrington, A., Corbett, J. E., Rosenholtz, R. E., & Freeman, W. T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73650-6_22 · Cited by 8 · see citing works
Forecasting of 3D Whole-Body Human Poses with Grasping Objects
Yan, H., Cui, Q., Xie, J., & Guo, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00170 · Cited by 8 · see citing works
Lee, S., Kim, H.-I., & Ro, Y. M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01028 · Cited by 8 · see citing works
Cross Attention with Transformer for Few-shot Medical Image Segmentation
Niu, Y., Li, Z., & Li, S. (2022) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme56794.2022.00137 · Cited by 7 · see citing works
- Authors
- Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- University of Maryland, Baltimore County (US)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 2940-2949
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2474574787
- Cited by
- 188
- References
- 51
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 9 · 2023: 12 · 2022: 22 · 2021: 17 · 2020: 49 · 2019: 25 · 2018: 17 · 2017: 17 · 2016: 7 · 2015: 1
- FWCI
- 7.64
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- cross-modal transfer, shared representations, cross-modal retrieval, scene classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review