Progressive Modality Cooperation for Multi-Modality Domain Adaptation
Open accessThe DOI matches its Crossref record; 85 of the 90 references it cites were found in a registry or an index.
Abstract
In this work, we propose a new generic multi-modality domain adaptation framework called Progressive Modality Cooperation (PMC) to transfer the knowledge learned from the source domain to the target domain by exploiting multiple modality clues (e.g., RGB and depth) under the multi-modality domain adaptation (MMDA) and the more general multi-modality domain adaptation using privileged information (MMDA-PI) settings. Under the MMDA setting, the samples in both domains have all the modalities. Through effective collaboration among multiple modalities, the two newly proposed modules in our PMC can select the reliable pseudo-labeled target samples, which captures the modality-specific information and modality-integrated information, respectively. Under the MMDA-PI setting, some modalities are missing in the target domain. Hence, to better exploit the multi-modality data in the source domain, we further propose the PMC with privileged information (PMC-PI) method by proposing a new multi-modality data generation (MMG) network. MMG generates the missing modalities in the target domain based on the source domain data by considering both domain distribution mismatch and semantics preservation, which are respectively achieved by using adversarial learning and conditioning on weighted pseudo semantic class labels. Extensive experiments on three image datasets and eight video datasets for various multi-modality cross-domain visual recognition tasks under both MMDA and MMDA-PI settings clearly demonstrate the effectiveness of our proposed PMC framework.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Research works printed but not linked: 3
- zheng, 2018, T2Net: Synthetic-to-realistic translation for solving single-image depth estimation tasks, Proc ECCV, 767
- 10.1109/iccvw.2019.00461 · doi:10.1109/iccvw.2019.00461
- jamal, 2018, Deep domain adaptation in action space, Proc BMVC, 264
Cited by
Deep Multimodal Learning with Missing Modality: A Survey
Wu, R., Hu, W., Chen, H., & Carneiro, G. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2409.07825 · Cited by 267
Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval
Fang, X., Liu, D., Zhou, P., & Hu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3222965 · Cited by 126
A systematic literature review on incomplete multimodal learning: techniques and challenges
Zhan, Y., Yang, R., You, J., Huang, M., Liu, W., & Liu, X. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2467083 · Cited by 72
Interact before Align: Leveraging Cross-Modal Knowledge for Domain Adaptive Action Recognition
Yang, L., Huang, Y., Sugano, Y., & Sato, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01431 · Cited by 59
Dong, H., Liu, M. H., Zhou, K., Chatzi, E., Kannala, J., Stachniss, C., & Fink, O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.18592 · Cited by 41
Dong, H., Liu, M. H., Zhou, K., Chatzi, E., Kannala, J., Stachniss, C., & Fink, O. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3651319 · Cited by 37
Capliez, E., Ienco, D., Gaetano, R., Baghdadi, N. N., Salah, A. H., Goff, M. L., & Chouteau, F. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3297077 · Cited by 32
Semantic Distribution-aware Contrastive Adaptation for Semantic SegmentationPreprint
Li, S., Xie, B., Zang, B., Liu, C. H., Cheng, X., Yang, R., & Wang, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05013 · Cited by 30
OSAN: A One-Stage Alignment Network to Unify Multimodal Alignment and Unsupervised Domain Adaptation
Liu, Y., Qiao, L., Lu, C., Yin, D., Chen, L., Peng, H., & Ren, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00346 · Cited by 25
Informative Data Selection With Uncertainty for Multimodal Object Detection
Zhang, X., Li, Z., Zou, Z., Gao, X., Xiong, Y., Jin, D., Li, J., & Liu, H. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270159 · Cited by 23
Cross-Dataset Point Cloud Recognition Using Deep-Shallow Domain Adaptation Network
Wang, F., Li, W., & Xu, D. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3092818 · Cited by 22
Self-citationA Structure-Preserving and Illumination-Consistent Cycle Framework for Image Harmonization
Cai, X., Shi, Q., Gao, Y., Li, S., Hua, W., & Xie, T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3260620 · Cited by 19
A survey of deep domain adaptation based on label set classification
Fan, M., Cai, Z., Zhang, T., & Wang, B. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12630-8 · Cited by 15
Su, R., Xu, D., Zhou, L., & Ouyang, W. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3089355 · Cited by 15
Self-citationEskandar, G., Marsden, R. A., Pandiyan, P., Döbler, M., Guirguis, K., & Yang, B. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982109 · Cited by 13
Federated Transfer Learning with Multimodal DataPreprint
Sun, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.03137 · Cited by 11
Multimodal Learning Under Imperfect Data Conditions: A SurveyPreprint
Liaqat, M. I., Abbas, Q., Nawaz, S., Zaheer, Z., Moscati, M., Hou, Y., Khan, M. H., Khan, S. H., André, E., & Schedl, M. (2025) · doi:10.36227/techrxiv.176410566.65375877/v1 · Cited by 11
Dang, M., Liu, B., Chen, Y., & Zhang, Z. (2026) · Psychoradiology · doi:10.1093/psyrad/kkag007 · Cited by 10
Xing, J., & Zhang, J. (2025) · Bioengineering · doi:10.3390/bioengineering12080871 · Cited by 9
Self-citationMargin-Based Modal Adaptive Learning for Visible-Infrared Person Re-Identification
Zhao, Q., Wu, H., & Zhu, J. (2023) · Sensors · doi:10.3390/s23031426 · Cited by 7
Qu, J., Zhao, J., Dong, W., He, J., Li, Z., & Li, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107633 · Cited by 6
Class structure‐aware adversarial loss for cross‐domain human action recognition
Chen, W., Liu, L., Lin, G., Chen, Y., & Wang, J. (2021) · IET Image Processing · doi:10.1049/ipr2.12309 · Cited by 4
FFEvent: Fast fourier-based knowledge transfer for event cameras
Lin, Y., Zhang, J., Yu, S., Xiao, J., & Lu, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130055 · Cited by 4
Self-citationLearning from Multiple Datasets for Recognizing Human Actions
Kumbhare, S. T., & Chowdhury, A. S. (2022) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3571600.3571605 · Cited by 3
Lu, Y., Singh, G., Saha, S., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00413 · Cited by 3
Event USKT : U-State Space Model in Knowledge Transfer for Event CamerasPreprint
Lin, Y., Zhang, J., Li, S., Xiao, J., Ding, X., Wu, W., & Lu, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.15276 · Cited by 3
Self-citationRoad Disease Detection based on Latent Domain Background Feature Separation and SuppressionPreprint
Zheng, J., & Ren, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.07616 · Cited by 3
Bimodal masked language modeling for bulk RNA-seq and DNA methylation representation learningPreprint
Gélard, M., Benkirane, H., Pierrot, T., Richard, G., & Cournède, P.-H. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.25.661237 · Cited by 1
Weakly Correlated Multimodal Domain Adaptation for Pattern Classification
Wang, S., Liu, Z., Zhang, Z., & Bennamoun, M. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3524976 · Cited by 1
A Universal Model for Cross Modality Mapping by Relational ReasoningPreprint
Li, Z., Lang, C., Liang, L., Wang, T., Feng, S., Wu, J., & Li, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13360 · Cited by 1
2021 Index IEEE Transactions on Image Processing Vol. 30
(2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3142569 · Cited by 1
Incomplete Modalities Restoration via Hierarchical Adaptation for Robust Multimodal Segmentation
Sun, Y., Dong, W., Wu, P., Feng, M., Huang, T., Li, X., & Shi, G. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3642612
Human Activity Understanding by Multilateral Relation Mining
楊, 麗. (2023) · Institutional Repositories DataBase (IRDB) · doi:10.15083/0002011980
Hsu, M., Hsu, F.-Y., Lin, Y., Chien, K.-H., Chen, J.-R., Su, C.-H., Ou, Y.-C., Hsu, C.-T., & Huang, P.-K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.23475
Metadata
- Authors
- Zhang, W., Xu, D., Zhang, J., & Ouyang, W.
- Affiliations
- The University of Sydney (AU)
- Beihang University (CN)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2021
- Volume
- 30
- Pages
- 3293-3306
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W3124928637
- Cited by
- 25
- References
- 90
- Citations by year
- 2026: 1 · 2025: 7 · 2023: 6 · 2022: 5 · 2021: 6
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Australian Research Council (ARC) Future Fellowship (FT180100116)
- ARC (DP200103223)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- privileged information, missing modalities, pseudo-labeling, adversarial learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.