The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Visual Prompt Multi-Modal Tracking
Open accessAbstract
Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multi-modal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2008) · Journal of Machine Learning Research · Cited by 33,051
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
(2010) · Cited by 14,134
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Fully-Convolutional Siamese Networks for Object Tracking
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_56 · Cited by 4,546 · cited 2×
“RGB-based tracking, a foundation task of visual object tracking, gains from large-scale benchmarks [9, 13, 27, 36, 39, 45] provided by the community, and many excellent works [2, 3, 5, 7, 22] have spurted out over the past decades.”
BackgroundMethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“Recently, as a new paradigm, prompt learning has emerged and drastically boosted the performance of many down-stream natural language processing (NLP) tasks [20, 30].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Recently, as a new paradigm, prompt learning has emerged and drastically boosted the performance of many down-stream natural language processing (NLP) tasks [20, 30].”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
High Performance Visual Tracking with Siamese Region Proposal Network
Li, B., Yan, J., Wu, W., Zhu, Z., & Hu, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00935 · Cited by 3,023 · cited 2×
“RGB-based tracking, a foundation task of visual object tracking, gains from large-scale benchmarks [9, 13, 27, 36, 39, 45] provided by the community, and many excellent works [2,3,5,7,22] have spurted out over the past decades.”
BackgroundMethodSiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks
Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., & Yan, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00441 · Cited by 2,607
GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
Huang, L., Zhao, X., & Huang, K. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2957464 · Cited by 1,809
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking
Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., & Ling, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00552 · Cited by 1,742 · cited 6×
“For example, the widely used RGB-based tracking datasets, GOT-10k [13], Track-ingNet [36], and LaSOT [9], contain 9.3K, 30.1K, and 1.1K sequences, corresponding to 1.4M, 14M, and 2.8M frames for training.”
BackgroundMethodInfluentialATOM: Accurate Tracking by Overlap Maximization
Danelljan, M., Bhat, G., Khan, F. S., & Felsberg, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00479 · Cited by 1,591
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., & Lu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00803 · Cited by 1,588
Self-citationLearning Discriminative Model Prediction for Tracking
Bhat, G., Danelljan, M., Van Gool, L., & Timofte, R. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00628 · Cited by 1,515
Learning Spatio-Temporal Transformer for Visual Tracking
Yan, B., Peng, H., Fu, J., Wang, D., & Lu, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01028 · Cited by 1,280 · cited 1×
“Overall performance on LasHeR test set [25]. mfDiMP [53] SGT [26] DAFNet [10] OSTrack [49] FANet [62] MaCNet [50] CAT [24] JMMAC [55] CMPP [42] APFNet [44] ProTrack [48] ViPT based tracker ProTrack [48], only has an improvement of 0.4% relative to their baseline STARK RGBD [46].”
ResultInfluentialSelf-citationTrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild
Müller, M., Bibi, A., Giancola, S., Alsubaihi, S., & Ghanem, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_19 · Cited by 1,192
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework
Ye, B., Chang, H., Ma, B., Shan, S., & Chen, X. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20047-2_20 · Cited by 808
Self-citationYouTube-VOS: Sequence-to-Sequence Video Object Segmentation
Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., & Huang, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_36 · Cited by 598
RGB-T object tracking: Benchmark and baseline
Li, C., Liang, X., Lu, Y., Zhao, N., & Tang, J. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.106977 · Cited by 581
The Seventh Visual Object Tracking VOT2019 Challenge Results
Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kamarainen, J.-K., Zajc, L. C., Drbohlav, O., Lukezic, A., Berg, A., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00276 · Cited by 461
Self-citationAdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
Chen, S., Ge, C., Tong, Z., Wang, J., Tong, Z., Wang, J., & Luo, P. (2022) · neural information processing systems · doi:10.52202/068431-1212 · Cited by 451
LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking
Li, C., Xue, W., Jia, Y., Qu, Z., Luo, B., Tang, J., & Sun, D. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3130533 · Cited by 307
Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline
Zhang, P., Zhao, J., Wang, D., Lu, H., & Ruan, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00868 · Cited by 278
Self-citationWeighted Sparse Representation Regularized Graph Learning for RGB-T Object Tracking
Li, C., Zhao, N., Lu, Y., Zhu, C., & Tang, J. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123289 · Cited by 260
The Eighth Visual Object Tracking VOT2020 Challenge Results
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.-K., Danelljan, M., Zajc, L. Č., Lukežič, A., Drbohlav, O., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-68238-5_39 · Cited by 259
Self-citationReal-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., & Theobalt, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.131 · Cited by 242
Multi-Modal Fusion for End-to-End RGB-T Tracking
Zhang, L., Danelljan, M., Gonzalez-Garcia, A., van de Weijer, J., & Shahbaz Khan, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00278 · Cited by 212
Attribute-Based Progressive Fusion Network for RGBT Tracking
Xiao, Y., Yang, M., Li, C., Liu, L., & Tang, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20187 · Cited by 208
Cross-Modal Pattern-Propagation for RGB-T Tracking
Wang, C., Xu, C., Cui, Z., Zhou, L., Zhang, T., Zhang, X., & Yang, J. J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00709 · Cited by 197
Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking
Zhang, P., Zhao, J., Bo, C., Wang, D., Lu, H., & Yang, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3060862 · Cited by 196
Self-citationQuality-Aware Feature Aggregation Network for Robust RGBT Tracking
Zhu, Y., Li, C., Tang, J., & Luo, B. (2021) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.2980735 · Cited by 188
Object Tracking by Jointly Exploiting Frame and Event Domain
Zhang, J., Yang, X., Fu, Y., Wei, X., Yin, B., & Dong, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01280 · Cited by 187
Spiking Transformers for Event-based Single Object Tracking
Zhang, J., Dong, B., Zhang, H., Ding, J., Heide, F., Yin, B., & Yang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00860 · Cited by 186
Dense Feature Aggregation and Pruning for RGBT Tracking
Zhu, Y., Li, C., Luo, B., Tang, J., & Wang, X. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350928 · Cited by 185
Learning Adaptive Attribute-Driven Representation for Real-Time RGB-T Tracking
Zhang, P., Wang, D., Lu, H., & Yang, X. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01495-3 · Cited by 184
Self-citationSiamCDA: Complementarity- and Distractor-Aware RGB-T Tracking Based on Siamese Network
Zhang, T., Liu, X., Zhang, Q., & Han, J. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3072207 · Cited by 182
Miangoleh, S. M. H., Dille, S., Mai, L., Paris, S., & Aksoy, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00956 · Cited by 179
VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows
Wang, X., Li, J., Zhu, L., Zhang, Z., Chen, Z., Li, X., Wang, Y., Tian, Y., & Wu, F. (2024) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2023.3318601 · Cited by 176
Deep Adaptive Fusion Network for High Performance RGBT Tracking
Gao, Y., Li, C., Zhu, Y., Tang, J., He, T., & Wang, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00017 · Cited by 172
Prompting for Multi-Modal Tracking
Yang, J., Li, Z., Zheng, F., Leonardis, A., & Song, J. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547851 · Cited by 172
DepthTrack: Unveiling the Power of RGBD Tracking
Yan, S., Yang, J., Käpylä, J., Zheng, F., Leonardis, A., & Kämäräinen, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01055 · Cited by 169
The Ninth Visual Object Tracking VOT2021 Challenge Results
Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R. P., Kämäräinen, J., Chang, H. J., Danelljan, M., Zajc, L. Č., Lukežič, A., et al. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00305 · Cited by 164
Self-citationObject Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning
Zhang, H., Zhang, L., Li, Z., & Zhang, J. (2020) · Sensors · doi:10.3390/s20020393 · Cited by 163
Li, C., Liu, L., Lu, A., Ji, Q., & Tang, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58542-6_14 · Cited by 162
The Tenth Visual Object Tracking VOT2022 Challenge Results
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R. P., Kämäräinen, J., Chang, H. J., Danelljan, M., Zajc, L. Č., Lukežič, A., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25085-9_25 · Cited by 145
Self-citationExploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129
Liu, Y., Jing, X., Nie, J., Gao, H., Liu, J., & Jiang, G. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2863604 · Cited by 103
Thermo-visual feature fusion for object tracking using multiple spatiogram trackers
Conaire, C. Ó., O'Connor, N. E., & Smeaton, A. F. (2007) · Machine Vision and Applications · doi:10.1007/s00138-007-0078-y · Cited by 102
Combined frame- and event-based detection and tracking
Liu, H., Moeys, D. P., Das, G. P., Neil, D., Liu, S., & Delbrück, T. (2016) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2016.7539103 · Cited by 96
DAL: A Deep Depth-Aware Long-term Tracker
Qian, Y., Yan, S., Lukežič, A., Kristan, M., Kämäräinen, J., & Matas, J. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412984 · Cited by 71
RGBD1K: A Large-Scale Dataset and Benchmark for RGB-D Object Tracking
Zhu, X., Xu, T., Tang, Z., Wu, Z., Liu, H., Yang, X. P., Wu, X., & Kittler, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25500 · Cited by 68
Luo, C., Sun, B., Yang, K., Lu, T., & Yeh, W. (2019) · Infrared Physics & Technology · doi:10.1016/j.infrared.2019.04.017 · Cited by 67
Prompt Vision Transformer for Domain GeneralizationPreprint
Zheng, Z., Yue, X., Wang, K., & You, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.08914 · Cited by 60
Zhang, X., Ye, P., Qiao, D., Zhao, J., Peng, S., & Xiao, G. (2019) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion43075.2019.9011253 · Cited by 54
Prompt-Matched Semantic SegmentationPreprint
Liu, L., Chang, J., Yu, B. X. B., Lin, L., Tian, Q. C., & Chen, C.-W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10159 · Cited by 33
Cited by
Prompt Engineering in Large Language Models
Marvin, G., Hellen, N., Jjingo, D., & Nakatumba-Nabende, J. (2024) · Algorithms for Intelligent Systems · doi:10.1007/978-981-99-7962-2_30 · Cited by 403
Faster Parameter-Efficient Tuning with Token Redundancy Reduction
Kim, K., Park, J., Kim, J., Kwon, H., & Sohn, K. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02810 · Cited by 5
“Prompt-based methods [14, 19, 21, 26, 33, 34, 41, 43] introduce a small number of learnable tokens that are concatenated with input tokens of transformer.”
Metadata
- Authors
- Zhu, J., Lai, S., Chen, X., Wang, D., & Lu, H.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 9516-9526
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386075647
- Cited by
- 349
- References
- 61
- FWCI
- 39.68
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.