Tracking by Instance Detection: A Meta-Learning Approach
Open accessAbstract
We consider the tracking problem as a special type of object detection problem, which we call instance detection. With proper initialization, a detector can be quickly converted into a tracker by learning the new instance from a single image. We find that model-agnostic meta-learning (MAML) offers a strategy to initialize the detector that satisfies our needs. We propose a principled three-step approach to build a high-performance tracker. First, pick any modern object detector trained with gradient descent. Second, conduct offline training (or initialization) with MAML. Third, perform domain adaptation using the initial frame. We follow this procedure to build two trackers, named Retina-MAML and FCOS-MAML, based on two modern detectors RetinaNet and FCOS. Evaluations on four benchmarks show that both trackers are competitive against state-of-the-art trackers. On OTB-100, Retina-MAML achieves the highest ever AUC of 0.712. On TrackingNet, FCOS-MAML ranks the first on the leader board with an AUC of 0.757 and the normalized precision of 0.822. Both trackers run in real-time at 40 FPS.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · see citing works
FCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013 · see citing works
Fully-Convolutional Siamese Networks for Object Tracking
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_56 · Cited by 4,546 · see citing works
Wu, Y., Lim, J., & Yang, M.-H. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2388226 · Cited by 3,371 · see citing works
High Performance Visual Tracking with Siamese Region Proposal Network
Li, B., Yan, J., Wu, W., Zhu, Z., & Hu, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00935 · Cited by 3,023 · see citing works
Learning Multi-domain Convolutional Neural Networks for Visual Tracking
Nam, H., & Han, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.465 · Cited by 2,806 · see citing works
ECO: Efficient Convolution Operators for Tracking
Danelljan, M., Bhat, G., Khan, F. S., & Felsberg, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.733 · Cited by 2,805 · see citing works
SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks
Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., & Yan, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00441 · Cited by 2,607 · see citing works
GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
Huang, L., Zhao, X., & Huang, K. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2957464 · Cited by 1,809 · see citing works
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking
Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., & Ling, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00552 · Cited by 1,742 · see citing works
ATOM: Accurate Tracking by Overlap Maximization
Danelljan, M., Bhat, G., Khan, F. S., & Felsberg, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00479 · Cited by 1,591 · see citing works
Distractor-Aware Siamese Networks for Visual Object Tracking
Zhu, Z. D., Wang, Q., Li, B., Wu, W., Yan, J., & Hu, W. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_7 · Cited by 1,575 · see citing works
Learning Discriminative Model Prediction for Tracking
Bhat, G., Danelljan, M., Van Gool, L., & Timofte, R. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00628 · Cited by 1,515 · see citing works
Learning Background-Aware Correlation Filters for Visual Tracking
Galoogahi, H. K., Fagg, A., & Lucey, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.129 · Cited by 1,270 · see citing works
TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild
Müller, M., Bibi, A., Giancola, S., Alsubaihi, S., & Ghanem, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_19 · Cited by 1,192 · see citing works
Acquisition of Localization Confidence for Accurate Object Detection
Jiang, B., Luo, R., Mao, J., Xiao, T., & Jiang, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_48 · Cited by 1,059 · see citing works
Meta-SGD: Learning to Learn Quickly for Few-Shot LearningPreprint
Li, Z., Zhou, F., Chen, F., & Li, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09835 · Cited by 1,026 · see citing works
The Sixth Visual Object Tracking VOT2018 Challenge Results
Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Zajc, L. Č., Vojír, T., Bhat, G., Lukežič, A., Eldesokey, A., et al. (2018) · ECCV Workshops · doi:10.1007/978-3-030-11009-3_1 · Cited by 805 · see citing works
Self-citationA Twofold Siamese Network for Real-Time Object Tracking
He, A., Luo, C., Tian, X., & Zeng, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00508 · Cited by 708 · see citing works
Self-citationWang, Q., Teng, Z., Xing, J., Gao, J., Hu, W., & Maybank, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00510 · Cited by 604 · see citing works
VITAL: VIsual Tracking via Adversarial Learning
Song, Y., Ma, C., Wu, X., Gong, L., Bao, L., Zuo, W., Shen, C., Lau, R. W. H., & Yang, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00937 · Cited by 595 · see citing works
CREST: Convolutional Residual Learning for Visual Tracking
Song, Y., Ma, C., Gong, L., Zhang, J., Lau, R. W. H., & Yang, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.279 · Cited by 572 · see citing works
Unveiling the Power of Deep Tracking
Bhat, G., Johnander, J., Danelljan, M., Khan, F. S., & Felsberg, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_30 · Cited by 514 · see citing works
Multi-cue Correlation Filters for Robust Visual Tracking
Wang, N., Zhou, W., Tian, Q. C., Hong, R., Wang, M., & Li, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00509 · Cited by 492 · see citing works
Siamese Cascaded Region Proposal Networks for Real-Time Visual Tracking
Fan, H., & Ling, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00814 · Cited by 477 · see citing works
Xu, T., Feng, Z.-H., Wu, X.-J., & Kittler, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2919201 · Cited by 411 · see citing works
Learning Dynamic Memory Networks for Object Tracking
Yang, T., & Chan, A. B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_10 · Cited by 382 · see citing works
GradNet: Gradient-Guided Network for Visual Object Tracking
Li, P., Chen, B., Ouyang, W., Wang, D., Yang, X., & Lu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00626 · Cited by 348 · see citing works
Jung, I., Son, J., Baek, M., & Han, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_6 · Cited by 297 · see citing works
Modeling and Propagating CNNs in a Tree Structure for Visual TrackingPreprint
Nam, H., Baek, M., & Han, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07242 · Cited by 280 · see citing works
SPM-Tracker: Series-Parallel Matching for Real-Time Visual Object Tracking
Wang, G., Luo, C., Xiong, Z., & Zeng, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00376 · Cited by 269 · see citing works
Self-citationMeta-tracker: Fast and Robust Online Adaptation for Visual Object Trackers
Park, E., & Berg, A. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_35 · Cited by 189 · see citing works
Correlation Tracking via Joint Discrimination and Reliability Learning
Sun, C., Wang, D., Lu, H., & Yang, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00058 · Cited by 185 · see citing works
Deep Meta Learning for Real-Time Target-Aware Visual Tracking
Choi, J., Kwon, J., & Lee, K. M. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00100 · Cited by 117 · see citing works
Bridging the Gap Between Detection and Tracking: A Unified Approach
Huang, L., Zhao, X., & Huang, K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00410 · Cited by 54 · see citing works
Joint Representation and Truncated Inference Learning for Correlation Filter Based Tracking
Yao, Y., Wu, X., Lei Zhang, A., Shan, S., & Zuo, W. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_34 · Cited by 48 · see citing works
Legal Reasoning and Big Data: Opportunities and ChallengesPreprint
Antoniou, G., Baryannis, G., Batsakis, S., et al. (2018) · EasyChair preprint · doi:10.29007/tkmv · Cited by 22 · see citing works
Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel PruningPreprint
Jung, I., You, K., Noh, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11170 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., & Lu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00803 · Cited by 1,588 · see citing works
Learning Spatio-Temporal Transformer for Visual Tracking
Yan, B., Peng, H., Fu, J., Wang, D., & Lu, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01028 · Cited by 1,280 · see citing works
“To improve the optimization efficiency, later works [9, 3, 27, 48, 55] adopt more advanced optimization methods like Gauss-Newton method or meta-learning-based update strategies.”
Method
- Authors
- Wang, G., Luo, C., Sun, X., Xiong, Z., & Zeng, W.
- Affiliations
- University of Science and Technology of China (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 6287-6296
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034617042
- Cited by
- 206
- References
- 43
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 37 · 2023: 35 · 2022: 48 · 2021: 55 · 2020: 14
- FWCI
- 11.72
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- visual object tracking, instance detection, meta-learning, model-agnostic meta-learning, domain adaptation, RetinaNet, FCOS, real-time tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
