DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
In this paper, we study masked autoencoder (MAE) pretraining on videos for matching-based downstream tasks, including visual object tracking (VOT) and video object segmentation (VOS). A simple extension of MAE is to randomly mask out frame patches in videos and reconstruct the frame pixels. However, we find that this simple baseline heavily relies on spatial cues while ignoring temporal relations for frame reconstruction, thus leading to sub-optimal temporal matching representations for VOT and VOS. To alleviate this problem, we propose DropMAE, which adaptively performs spatial-attention dropout in the frame reconstruction to facilitate temporal correspondence learning in videos. We show that our DropMAE is a strong and efficient temporal matching learner, which achieves better finetuning results on matching-based tasks than the ImageNet-based MAE with$2\times$faster pre-training speed. Moreover, we also find that motion diversity in pre-training videos is more important than scene diversity for improving the performance on VOT and VOS. Our pre-trained DropMAE model can be directly loaded in existing ViT-based trackers for fine-tuning without further modifications. Notably, DropMAE sets new state-of-the-art performance on 8 out of 9 highly competitive video tracking and segmentation datasets. Our code and pre-trained models are available at https://github.com/jimmy-dq/DropMAE.git.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
High-Speed Tracking with Kernelized Correlation Filters
Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2345390 · Cited by 5,753
Fully-Convolutional Siamese Networks for Object Tracking
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_56 · Cited by 4,546
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664
Wu, Y., Lim, J., & Yang, M.-H. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2388226 · Cited by 3,371
High Performance Visual Tracking with Siamese Region Proposal Network
Li, B., Yan, J., Wu, W., Zhu, Z., & Hu, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00935 · Cited by 3,023
Learning Multi-domain Convolutional Neural Networks for Visual Tracking
Nam, H., & Han, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.465 · Cited by 2,806
Research works printed but not linked: 64
- Bao, 2021, Beit: Bert pre-training of image transformers, arXiv
- Bhat, 2019, Lean-ring discriminative model prediction for tracking, 6182
- Cao, 2020, Parametric instance classification for unsuper-vised visual feature learning
- Carreira, 2019, A short note on the kinetics700 human action dataset, arXiv
- Chen, 2022, Back-bone is all your need: A simplified architecture for visual object tracking
- Chen, 2020, A simple frame-work for contrastive learning of visual representations
- Cheng, 2021, Rethinking spacetime networks with improved memory coverage for efficient video object segmentation, 11781
- 10.1109/CVPR.2018.00774 · DOI · Paper page
- 10.1109/CVPR52688.2022.01324 · DOI · Paper page
- Devlin, 2019, Language models are few-shot learners
- 10.1007/978-3-030-01261-8_28 · DOI · Paper page
- Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
- Dosovitskiy, 2014, Discriminative unsupervised feature learning with convolutional neural networks, 766
- 10.1109/CVPR46437.2021.00585 · DOI · Paper page
- 10.48550/arXiv.l812.06148 · arXiv · doi:10.48550/arxiv.l812.06148
- Feichtenhofer, 2022, Masked autoen-coders as spatiotemporal learners, arXiv
- 10.1109/ICCV.2017.129 · DOI · Paper page
- 10.1007/978-3-031-20047-2_9 · DOI · Paper page
- Gidaris, 2018, Unsupervised representation learning by predicting image rotations, arXiv
- 10.1109/ICCV.2017.196 · DOI · Paper page
- Kay, 2017, The kinetics human action video dataset, arXiv
- 10.1109/CVPR.2019.00441 · DOI · Paper page
- Li, 2021, An informative tracking benchmark, arXiv
- 10.1007/978-3-031-20077-9_17 · DOI · Paper page
- 10.1145/3240508.3240709 · DOI
- 10.1109/TITS.2021.3103601 · DOI
- Lin, 2021, Swintrack: A simple and strong baseline for transformer tracking, arXiv
- 10.1007/978-3-319-10602-1_48 · DOI · Paper page
- 10.1109/CVPR52688.2022.00142 · DOI
- 10.1109/ICASSP43922.2022.9747821 · DOI
- 10.1109/CVPR.2017.515 · DOI · Paper page
- Maninis, 2018, Gool. ideo object segmentation without temporal information
- 10.1109/ICCV48922.2021.01319 · DOI · Paper page
- 10.1109/TPAMI.2019.2901464 · DOI · Paper page
- 10.1109/ICCV.2019.00932 · DOI · Paper page
- 10.1007/978-3-031-20047-2_33 · DOI
- 10.1109/CVPR.2016.85 · DOI · Paper page
- 10.1007/978-3-031-20047-2_25 · DOI
- Ridnik, 2021, Imagenet-21k pretraining for the masses, arXiv
- 10.1007/s11263-015-0816-y · DOI · Paper page
- 10.1109/CVPR.2016.158 · DOI · Paper page
- Touvron, 2021, Training data-efficient image transformers distillation through attention
- 10.1109/CVPR42600.2020.00661 · DOI · Paper page
- Vondrick, 2016, Generating videos with scene dynamics
- 10.1109/CVPR46437.2021.00162 · DOI · Paper page
- 10.1109/CVPR.2018.00511 · DOI
- 10.1109/CVPR46437.2021.01355 · DOI · Paper page
- 10.1109/ICCV.2019.00408 · DOI · Paper page
- Wang, 2021, Do different tracking tasks require different appearance models?
- 10.1109/ICME51207.2021.9428441 · DOI
- 10.1109/CVPR46437.2021.00301 · DOI
- 10.1007/978-3-030-20873-8_8 · DOI
- 10.1109/CVPR52688.2022.00855 · DOI · Paper page
- Xie, 2022, Sim-mim: A simple framework for masked image modeling, 9653
- 10.1007/978-3-031-19803-8_43 · DOI
- 10.1109/CVPR42600.2020.00675 · DOI
- Yang, 2021, Associating objects with transformers for video object segmentation, 2491
- 10.1007/978-3-031-20047-2_20 · DOI · Paper page
- Zhang, 2022, Uast: Uncertainty-aware siamese tracking
- 10.1109/ICCV.2019.00411 · DOI · Paper page
- Zhang, 2016, Image Colorization
- 10.1109/CVPR42600.2020.00698 · DOI
- 10.1109/ICCV48922.2021.01309 · DOI · Paper page
- Zhang, 2017, Deeper and wider siamese networks for raal-time visual tracking
Cited by
SAM2LONG: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree
Ding, S., Qian, R., Dong, X., Zhang, P., Zang, Y., Cao, Y., Guo, Y., Lin, D., & Wang, J. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01264 · Cited by 19
Metadata
- Authors
- Wu, Q., Yang, T., Liu, Z., Wu, B., Shan, Y., & Chan, A. B.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 14561-14571
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386066394
- Cited by
- 178
- References
- 41
- FWCI
- 20.24
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks. https://app.cochat.ai/papers/10.1109/cvpr52729.2023.01399
- The paper
- Wu, Q., Yang, T., Liu, Z., Wu, B., Shan, Y., & Chan, A. B. (2023). DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52729.2023.01399
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.