CoChat

DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks

Wu, Qiangqiang, Yang, Tianyu, Liu, Ziquan, Wu, Baoyuan, Shan, Ying, Chan, Antoni B. · 2023 · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Cited by 178

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.

Abstract

In this paper, we study masked autoencoder (MAE) pretraining on videos for matching-based downstream tasks, including visual object tracking (VOT) and video object segmentation (VOS). A simple extension of MAE is to randomly mask out frame patches in videos and reconstruct the frame pixels. However, we find that this simple baseline heavily relies on spatial cues while ignoring temporal relations for frame reconstruction, thus leading to sub-optimal temporal matching representations for VOT and VOS. To alleviate this problem, we propose DropMAE, which adaptively performs spatial-attention dropout in the frame reconstruction to facilitate temporal correspondence learning in videos. We show that our DropMAE is a strong and efficient temporal matching learner, which achieves better finetuning results on matching-based tasks than the ImageNet-based MAE with$2\times$faster pre-training speed. Moreover, we also find that motion diversity in pre-training videos is more important than scene diversity for improving the performance on VOT and VOS. Our pre-trained DropMAE model can be directly loaded in existing ViT-based trackers for fine-tuning without further modifications. Notably, DropMAE sets new state-of-the-art performance on 8 out of 9 highly competitive video tracking and segmentation datasets. Our code and pre-trained models are available at https://github.com/jimmy-dq/DropMAE.git.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01399

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 63

  1. Bao, 2021, Beit: Bert pre-training of image transformers, arXiv
  2. Bhat, 2019, Lean-ring discriminative model prediction for tracking, 6182
  3. Cao, 2020, Parametric instance classification for unsuper-vised visual feature learning
  4. Carreira, 2019, A short note on the kinetics700 human action dataset, arXiv
  5. Chen, 2022, Back-bone is all your need: A simplified architecture for visual object tracking
  6. Chen, 2020, A simple frame-work for contrastive learning of visual representations
  7. Cheng, 2021, Rethinking spacetime networks with improved memory coverage for efficient video object segmentation, 11781
  8. 10.1109/CVPR.2018.00774 · DOI · Paper page
  9. 10.1109/CVPR52688.2022.01324 · DOI · Paper page
  10. Devlin, 2019, Language models are few-shot learners
  11. 10.1007/978-3-030-01261-8_28 · DOI · Paper page
  12. Dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale
  13. Dosovitskiy, 2014, Discriminative unsupervised feature learning with convolutional neural networks, 766
  14. 10.1109/CVPR46437.2021.00585 · DOI · Paper page
  15. 10.48550/arXiv.l812.06148 · arXiv · doi:10.48550/arxiv.l812.06148
  16. Feichtenhofer, 2022, Masked autoen-coders as spatiotemporal learners, arXiv
  17. 10.1109/ICCV.2017.129 · DOI · Paper page
  18. 10.1007/978-3-031-20047-2_9 · DOI · Paper page
  19. Gidaris, 2018, Unsupervised representation learning by predicting image rotations, arXiv
  20. 10.1109/ICCV.2017.196 · DOI · Paper page
  21. Kay, 2017, The kinetics human action video dataset, arXiv
  22. 10.1109/CVPR.2019.00441 · DOI · Paper page
  23. Li, 2021, An informative tracking benchmark, arXiv
  24. 10.1007/978-3-031-20077-9_17 · DOI · Paper page
  25. 10.1145/3240508.3240709 · DOI
  26. 10.1109/TITS.2021.3103601 · DOI
  27. Lin, 2021, Swintrack: A simple and strong baseline for transformer tracking, arXiv
  28. 10.1007/978-3-319-10602-1_48 · DOI · Paper page
  29. 10.1109/CVPR52688.2022.00142 · DOI
  30. 10.1109/ICASSP43922.2022.9747821 · DOI
  31. 10.1109/CVPR.2017.515 · DOI · Paper page
  32. Maninis, 2018, Gool. ideo object segmentation without temporal information
  33. 10.1109/ICCV48922.2021.01319 · DOI · Paper page
  34. 10.1109/TPAMI.2019.2901464 · DOI · Paper page
  35. 10.1109/ICCV.2019.00932 · DOI · Paper page
  36. 10.1007/978-3-031-20047-2_33 · DOI
  37. 10.1109/CVPR.2016.85 · DOI · Paper page
  38. 10.1007/978-3-031-20047-2_25 · DOI
  39. Ridnik, 2021, Imagenet-21k pretraining for the masses, arXiv
  40. 10.1007/s11263-015-0816-y · DOI · Paper page
  41. Touvron, 2021, Training data-efficient image transformers distillation through attention
  42. 10.1109/CVPR42600.2020.00661 · DOI · Paper page
  43. Vondrick, 2016, Generating videos with scene dynamics
  44. 10.1109/CVPR46437.2021.00162 · DOI · Paper page
  45. 10.1109/CVPR.2018.00511 · DOI
  46. 10.1109/CVPR46437.2021.01355 · DOI · Paper page
  47. 10.1109/ICCV.2019.00408 · DOI · Paper page
  48. Wang, 2021, Do different tracking tasks require different appearance models?
  49. 10.1109/ICME51207.2021.9428441 · DOI
  50. 10.1109/CVPR46437.2021.00301 · DOI
  51. 10.1007/978-3-030-20873-8_8 · DOI
  52. 10.1109/CVPR52688.2022.00855 · DOI · Paper page
  53. Xie, 2022, Sim-mim: A simple framework for masked image modeling, 9653
  54. 10.1007/978-3-031-19803-8_43 · DOI
  55. 10.1109/CVPR42600.2020.00675 · DOI
  56. Yang, 2021, Associating objects with transformers for video object segmentation, 2491
  57. 10.1007/978-3-031-20047-2_20 · DOI · Paper page
  58. Zhang, 2022, Uast: Uncertainty-aware siamese tracking
  59. 10.1109/ICCV.2019.00411 · DOI · Paper page
  60. Zhang, 2016, Image Colorization
  61. 10.1109/CVPR42600.2020.00698 · DOI
  62. 10.1109/ICCV48922.2021.01309 · DOI · Paper page
  63. Zhang, 2017, Deeper and wider siamese networks for raal-time visual tracking

Cited by

See all 178

Metadata

Authors
Wu, Q., Yang, T., Liu, Z., Wu, B., Shan, Y., & Chan, A. B.
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
14561-14571
ISSN
1063-6919 (ISSN-L)
OpenAlex
W4386066394
Cited by
178
References
42
FWCI
20.24
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.