CoChat
Sign Up
PreprintCited by 3,594Advanced Neural Network ApplicationsMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

Transformers in Vision: A Survey

Open access
Khan, Salman, Naseer, Muzammal, Hayat, Munawar, Zamir, Syed Waqas, Khan, Fahad Shahbaz, Shah, Mubarak·2021·arXiv
DOI 10.48550/arxiv.2101.01169arXiv 2101.01169
Open this paper in CoChat — free accountOpen PDF

Abstract

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2101.01169

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Although there hasn't been a ViT dedicated for DeepFake generation or detection yet, the unprecedentedly fast growing pace and increasing ubiquity in ViTs to tackle various computer vision problems will eventually turn the landscape of DeepFake generation and detection into a ViT-based one.

    “Although there hasn't been a ViT dedicated for DeepFake generation or detection yet, the unprecedentedly fast growing pace and increasing ubiquity in ViTs (Khan et al. 2021) to tackle various computer vision problems will eventually turn the landscape of DeepFake generation and detection into a ViT-based one.”

    From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Khan et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 230

  1. [1][1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' in NeurIPS , 2017. -
  2. [3][3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'BERT: Pretraining of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
  3. [6][6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. , 'Language models are few-shot learners,' arXiv preprint arXiv:2005.14165 , 2020. - · arXiv
  4. [7][7] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, 'RoBERTa: A robustly optimized bert pretraining approach,' arXiv preprint arXiv:1907.11692 , 2019. - · arXiv
  5. [8][8] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, 'Exploring the limits of transfer learning with a unified text-to-text transformer,' arXiv preprint arXiv:1910.10683 , 2019. - · arXiv
  6. [9][9] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, 'Gshard: Scaling giant models with conditional computation and automatic sharding,' arXiv preprint arXiv:2006.16668 , 2020. - · arXiv
  7. [10][10] W. Fedus, B. Zoph, and N. Shazeer, 'Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,' arXiv preprint arXiv:2101.03961 . - · arXiv
  8. [11][11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020. - · arXiv
  9. [12][12] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J´ egou, 'Training data-efficient image transformers & distillation through attention,' arXiv preprint arXiv:2012.12877 , 2020. - · arXiv
  10. [13][13] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, 'End-to-end object detection with transformers,' arXiv preprint arXiv:2005.12872 , 2020. - · arXiv
  11. [14][14] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, 'Deformable DETR: Deformable transformers for end-to-end object detection,' arXiv preprint arXiv:2010.04159 , 2020. - · arXiv
  12. [15][15] L. Ye, M. Rochan, Z. Liu, and Y. Wang, 'Cross-modal selfattention network for referring image segmentation,' in CVPR , 2019. -
  13. [16][16] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, 'Learning texture transformer network for image super-resolution,' in CVPR , 2020. -
  14. [17][17] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, 'VideoBERT: A joint model for video and language representation learning,' in ICCV , 2019. -
  15. [18][18] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, 'Video action transformer network,' in CVPR , 2019. -
  16. [19][19] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, 'Pre-trained image processing transformer,' arXiv preprint arXiv:2012.00364 , 2020. - · arXiv
  17. [21][21] H. Tan and M. Bansal, 'LXMERT: Learning cross-modality encoder representations from transformers,' in EMNLP-IJCNLP , 2019. -
  18. [22][22] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai, 'VL-BERT: Pre-training of generic visual-linguistic representations,' arXiv preprint arXiv:1908.08530 , 2019. - · arXiv
  19. [23][23] X. Wang, C. Yeshwanth, and M. Nießner, 'SceneFormer: Indoor scene generation with transformers,' arXiv preprint arXiv:2012.09793 , 2020. - · arXiv
  20. [24][24] M. Kumar, D. Weissenborn, and N. Kalchbrenner, 'Colorization transformer,' in ICLR , 2021. -
  21. [25][25] C. Doersch, A. Gupta, and A. Zisserman, 'CrossTransformers: spatially-aware few-shot transfer,' NeurIPS , 2020. -
  22. [26][26] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, 'Few-shot learning via embedding adaptation with set-to-set functions,' in CVPR , 2020. -
  23. [27][27] S. Chaudhari, G. Polatkan, R. Ramanath, and V. Mithal, 'An attentive survey of attention models,' arXiv preprint arXiv:1904.02874 , 2019. - · arXiv
  24. [28][28] A. de Santana Correia and E. L. Colombini, 'Attention, please! asurvey of neural attention models in deep learning,' arXiv preprint arXiv:2103.16775 , 2021. - · arXiv
  25. [29][29] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, 'Show and tell: A neural image caption generator,' in CVPR , 2015. -
  26. [30][30] Y. Bengio, I. Goodfellow, and A. Courville, Deep learning . MIT press, 2017. -
  27. [34][34] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, 'Efficient transformers: A survey,' arXiv preprint arXiv:2009.06732 , 2020. - · arXiv
  28. [35][35] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. Tay, J. Feng, and S. Yan, 'Tokens-to-token vit: Training vision transformers from scratch on imagenet,' arXiv preprint arXiv:2101.11986 , 2021. - · arXiv
  29. [36][36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' arXiv preprint arXiv:2103.14030 , 2021. - · arXiv
  30. [37][37] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, 'Twins: Revisiting the design of spatial attention in vision transformers,' arXiv preprint arXiv:2104.13840 , 2021. - · arXiv
  31. [38][38] S. Wang, B. Li, M. Khabsa, H. Fang, and H. Ma, 'Linformer: Selfattention with linear complexity,' arXiv preprint arXiv:2006.04768 , 2020. - · arXiv
  32. [39][39] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, 'Selfattention generative adversarial networks,' in International conference on machine learning , pp. 7354-7363, PMLR, 2019. -
  33. [40][40] J. P´ erez, J. Marinkovi´ c, and P. Barcel´ o, 'On the turing completeness of modern neural network architectures,' in International Conference on Learning Representations , 2018. -
  34. [41][41] J.-B. Cordonnier, A. Loukas, and M. Jaggi, 'On the relationship between self-attention and convolutional layers,' in International Conference on Learning Representations , 2019. -
  35. [42][42] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, 'Deformable convolutional networks,' in Proceedings of the IEEE international conference on computer vision , pp. 764-773, 2017. -
  36. [43][43] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'UNITER: Universal image-text representation learning,' in ECCV , 2020. -
  37. [44][44] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. , 'Oscar: Object-semantics aligned pretraining for vision-language tasks,' in ECCV , 2020. -
  38. [45][45] K. Lin, L. Wang, and Z. Liu, 'End-to-end human pose and mesh reconstruction with transformers,' arXiv preprint arXiv:2012.09760 , 2020. - · arXiv
  39. [46][46] S. Gidaris, P. Singh, and N. Komodakis, 'Unsupervised representation learning by predicting image rotations,' in ICLR , 2018. -
  40. [48][48] L. Jing and Y. Tian, 'Self-supervised visual feature learning with deep neural networks: A survey,' TPAMI , 2020. -
  41. [49][49] X. Liu, F. Zhang, Z. Hou, Z. Wang, L. Mian, J. Zhang, and J. Tang, 'Self-supervised learning: Generative or contrastive,' arXiv preprint arXiv:2006.08218 , 2020. - · arXiv
  42. [51][51] R. Zhang, P. Isola, and A. A. Efros, 'Colorful image colorization,' in ECCV , 2016. -
  43. [52][52] C. Ledig, L. Theis, F. Husz´ ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. , 'Photorealistic single image super-resolution using a generative adversarial network,' in CVPR , 2017. -
  44. [53][53] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. Efros, 'Context encoders: Feature learning by inpainting,' in CVPR , 2016. -
  45. [54][54] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial nets,' in NeurIPS , 2014. -
  46. [56][56] U. Ahsan, R. Madhok, and I. Essa, 'Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition,' in WACV , 2019. -
  47. [57][57] M. Noroozi and P. Favaro, 'Unsupervised learning of visual representations by solving jigsaw puzzles,' in ECCV , 2016. -
  48. [58][58] D. Kim, D. Cho, D. Yoo, and I. S. Kweon, 'Learning image representations by completing damaged jigsaw puzzles,' WACV , 2018. -
  49. [59][59] L. Jing, X. Yang, J. Liu, and Y. Tian, 'Self-supervised spatiotemporal feature learning via video rotation prediction,' arXiv preprint arXiv:1811.11387 , 2018. - · arXiv
  50. [60][60] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang, 'Unsupervised representation learning by sorting sequences,' in ICCV , 2017. -
  51. [61][61] I. Misra, C. L. Zitnick, and M. Hebert, 'Shuffle and learn: unsupervised learning using temporal order verification,' in ECCV , 2016. -
  52. [62][62] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, 'Learning and using the arrow of time,' in CVPR , 2018. -
  53. [63][63] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, 'VisualBERT: A simple and performant baseline for vision and language,' in Arxiv preprint arXiv:1908.03557 , 2019. - · arXiv
  54. [64][64] B. Korbar, D. Tran, and L. T., 'Cooperative learning of audio and video models from self-supervised synchronization,' in NeurIPS , 2018. -
  55. [65][65] R. Arandjelovic and A. Zisserman, 'Look, listen and learn,' in ICCV , 2017. -
  56. [67][67] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in CVPR , 2016. -
  57. [68][68] J. L. Ba, J. R. Kiros, and G. E. Hinton, 'Layer normalization,' arXiv preprint arXiv:1607.06450 , 2016. - · arXiv
  58. [69][69] A. Buades, B. Coll, and J.-M. Morel, 'A non-local algorithm for image denoising,' in CVPR , 2005. -
  59. [70][70] X. Wang, R. Girshick, A. Gupta, and K. He, 'Non-local neural networks,' in CVPR , 2018. -
  60. [71][71] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. , 'The kinetics human action video dataset,' arXiv preprint arXiv:1705.06950 , 2017. - · arXiv
  61. [72][72] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, 'CCNet: Criss-cross attention for semantic segmentation,' in ICCV , 2019. -
  62. [73][73] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, 'The cityscapes dataset for semantic urban scene understanding,' in CVPR , 2016. -
  63. [74][74] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, 'Scene parsing through ade20k dataset,' in CVPR , 2017. -
  64. [75][75] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: Common objects in context,' in ECCV , 2014. -
  65. [76][76] X. Liang, K. Gong, X. Shen, and L. Lin, 'Look into person: Joint body parsing & pose estimation network and a new benchmark,' TPAMI , 2018. -
  66. [77][77] G. J. Brostow, J. Fauqueur, and R. Cipolla, 'Semantic object classes in video: A high-definition ground truth database,' Pattern Recognition Letters , 2009. -
  67. [78][78] H. Hu, Z. Zhang, Z. Xie, and S. Lin, 'Local relation networks for image recognition,' in ICCV , 2019. -
  68. [79][79] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, 'Attention augmented convolutional networks,' in ICCV , 2019. -
  69. [80][80] P. Shaw, J. Uszkoreit, and A. Vaswani, 'Self-attention with relative position representations,' in NAACL , 2018. -
  70. [81][81] N. Parmar, P. Ramachandran, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, 'Stand-alone self-attention in vision models,' in NeurIPS , 2019. -
  71. [82][82] H. Zhao, J. Jia, and V. Koltun, 'Exploring self-attention for image recognition,' in CVPR , 2020. -
  72. [83][83] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, 'Intriguing properties of neural networks,' arXiv preprint arXiv:1312.6199 , 2013. - · arXiv
  73. [84][84] M. M. Naseer, S. H. Khan, M. H. Khan, F. S. Khan, and F. Porikli, 'Cross-domain transferability of adversarial perturbations,' in NeurIPS , 2019. -
  74. [85][85] M. Naseer, K. Ranasinghe, S. Khan, F. S. Khan, and F. Porikli, 'On improving adversarial transferability of vision transformers,' arXiv preprint arXiv:2106.04169 , 2021. - · arXiv
  75. [86][86] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Doll´ ar, 'Designing network design spaces,' in CVPR , 2020. -
  76. [87][87] M. Tan and Q. V. Le, 'EfficientNet: Rethinking model scaling for convolutional neural networks,' in ICML , 2019. -
  77. [88][88] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, 'Transformer in transformer,' arXiv preprint arXiv:2103.00112 , 2021. - · arXiv
  78. [89][89] Z. Jiang, Q. Hou, L. Yuan, D. Zhou, Y. Shi, X. Jin, A. Wang, and J. Feng, 'All tokens matter: Token labeling for training better vision transformers,' arXiv preprint arXiv:2104.10858 , 2021. - · arXiv
  79. [90][90] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, 'Cutmix: Regularization strategy to train strong classifiers with localizable features,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 6023-6032, 2019. -
  80. [93][93] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, 'Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,' arXiv preprint arXiv:2102.12122 , 2021. - · arXiv
  81. [96][96] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, 'Cvt: Introducing convolutions to vision transformers,' arXiv preprint arXiv:2103.15808 , 2021. - · arXiv
  82. [99][99] W. Wang, L. Yao, L. Chen, D. Cai, X. He, and W. Liu, 'Crossformer: A versatile vision transformer based on cross-scale attention,' arXiv preprint arXiv:2108.00154 , 2021. - · arXiv
  83. [102][102] P. Zhang, X. Dai, J. Yang, B. Xiao, L. Yuan, L. Zhang, and J. Gao, 'Multi-scale vision longformer: A new vision transformer for high-resolution image encoding,' ICCV 2021 , 2021. -
  84. [103][103] I. Beltagy, M. E. Peters, and A. Cohan, 'Longformer: The longdocument transformer,' arXiv preprint arXiv:2004.05150 , 2020. - · arXiv
  85. [104][104] C.-F. Chen, Q. Fan, and R. Panda, 'Crossvit: Cross-attention multi-scale vision transformer for image classification,' arXiv preprint arXiv:2103.14899 , 2021. - · arXiv
  86. [105][105] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu, 'Incorporating convolution designs into visual transformers,' arXiv preprint arXiv:2103.11816 , 2021. - · arXiv
  87. [109][109] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, 'Backpropagation applied to handwritten zip code recognition,' Neural computation , vol. 1, no. 4, pp. 541-551, 1989. -
  88. [110][110] Q. Zhang and Y. Yang, 'Rest: An efficient transformer for visual recognition,' arXiv preprint arXiv:2105.13677 , 2021. - · arXiv
  89. [111][111] Z. Zhang, H. Zhang, L. Zhao, T. Chen, and T. Pfister, 'Aggregating nested transformers,' in arXiv preprint arXiv:2105.12723 , 2021. - · arXiv
  90. [114][114] Y. Liu, G. Sun, Y. Qiu, L. Zhang, A. Chhatkuli, and L. Van Gool, 'Transformer in convolutional neural networks,' arXiv preprint arXiv:2106.03180 , 2021. - · arXiv
  91. [115][115] X. Chen, S. Xie, and K. He, 'An empirical study of training selfsupervised visual transformers,' arXiv e-prints , pp. arXiv-2104, 2021. -
  92. [116][116] X. Chen, H. Fan, R. Girshick, and K. He, 'Improved baselines with momentum contrastive learning,' arXiv preprint arXiv:2003.04297 , 2020. - · arXiv
  93. [117][117] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, 'Momentum contrast for unsupervised visual representation learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9729-9738, 2020. -
  94. [118][118] Z. Xie, Y. Lin, Z. Yao, Z. Zhang, Q. Dai, Y. Cao, and H. Hu, 'Self-supervised learning with swin transformers,' arXiv preprint arXiv:2105.04553 , 2021. - · arXiv
  95. [119][119] J.-B. Grill, F. Strub, F. Altch´ e, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. , 'Bootstrap your own latent: A new approach to selfsupervised learning,' arXiv preprint arXiv:2006.07733 , 2020. - · arXiv
  96. [120][120] M. Caron, H. Touvron, I. Misra, H. J´ egou, J. Mairal, P. Bojanowski, and A. Joulin, 'Emerging properties in self-supervised vision transformers,' arXiv preprint arXiv:2104.14294 , 2021. - · arXiv
  97. [121][121] C. Li, J. Yang, P. Zhang, M. Gao, B. Xiao, X. Dai, L. Yuan, and J. Gao, 'Efficient self-supervised vision transformers for representation learning,' arXiv preprint arXiv:2106.09785 , 2021. - · arXiv
  98. [125][125] S. Ren, K. He, R. Girshick, and J. Sun, 'Faster R-CNN: Towards real-time object detection with region proposal networks,' TPAMI , 2016. -
  99. [126][126] R. Girshick, 'Fast R-CNN,' in ICCV , 2015. -
  100. [127][127] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick, 'Mask R-CNN,' in ICCV , 2017. -
  101. [128][128] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, 'You only look once: Unified, real-time object detection,' in CVPR , 2016. -
  102. [129][129] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, 'SSD: Single shot multibox detector,' in ECCV , 2016. -
  103. [130][130] T. Prangemeier, C. Reich, and H. Koeppl, 'Attention-based transformers for instance segmentation of cells in microstructures,' in 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , pp. 700-707, IEEE, 2020. -
  104. [131][131] T.-Y. Lin, P. Doll´ ar, R. Girshick, K. He, B. Hariharan, and S. Belongie, 'Feature pyramid networks for object detection,' in CVPR , 2017. -
  105. [133][133] H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen, 'Axial-DeepLab: Stand-alone axial-attention for panoptic segmentation,' arXiv preprint arXiv:2003.07853 , 2020. - · arXiv
  106. [136][136] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Doll´ ar, 'Panoptic segmentation,' in CVPR , 2019. -
  107. [137][137] G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder, 'The mapillary vistas dataset for semantic understanding of street scenes,' in ICCV , 2017. -
  108. [138][138] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'ImageNet: A large-scale hierarchical image database,' in CVPR , 2009. -
  109. [139][139] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling context in referring expressions,' in ECCV , 2016. -
  110. [140][140] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, 'Generation and comprehension of unambiguous object descriptions,' in CVPR , 2016. -
  111. [141][141] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, 'Referitgame: Referring to objects in photographs of natural scenes,' in EMNLP , 2014. -
  112. [142][142] N. Parmar, A. Vaswani, J. Uszkoreit, Ł. Kaiser, N. Shazeer, A. Ku, and D. Tran, 'Image transformer,' in ICML , 2018. -
  113. [143][143] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, 'Generative pretraining from pixels,' in ICML , 2020. -
  114. [144][144] P. Esser, R. Rombach, and B. Ommer, 'Taming transformers for high-resolution image synthesis,' arXiv:2012.09841 , 2020. - · arXiv
  115. [146][146] A. K. Bhunia, S. Khan, H. Cholakkal, R. M. Anwer, F. S. Khan, and M. Shah, 'Handwriting transformers,' arXiv preprint arXiv:2104.03964 , 2021. - · arXiv
  116. [147][147] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. , 'Conditional image generation with pixelcnn decoders,' in NeurIPS , 2016. -
  117. [149][149] A. Coates, A. Ng, and H. Lee, 'An analysis of single-layer networks in unsupervised feature learning,' in AISTATS , 2011. -
  118. [150][150] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A simple framework for contrastive learning of visual representations,' arXiv preprint arXiv:2002.05709 , 2020. - · arXiv
  119. [151][151] P. Bachman, R. Hjelm, and W. Buchwalter, 'Learning representations by maximizing mutual information across views,' in NeurIPS , 2019. -
  120. [152][152] O. J. H´ enaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, 'Data-efficient image recognition with contrastive predictive coding,' arXiv preprint arXiv:1905.09272 , 2019. - · arXiv
  121. [153][153] Y. Tian, D. Krishnan, and P. Isola, 'Contrastive multiview coding,' arXiv preprint arXiv:1906.05849 , 2019. - · arXiv
  122. [155][155] A. Radford, L. Metz, and S. Chintala, 'Unsupervised representation learning with deep convolutional generative adversarial networks,' arXiv preprint arXiv:1511.06434 , 2015. - · arXiv
  123. [156][156] C. Gao, Y. Chen, S. Liu, Z. Tan, and S. Yan, 'Adversarialnas: Adversarial neural architecture search for gans,' in CVPR , pp. 56805689, 2020. -
  124. [157][157] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, 'Analyzing and improving the image quality of stylegan,' in CVPR , pp. 8110-8119, 2020. -
  125. [158][158] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, 'Generative adversarial text to image synthesis,' in ICML , 2016. -
  126. [159][159] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, 'StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks,' in ICCV , 2017. -
  127. [160][160] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, 'StackGAN++: Realistic image synthesis with stacked generative adversarial networks,' TPAMI , 2018. -
  128. [161][161] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, 'AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks,' in CVPR , 2018. -
  129. [162][162] D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' arXiv preprint arXiv:1312.6114 , 2013. - · arXiv
  130. [165][165] Z. Wang, X. Cun, J. Bao, and J. Liu, 'Uformer: A general u-shaped transformer for image restoration,' arXiv preprint arXiv:2106.03106 , 2021. - · arXiv
  131. [166][166] Z. Lu, H. Liu, J. Li, and L. Zhang, 'Efficient transformer for single image super-resolution,' arXiv preprint arXiv:2108.11084 , 2021. - · arXiv
  132. [167][167] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, 'Image super-resolution using very deep residual channel attention networks,' in ECCV , 2018. -
  133. [168][168] T. Dai, J. Cai, Y. Zhang, S. Xia, and L. Zhang, 'Second-order attention network for single image super-resolution,' in CVPR , 2019. -
  134. [169][169] B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen, 'Single image super-resolution via a holistic attention network,' in ECCV , 2020. -
  135. [171][171] Y. Tai, J. Yang, and X. Liu, 'Image super-resolution via deep recursive residual network,' in CVPR , 2017. -
  136. [172][172] W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. Huang, 'Image super-resolution via dual-state recurrent networks,' in CVPR , 2018. -
  137. [173][173] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, 'Residual dense network for image restoration,' TPAMI , 2020. -
  138. [175][175] S.-J. Park, H. Son, S. Cho, K.-S. Hong, and S. Lee, 'SRFEAT: Single image super-resolution with feature discrimination,' in ECCV , 2018. -
  139. [176][176] M. S. Sajjadi, B. Scholkopf, and M. Hirsch, 'EnhanceNet: Single image super-resolution through automated texture synthesis,' in ICCV , 2017. -
  140. [177][177] C. Ledig, L. Theis, F. Husz´ ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. , 'Photorealistic single image super-resolution using a generative adversarial network,' in CVPR , 2017. -
  141. [178][178] J. Johnson, A. Alahi, and L. Fei-Fei, 'Perceptual losses for realtime style transfer and super-resolution,' in ECCV , 2016. -
  142. [179][179] J. Ho, N. Kalchbrenner, D. Weissenborn, and T. Salimans, 'Axial attention in multidimensional transformers,' arXiv preprint arXiv:1912.12180 , 2019. - · arXiv
  143. [180][180] G. Li, N. Duan, Y. Fang, M. Gong, D. Jiang, and M. Zhou, 'Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training.,' in AAAI , 2020. -
  144. [181][181] J. Lu, D. Batra, D. Parikh, and S. Lee, 'Vilbert: Pretraining taskagnostic visiolinguistic representations for vision-and-language tasks,' in NeurIPS , 2019. -
  145. [182][182] S. Lee, Y. Yu, G. Kim, T. Breuel, J. Kautz, and Y. Song, 'Parameter efficient multimodal transformers for video representation learning,' arXiv preprint arXiv:2012.04124 , 2020. - · arXiv
  146. [183][183] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, 'VQA: Visual question answering,' in ICCV , 2015. -
  147. [184][184] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, 'From recognition to cognition: Visual commonsense reasoning,' in CVPR , 2019. -
  148. [185][185] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, 'Stacked cross attention for image-text matching,' in ECCV , 2018. -
  149. [186][186] A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi, 'A corpus for reasoning about natural language grounded in photographs,' arXiv preprint arXiv:1811.00491 , 2018. - · arXiv
  150. [187][187] J. Carreira, E. Noland, C. Hillier, and A. Zisserman, 'A short note on the kinetics-700 human action dataset,' arXiv:1907.06987 , 2019. - · arXiv
  151. [188][188] K. Soomro, A. R. Zamir, and M. Shah, 'UCF101: A dataset of 101 human actions classes from videos in the wild,' arXiv preprint arXiv:1212.0402 , 2012. - · arXiv
  152. [189][189] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, 'Audio set: An ontology and human-labeled dataset for audio events,' in ICASSP , 2017. -
  153. [190][190] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, 'Hollywood in homes: Crowdsourcing data collection for activity understanding,' in ECCV , 2016. -
  154. [191][191] H. Tan and M. Bansal, 'Vokenization: Improving language understanding with contextualized, visual-grounded supervision,' in EMNLP , 2020. -
  155. [192][192] W. Hao, C. Li, X. Li, L. Carin, and J. Gao, 'Towards learning a generic agent for vision-and-language navigation via pretraining,' in CVPR , 2020. -
  156. [193][193] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, 'Improving vision-and-language navigation with image-text pairs from the web,' arXiv preprint arXiv:2004.14973 , 2020. - · arXiv
  157. [194][194] K. Chen, J. K. Chen, J. Chuang, M. V´ azquez, and S. Savarese, 'Topological planning with transformers for vision-andlanguage navigation,' arXiv preprint arXiv:2012.05292 , 2020. - · arXiv
  158. [195][195] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning transferable visual models from natural language supervision,' Image , vol. 2, p. T2, 2021. -
  159. [196][196] P. Sharma, N. Ding, S. Goodman, and R. Soricut, 'Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,' in ACL , 2018. -
  160. [197][197] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, 'Unified vision-language pre-training for image captioning and vqa,' in AAAI , vol. 34, pp. 13041-13049, 2020. -
  161. [198][198] C. Sun, F. Baradel, K. Murphy, and C. Schmid, 'Learning video representations using contrastive bidirectional transformer,' arXiv preprint arXiv:1906.05743 , 2019. - · arXiv
  162. [199][199] C. Alberti, J. Ling, M. Collins, and D. Reitter, 'Fusion of detected objects in text for visual question answering,' in EMNLP , 2019. -
  163. [200][200] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. , 'Visual genome: Connecting language and vision using crowdsourced dense image annotations,' IJCV , 2017. -
  164. [201][201] V. Ordonez, G. Kulkarni, and T. L. Berg, 'Im2text: Describing images using 1 million captioned photographs,' in NeurIPS , 2011. -
  165. [202][202] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, 'Vinvl: Revisiting visual representations in visionlanguage models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5579-5588, 2021. -
  166. [203][203] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion, 'Mdetr-modulated detection for end-to-end multimodal understanding,' arXiv preprint arXiv:2104.12763 , 2021. - · arXiv
  167. [205][205] M. Li and L. Sigal, 'Referring transformer: A one-step approach to multi-task visual grounding,' arXiv preprint arXiv:2106.03089 , 2021. - · arXiv
  168. [206][206] Y. Du, Z. Fu, Q. Liu, and Y. Wang, 'Visual grounding with transformers,' arXiv preprint arXiv:2105.04281 , 2021. - · arXiv
  169. [207][207] S. Ging, M. Zolfaghari, H. Pirsiavash, and T. Brox, 'COOT: Cooperative hierarchical transformer for video-text representation learning,' arXiv preprint arXiv:2011.00597 , 2020. - · arXiv
  170. [208][208] H. Seong, J. Hyun, and E. Kim, 'Video multitask transformer network,' in ICCV Workshops , pp. 0-0, 2019. -
  171. [209][209] Y. Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia, 'End-to-end video instance segmentation with transformers,' arXiv preprint arXiv:2011.14503 , 2020. - · arXiv
  172. [210][210] L. Zhou, Y. Zhou, J. Corso, R. Socher, and C. Xiong, 'End-toend dense video captioning with masked transformer,' in CVPR , 2018. -
  173. [211][211] D. Neimark, O. Bar, M. Zohar, and D. Asselmann, 'Video transformer network,' arXiv preprint arXiv:2102.00719 , 2021. - · arXiv
  174. [212][212] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Luˇ ci´ c, and C. Schmid, 'Vivit: A video vision transformer,' arXiv preprint arXiv:2103.15691 , 2021. - · arXiv
  175. [213][213] G. Bertasius, H. Wang, and L. Torresani, 'Is space-time attention all you need for video understanding?,' in Proceedings of the International Conference on Machine Learning (ICML) , July 2021. -
  176. [214][214] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles, 'Dense-captioning events in videos,' in ICCV , pp. 706-715, 2017. -
  177. [215][215] L. Zhou, C. Xu, and J. Corso, 'Towards automatic learning of procedures from web instructional videos,' in AAAI , vol. 32, 2018. -
  178. [216][216] C. Plizzari, M. Cannici, and M. Matteucci, 'Spatial temporal transformer network for skeleton-based action recognition,' arXiv preprint arXiv:2008.07404 , 2020. - · arXiv
  179. [217][217] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, 'NTU RGB+D: A large scale dataset for 3d human activity analysis,' in CVPR , 2016. -
  180. [218][218] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot, 'NTU RGB+D 120: A large-scale benchmark for 3d human activity understanding,' TPAMI , 2019. -
  181. [220][220] J. Wang, G. Bertasius, D. Tran, and L. Torresani, 'Long-short temporal contrastive learning of video transformers,' arXiv preprint arXiv:2106.09212 , 2021. - · arXiv
  182. [221][221] L. Yang, Y. Fan, and N. Xu, 'Video instance segmentation,' in ICCV , pp. 5188-5197, 2019. -
  183. [222][222] G. Bertasius and L. Torresani, 'Classifying, segmenting, and tracking object instances in video with mask propagation,' in CVPR , pp. 9739-9748, 2020. -
  184. [223][223] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, et al. , 'Metadataset: A dataset of datasets for learning to learn from few examples,' in ICLR , 2020. -
  185. [224][224] T. N. Kipf and M. Welling, 'Semi-supervised classification with graph convolutional networks,' arXiv preprint arXiv:1609.02907 , 2016. - · arXiv
  186. [225][225] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, 'Deep sets,' in NeurIPS , 2017. -
  187. [227][227] H. Edwards and A. Storkey, 'Towards a neural statistician,' arXiv preprint arXiv:1606.02185 , 2016. - · arXiv
  188. [228][228] J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, 'Set transformer: A framework for attention-based permutationinvariant neural networks,' in ICML , 2019. -
  189. [229][229] J. Lee, Y. Lee, and Y. W. Teh, 'Deep amortized clustering,' arXiv preprint arXiv:1909.13433 , 2019. - · arXiv
  190. [230][230] H. Zhao, L. Jiang, J. Jia, P. Torr, and V. Koltun, 'Point transformer,' arXiv preprint arXiv:2012.09164 , 2020. - · arXiv
  191. [231][231] M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, and S.-M. Hu, 'Pct: Point cloud transformer,' arXiv preprint arXiv:2012.09688 , 2020. - · arXiv
  192. [232][232] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, '3D ShapeNets: A deep representation for volumetric shapes,' in CVPR , 2015. -
  193. [233][233] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, 'ShapeNet: An information-rich 3d model repository,' arXiv preprint arXiv:1512.03012 , 2015. - · arXiv
  194. [234][234] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, 'Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments,' TPAMI , 2013. -
  195. [235][235] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll, 'Recovering accurate 3d human pose in the wild using imus and a moving camera,' in ECCV , 2018. -
  196. [236][236] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox, 'FreiHAND: A dataset for markerless capture of hand pose and shape from single rgb images,' in ICCV , 2019. -
  197. [239][239] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, 'From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,' TACL , 2014. -
  198. [240][240] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, 'Making the v in vqa matter: Elevating the role of image understanding in visual question answering,' in CVPR , 2017. -
  199. [241][241] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, 'Flickr30k entities: Collecting regionto-phrase correspondences for richer image-to-sentence models,' in ICCV , 2015. -
  200. [242][242] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, 'PointNet++: Deep hierarchical feature learning on point sets in a metric space,' NeurIPS , 2017. -

200 of 230 shown

Cited by

Sorted by citations

Authors
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M.
Affiliations
  • Australian National University (AU)
  • Monash University (AU)
  • Inception Institute of Artificial Intelligence (AE)
  • Linköping University (SE)
  • University of Central Florida (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
54
Issue
10s
Pages
1-41
ISSN
0360-0300 (ISSN-L)
DOI
10.48550/arxiv.2101.01169
arXiv
2101.01169
OpenAlex
W3119997354
Cited by
3,594
References
0
Citations by year
2026: 673 · 2025: 1005 · 2024: 825 · 2023: 633 · 2022: 328 · 2021: 130
FWCI
228.23
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
vision transformer, self-attention, pre-training, image classification, object detection, human action recognition, semantic segmentation, generative models, visual question answering, visual reasoning, visual grounding, activity recognition, image super-resolution, image enhancement, image colorization, point cloud classification, point cloud segmentation, multimodal processing, long-range dependencies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review