The DOI matches its Crossref record; 181 of the 181 references it cites were found in a registry or an index.
A Survey of Visual Transformers
Open accessAbstract
Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01155 · Cited by 9,641 · cited 1×
“Inspired by the great success of the self-attention mechanism in NLP [1], [13], some CNN-based models attempted to capture the long-range dependencies via an additional self-attention layer in either spatial [14]–[16] or channel level [17]–[19], while others try to replace the traditional convolutions entirely with gl…”
BackgroundMasked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Deformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513 · cited 1×
“Deformable DETR: Following their previous work [120], Zhu et al.”
MethodCascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
FCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 1×
“Compared with previous sequence transduction models [50], [51], such original Transformer inherits the encoder-decoder structure but discards the recurrence and convolutions entirely by using multi-head attention mechanisms and point-wise feed-forward networks.”
MethodXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“Without extra auxiliary training, Unified VLP only adopts MLM during pre-training and attains superior results on Visual Question Answering (VQA) [204] and Visual Captioning (VC) [205] tasks.”
ResultCBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · arXiv · doi:10.48550/arxiv.1807.06521 · Cited by 6,261 · cited 1×
“…to capture the long-range dependencies through adding a self-attention layer at either spatial level [14], [15], [16] or channel level [17], [18], [19], while others try to replace the traditional convolutions entirely with the global [20] or local self-attention blocks [21], [22], [23],…”
BackgroundDeep High-Resolution Representation Learning for Human Pose Estimation
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00584 · Cited by 5,984
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 10×
“Combining masked modeling and contrastive learning pre-training strategy, Data2Vec [152] proposes a self-distilled network treating the masked features as a type of data augmentation, whose structure is analogous to DINO (2021) [73].”
BackgroundMethodInfluentialObject Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 16×
“T2T-ViT [64], PVT [42], CvT [37] and PiT [65] share a motivation that transferring the hierarchical structure into Transformer but they perform downsampling differently.”
BackgroundMethodResultInfluentialVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 3×
“The most recent dominant models are the self-supervised Transformers, which are pre-trained over sufficient datasets and then fine-tuned over a small sample set for a given downstream task [2], [3], [4], [5], [6], [7], [8], [9].”
BackgroundMethodTransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 2×
“TransUNet [98] is the first for medical image segmentation.”
BackgroundSelf-citationRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 7×
“It can be viewed as either a variant of SETR with MLA decoder [76], or a hybrid model of U-Net [128] and Transformer.”
BackgroundMethodInfluentialSelf-citationTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Zhao, H., Jiang, L., Jia, J., Torr, P., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01595 · Cited by 2,476
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,444
Pvt v2: Improved baselines with pyramid vision transformer
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0274-8 · Cited by 2,439 · cited 2×
“For example, the hierarchical structure, such as PVT [41], [65], constructs the visual Transformer as a high-to-low resolution process to learn multiscale features.”
BackgroundZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 5×
“T2T-ViT [64], PVT [42], CvT [37] and PiT [65] share a motivation that transferring the hierarchical structure into Transformer but they perform downsampling differently.”
BackgroundMethodInfluentialLXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · cited 1×
“Additionally, the visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], [32], tracking [33], generation [34], and enhancement [35].”
MethodSelf-citationGCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond
Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00246 · Cited by 2,307
Unified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · cited 1×
“In detail, a set of unified queries is applied to parallelly perform multiple representation learning: classification, box TABLE IV COMPARISON BETWEEN UPERNET [133] AND TRANSFORMER-BASED MODEL ON ADE20K VAL.”
MethodSelf-citationViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Guo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., & Hu, S.-M. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0229-5 · Cited by 2,100
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,074
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
TSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,051
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
ResNeSt: Split-Attention Networks
Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.08955 · Cited by 1,809
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01422 · Cited by 1,617
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., & Lu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00803 · Cited by 1,588 · cited 2×
“Additionally, the visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], [32], tracking [33], generation [34], and enhancement [35].”
BackgroundMethodOscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,542
Relation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Self-citationHan, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · cited 8×
“Twins [60] employs a spatially separable self-attention mechanism, similar to depthwise convolution [172] or windowwise TNT [59], to model the local–global representation.”
BackgroundMethodInfluentialSelf-citationBottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267
Going deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259
MetaFormer is Actually What You Need for Vision
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01055 · Cited by 1,255
Attention Augmented Convolutional Networks
Bello, I., Zoph, B., Le, Q., Vaswani, A., & Shlens, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00338 · Cited by 1,205
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 1×
“Following [137], VisualBERT [140] extends such a single-stream framework for various image–text tasks and adds a segment embedding to distinguish between textual and visual tokens.”
MethodVideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K., & Schmid, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,172
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01325 · Cited by 1,102
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085 · cited 1×
“…long-range dependencies through adding a self-attention layer at either spatial level [14], [15], [16] or channel level [17], [18], [19], while others try to replace the traditional convolutions entirely with the global [20] or local self-attention blocks [21], [22], [23], [24], [25], [26], [27].”
BackgroundConditional DETR for Fast Training Convergence
Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., & Wang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00363 · Cited by 1,055
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2019) · arXiv · doi:10.48550/arxiv.1912.11370 · Cited by 1,039
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_29 · Cited by 1,039
Exploring Self-Attention for Image Recognition
Zhao, H., Jia, J., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01009 · Cited by 1,002
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers
Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., & Tai, C.-L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00116 · Cited by 990 · cited 3×
“Camera-Based: MonoDTR [120], MonoDETR [121], DETR3D [122], TransFusion [123].”
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977 · cited 4×
“have demonstrated the efficiency of selfattention block [24] without the help from CNNs, such pure attention model is still inferior to the State-of-The-Art (SoTA) CNN models on the prevailing benchmarks.”
BackgroundMethodResultInfluentialViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951 · cited 5×
“Oscar [144], Unified [145], ViLT [146], VinVl [147], CLIP [148], DALL-E [149],”
BackgroundMethodInfluentialVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925 · cited 11×
“• The convolutional patchify stem (ViTc [59]) and early convolutional stage (CoAtNet [40]) can significantly boost the accuracy of the Transformers, especially for large models.”
BackgroundMethodInfluentialVinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00553 · Cited by 907
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864
Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2104.13840 · Cited by 817
Self-citationPoint-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01871 · Cited by 801
ConViT: improving vision transformers with soft convolutional inductive biases
d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., & Sagun, L. (2022) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac9830 · Cited by 782
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 729
End-to-End Video Instance Segmentation with Transformers
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00863 · Cited by 679
Self-citationRethinking Spatial Dimensions of Vision Transformers
Heo, B., Yun, S., Han, D., Chun, S., Choe, J., & Oh, S. J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01172 · Cited by 640
Attention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 635
Incorporating Convolution Designs into Visual Transformers
Yuan, K., Guo, S., Liu, Z., Zhou, A., Yu, F., & Wu, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00062 · Cited by 611
Masked Autoencoders for Point Cloud Self-supervised Learning
Pang, Y., Wang, W., Tay, F. E. H., Liu, W., Tian, Y., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20086-1_35 · Cited by 605
BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation
Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., & Yan, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00860 · Cited by 601
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Prakash, A., Chitta, K., & Geiger, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00700 · Cited by 584
Local Relation Networks for Image Recognition
Hu, H., Zhang, Z., Xie, Z., & Lin, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00356 · Cited by 570
SOLOv2: Dynamic and Fast Instance Segmentation
Wang, X., Zhang, R., Kong, T., Li, L., & Shen, C. (2020) · arXiv · doi:10.48550/arxiv.2003.10152 · Cited by 549
PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers
Yu, X., Rao, Y., Wang, Z., Liu, Z., Lu, J., & Zhou, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01227 · Cited by 544
UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
Dai, Z., Cai, B., Lin, Y., & Chen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00165 · Cited by 520
Voxel Transformer for 3D Object Detection
Mao, J., Xue, Y., Niu, M., Bai, H., Feng, J., Liang, X., Xu, H., & Xu, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00315 · Cited by 520
An End-to-End Transformer Model for 3D Object Detection
Misra, I., Girdhar, R., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00290 · Cited by 499
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Wang, H., Zhu, Y., Adam, H., Yuille, A., & Chen, L.-C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00542 · Cited by 497
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 6×
“Based on the assumption of positional encoding [57] and the observation in [173], ResT [57] and CPVT [56] try to adapt the inherent positional information of the convolution to the arbitrary size of inputs instead of interpolating the positional encoding.”
BackgroundMethodInfluentialAnchor DETR: Query Design for Transformer-Based Detector
Wang, Y., Zhang, X., Yang, T., & Sun, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20158 · Cited by 482
Self-citation3D Object Detection with Pointformer
Pan, X., Xia, Z., Song, S., Li, L. E., & Huang, G. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00738 · Cited by 464
Dynamic DETR: End-to-End Object Detection with Dynamic Attention
Dai, X., Chen, Y., Yang, J., Zhang, P., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00298 · Cited by 456 · cited 3×
“Thus, the decoder is necessary but it requires more spatial prior [77], [81]– [84], [86], [87] owing to its slow convergence [88].”
BackgroundMethodScaling Local Self-Attention for Parameter Efficient Visual Backbones
Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., & Shlens, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01270 · Cited by 453
DeepViT: Towards Deeper Vision Transformer
Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2103.11886 · Cited by 398
TransVG: End-to-End Visual Grounding with Transformers
Deng, J., Yang, Z., Chen, T., Zhou, W., & Li, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00179 · Cited by 372
LocalViT: Analyzing Locality in Vision Transformers
Li, Y., Zhang, K., Cao, J., Timofte, R., Magno, M., Benini, L., & Van Goo, L. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342025 · Cited by 372
Visual Transformers: Token-based Image Representation and Processing for Computer Vision
Wu, B., Xu, C., Dai, X., Wan, A., Zhang, P., Yan, Z., Tomizuka, M., Gonzalez, J., Keutzer, K., & Vajda, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03677 · Cited by 357
Rethinking Transformer-based Set Prediction for Object Detection
Sun, Z., Cao, S., Yang, Y., & Kitani, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00359 · Cited by 356
Fast Convergence of DETR with Spatially Modulated Co-Attention
Gao, P., Zheng, M., Wang, X., Dai, J., & Li, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00360 · Cited by 345
SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer
Xiang, P., Wen, X., Liu, Y.-S., Cao, Y.-P., Wan, P., Zheng, W., & Han, Z. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00545 · Cited by 344
Self-citationMulti-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
Zhang, P., Dai, X., Yang, J., Xiao, B., Yuan, L., Zhang, L., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00299 · Cited by 330
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
Chen, X., Zhang, T., Wang, Y., Wang, Y., & Zhao, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00022 · Cited by 328
Self-citationFang, Y., Yang, S., Wang, X., Li, Y., Fang, C., Shan, Y., Feng, B., & Liu, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00683 · Cited by 320
UniT: Multimodal Multitask Learning with a Unified Transformer
Hu, R., & Singh, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00147 · Cited by 303
Group-Free 3D Object Detection via Transformers
Liu, Z., Zhang, Z., Cao, Y., Hu, H., & Tong, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00294 · Cited by 298
Embracing Single Stride 3D Object Detector with Sparse Transformer
Fan, L., Pang, Z., Zhang, T., Wang, Y.-X., Zhao, H., Wang, F., Wang, N., & Zhang, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00827 · Cited by 293
Self-citationVOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 288
COTR: Correspondence Transformer for Matching Across Images
Jiang, W., Trulls, E., Hosang, J., Tagliasacchi, A., & Yi, K. M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00615 · Cited by 282
Improving 3D Object Detection with Channel-wise Transformer
Shenga, H., Cai, S., Liu, Y., Deng, B., Huang, J., Hua, X.-S., & Zhao, M.-J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00274 · Cited by 282
Self-citationMasked Discrimination for Self-supervised Learning on Point Clouds
Liu, H., Cai, M., & Lee, Y. J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20086-1_38 · Cited by 280
Zhang, D., Zhang, H., Tang, J., Wang, M., Hua, X., & Sun, Q. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58604-1_20 · Cited by 277
Self-Supervised Learning with Swin TransformersPreprint
Xie, Z., Lin, Y., Yao, Z., Zhang, Z., Dai, Q., Cao, Y., & Hu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04553 · Cited by 266
Focal Self-attention for Local-Global Interactions in Vision Transformers
Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00641 · Cited by 256
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07074 · Cited by 254
How Much Position Information Do Convolutional Neural Networks Encode?
Islam, M. A., Jia, S., & Bruce, N. D. B. (2020) · arXiv · doi:10.48550/arxiv.2001.08248 · Cited by 253
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes
Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., & Guibas, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_25 · Cited by 253
Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds
He, C., Li, R., Li, S., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00823 · Cited by 237
Zhang, Y., He, N., Yang, J., Li, Y., Wei, D., Huang, Y., Zhang, Y., He, Z., & Zheng, Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-16443-9_11 · Cited by 236
Self-citationMonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer
Huang, K.-C., Wu, T.-H., Su, H.-T., & Hsu, W. H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00398 · Cited by 234
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03555 · Cited by 231
Multi-View Transformer for 3D Visual Grounding
Huang, S., Chen, Y., Jia, J., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01508 · Cited by 226
Convolution in Convolution for Network in Network
Pang, Y., Sun, M., Jiang, X., & Li, X. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2676130 · Cited by 222
End-to-End Object Detection with Adaptive Clustering TransformerPreprint
Zheng, M., Gao, P., Zhang, R., Li, K., Wang, X., Li, H., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09315 · Cited by 221
Park, C., Jeong, Y., Cho, M., & Park, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01644 · Cited by 217
MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
Zhang, R., Qiu, H., Wang, T., Guo, Z., Cui, Z., Qiao, Y., Li, H., & Gao, P. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00840 · Cited by 208
Why Deep Learning Works: A Manifold Disentanglement Perspective
Brahma, P. P., Wu, D., & She, Y. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2496947 · Cited by 190
Efficient DETR: Improving End-to-End Object Detector with Dense Prior
Yao, Z., Ai, J., Li, B., & Zhang, C. (2021) · arXiv · doi:10.48550/arxiv.2104.01318 · Cited by 190
TubeDETR: Spatio-Temporal Video Grounding with Transformers
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01595 · Cited by 187
3DCTN: 3D Convolution-Transformer Network for Point Cloud Classification
Lu, D., Xie, Q., Gao, K., Xu, L., & Li, J. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3198836 · Cited by 166
PnP-DETR: Towards Efficient Visual Analysis with Transformers
Wang, T., Yuan, L., Chen, Y., Feng, J., & Yan, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00462 · Cited by 164
All Tokens Matter: Token Labeling for Training Better Vision Transformers
Jiang, Z., Hou, Q., Yuan, L., Zhou, D., Shi, Y., Jin, X., Wang, A., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10858 · Cited by 158 · cited 10×
“1) Compared with the most structure-improved methods, the basic training strategies, such as DeiT [40] and LV-ViT [43], are more universal for various models.”
BackgroundMethodInfluentialISTR: End-to-End Instance Segmentation with TransformersPreprint
Hu, J., Cao, L., Lu, Y., Zhang, S., Wang, Y., Li, K., Huang, F., Shao, L., & Ji, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00637 · Cited by 145
Self-citationTransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
He, D., Zhao, Y., Luo, J., Hui, T., Huang, S., Zhang, A., & Liu, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475397 · Cited by 136
Multi-view 3D Reconstruction with Transformers
Wang, D., Cui, X., Chen, X., Zou, Z., Shi, T., Salcudean, S. E., Wang, Z. J., & Ward, R. K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00567 · Cited by 128
Attention-Based Transformers for Instance Segmentation of Cells in Microstructures
Prangemeier, T., Reich, C., & Koeppl, H. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313305 · Cited by 118
Visual Grounding with Transformers
Du, Y., Fu, Z., Liu, Q., & Wang, Y. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859880 · Cited by 115
Self-citationMultiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)
Hou, Y., & Zheng, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475310 · Cited by 110
Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding
Jiang, H., Lin, Y., Han, D., Song, S., & Huang, G. M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01507 · Cited by 87
Refiner: Refining Self-attention for Vision TransformersPreprint
Zhou, D., Shi, Y., Kang, B., Yu, W., Jiang, Z., Li, Y., Jin, X., Hou, Q., & Feng, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03714 · Cited by 85
Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
Long, X., Liu, L., Li, W., Theobalt, C., & Wang, W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00816 · Cited by 83
Multi-view Analysis of Unregistered Medical Images Using Cross-View Transformers
van Tulder, G., Tong, Y., & Marchiori, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_10 · Cited by 78
MVT: Multi-view Vision Transformer for 3D Object Recognition
Chen, S., Yu, T., & Li, P. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.71 · Cited by 73
Song, D., Fu, B., Li, F., Xiong, J., He, J., Zhang, X., & Qiao, Y. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3077484 · Cited by 70
Global and Local Knowledge-Aware Attention Network for Action Recognition
Zheng, Z., An, G., Wu, D. O., & Ruan, Q. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978613 · Cited by 59
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., & Zhang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.01305 · Cited by 57
Vision Transformers with Patch Diversification
Gong, C., Wang, D., Li, M., Chandra, V., & Liu, Q. (2021) · arXiv · doi:10.48550/arxiv.2104.12753 · Cited by 41
Liu, Y., Zhang, Y., Wang, Y., Zhang, Y., Jiang, T., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01491 · Cited by 23
Self-citationDeep Point Cloud ReconstructionPreprint
Choe, J., Joung, B., Rameau, F., Park, J., & Kweon, I. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11704 · Cited by 18
HRViT: Multi-Scale High-Resolution Vision TransformerPreprint
Gu, J., Kwon, H., Wang, D., Ye, W., Li, M., Chen, Y., Lai, L., Chandra, V., & Pan, D. Z. (2021) · arXiv (Cornell University) · Cited by 8
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
Semantic segmentation using Vision Transformers: A survey
Thisanke, H., Deshan, C., Chamith, K., Seneviratne, S., Vidanaarachchi, R., & Herath, D. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106669 · Cited by 288
Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation
Azad, R., Niggemeier, L., Hüttemann, M., Kazerouni, A., Aghdam, E. K., Velichko, Y., Bagci, U., & Merhof, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00132 · Cited by 254
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174
“The rapid advancements in LLMs present a viable solution to these challenges, thanks to their potent natural language understanding and reasoning abilities, coupled with recent progress in incorporating visual data processing [79].”
Metadata
- Authors
- Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., & He, Z.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2024-06
- Volume
- 35
- Issue
- 6
- Pages
- 7478-7498
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W4226512186
- Cited by
- 549
- References
- 181
- FWCI
- 47.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.