Attention mechanisms in computer vision: A survey
Open accessAbstract
Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multimodal tasks, and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention, and branch attention; a related repository https://github.com/MenghaoGuo/Awesome-Vision-Attentions is dedicated to collecting related work. We also suggest future directions for attention mechanism research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evolution of LMs has led to the proliferation of various attention mechanisms.
“The evolution of LMs has led to the proliferation of various attention mechanisms (Guo et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Guo et al. 2022
The Abstract and Conclusions describe the development of various attention mechanisms in computer vision. Section 4.5 notes the success of attention-based models in natural language processing, but the paper does not say that the evolution of language models caused the proliferation of attention mechanisms. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6× · see citing works
“ResNeSt [115] also adopts this attention mechanism to improve the CNN backbone in a more general way, giving excellent results on ResNet [146] and ResNeXt [176].”
BackgroundMethodResultInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4× · see citing works
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9× · see citing works
“Given an input feature map X and the previous time step LSTM hidden state ht−1 ∈ Rd, a spatial attention mechanism pays more attention to the semantically useful regions, guided by LSTM hidden state ht−1.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3× · see citing works
“SGD [104] and Adam [72] are well-suited for optimizing convolutional neural networks.”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2× · see citing works
“ViT demonstrates that a pure attention-based network can achieve better results than a convolutional neural network especially for large datasets such as JFT-300 [164] and ImageNet-21K [165].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 3× · see citing works
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 4× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundInfluentialSelf-citationCBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Large-scale attention-based pre-trained models have had great success in natural language processing [85], [179].”
BackgroundEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · cited 1× · see citing works
“ANN [68] suggests that using all positional features as key and vectors is redundant and adopts spatial pyramid pooling [157, 158] to obtain a few representative key and value features to use instead, to reduce computation.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · cited 1× · see citing works
“ViT stacks a number of MHA layers with fully connected layers, layer normalization [162] and the GELU [163] activation function.”
BackgroundControl of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881 · cited 1× · see citing works
“M ETHODS for diverting attention to the most important regions of an image and disregarding irrelevant parts are called attention mechanisms; the human visual system uses one [1], [2], [3], [4] to assist in analyzing and understanding complex scenes efficiently and effectively.”
BackgroundDecoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 2× · see citing works
“ResNeSt [115] also adopts this attention mechanism to improve the CNN backbone in a more general way, giving excellent results on ResNet [146] and ResNeXt [176].”
ResultSqueeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 12× · see citing works
“[128] introduced self-attention into computer vision.”
BackgroundMethodInfluentialA model of saliency-based visual attention for rapid scene analysis
Itti, L., Christof Koch, A., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,289 · cited 1× · see citing works
“M ETHODS for diverting attention to the most important regions of an image and disregarding irrelevant parts are called attention mechanisms; the human visual system uses one [1], [2], [3], [4] to assist in analyzing and understanding complex scenes efficiently and effectively.”
BackgroundECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01155 · Cited by 9,641 · cited 6× · see citing works
“[37] proposed the efficient channel attention (ECA) block which instead uses a 1D convolution to determine the interaction between channels, instead of dimensionality reduction.”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Deformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513 · cited 6× · see citing works
“[7] proposed deformable convolutional networks (deformable ConvNets) to be invariant to geometric transformations, but they pay attention to the important regions in a different manner.”
BackgroundMethodInfluentialDual Attention Network for Scene Segmentation
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00326 · Cited by 7,351 · see citing works
Self-citationCoordinate Attention for Efficient Mobile Network Design
Hou, Q., Zhou, D., & Feng, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01350 · Cited by 6,961 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · cited 2× · see citing works
“Self-attention was firstly proposed in [33] and rapidly provided great advances in the field of natural language processing [33], [38], [39].”
BackgroundImage Super-Resolution Using Very Deep Residual Channel Attention Networks
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_18 · Cited by 6,003 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 2× · see citing works
“Recently, MoCoV3 [84], DINO [180], BEiT [85] and MAE [169] have demonstrated that attention-based models are also well suited to visual tasks.”
BackgroundAttention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458 · cited 2× · see citing works
“[19] proposed a simple and yet effective mechanism, the attention gate (AG), to focus on targeted regions while suppressing feature activations in irrelevant regions.”
BackgroundPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 3× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
Background(1999) · Cited by 5,147 · see citing works
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
Self-citationProceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · cited 6× · see citing works
“ResNeSt [115] also adopts this attention mechanism to improve the CNN backbone in a more general way, giving excellent results on ResNet [146] and ResNeXt [176].”
BackgroundMethodResultInfluentialResidual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073 · cited 10× · see citing works
“Attention mechanisms have become an indispensable technique in the field of computer vision in the era of deep learning.”
BackgroundMethodInfluentialRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
BackgroundWu, Y., & He, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_1 · Cited by 3,601 · see citing works
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 2× · see citing works
“Three more specific surveys [141], [142], [143] summarize the development of visual transformers while our paper reviews attention mechanisms in vision more generally, not just selfattention mechanisms.”
BackgroundGaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 2× · see citing works
“normalization [163], and the GELU [164] activation function.”
BackgroundLi, X., Wang, W., Hu, X., & Yang, J. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00060 · Cited by 3,416 · cited 2× · see citing works
“[114] to propose an automatic selection operation called selective kernel (SK) convolution.”
BackgroundRecurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · cited 7× · see citing works
“[88] proposed a deep recurrent network, similar to RAM [31], capable of processing a multi-resolution crop of the input image, called a glimpse, for multiple object recognition task.”
BackgroundMethodInfluentialTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1× · see citing works
“Self-attention was proposed and has had great success in the field of natural language processing (NLP) [33], [38], [39], [149], [150], [151], [152].”
BackgroundSpatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_23 · Cited by 3,034 · cited 1× · see citing works
“ANN [68] suggests that using all positional features as key and vectors is redundant and adopts spatial pyramid pooling [157, 158] to obtain a few representative key and value features to use instead, to reduce computation.”
MethodDeformable ConvNets V2: More Deformable, Better Results
Zhu, X., Hu, H., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00953 · Cited by 2,943 · cited 3× · see citing works
“features is the major characteristic of the second phase; DCNs [7, 36] are representative works.”
BackgroundZhao, H., Jiang, L., Jia, J., Torr, P. H. S., & Koltun, V. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01595 · Cited by 2,847 · see citing works
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,649 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundSelf-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · cited 1× · see citing works
“They also explore the behavior of positional embedding and show that relative positional embeddings [161] are suitable.”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · cited 2× · see citing works
“ViT demonstrates that a pure attention-based network can achieve better results than a convolutional neural network especially for large datasets such as JFT-300 [164] and ImageNet-21K [165].”
BackgroundTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundOn the momentum term in gradient descent learning algorithms
Ning Qian, A. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,318 · cited 1× · see citing works
“SGD [181] and Adam [182] are well-suited for optimizing convolutional neural networks.”
BackgroundPre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Self-citationGCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond
Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00246 · Cited by 2,307 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundGuo, M.-H., Cai, J.-X., Liu, Z.-N., Mu, T.-J., Martin, R. R., & Hu, S.-M. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0229-5 · Cited by 2,100 · cited 4× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundInfluentialSelf-citationMLP-Mixer: An all-MLP Architecture for Vision
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01601 · Cited by 2,087 · see citing works
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.667 · Cited by 2,073 · cited 7× · see citing works
“SCACNN [17] CVPR2017 ICap a)compute channel attention map and spatial attention map using visual features and previous hidden state b) stack channel attention and spatial attention in serial a) focus the network on discriminative regions b) emphasize important channels c) capture global information d) avoid high compu…”
BackgroundMethodInfluentialProceedings of The 32nd International Conference on Machine Learning
Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015) · Cited by 2,020 · see citing works
Second-Order Attention Network for Single Image Super-Resolution
Dai, T., Cai, J., Zhang, Y., Xia, S.-T., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01132 · Cited by 2,010 · cited 1× · see citing works
“segmentation [156, 40], face recognition [145, 126], person re-identification [78, 14], action recognition [128, 37], few-show learning [103, 58], medical image processing [99, 47], image generation [46, 159], pose estimation [27], super resolution [30, 164], 3D vision [137, 48], and multi-modal task [116, 141].”
BackgroundResNeSt: Split-Attention Networks
Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J. W., Manmatha, R., et al. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00309 · Cited by 1,965 · cited 1× · see citing works
“ResNeSt [115] also adopts this attention mechanism to improve the CNN backbone in a more general way, giving excellent results on ResNet [146] and ResNeXt [176].”
ResultAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works
Self-citationDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · see citing works
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network
Peng, C., Zhang, X., Yu, G., Luo, G., & Sun, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.189 · Cited by 1,750 · see citing works
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · cited 2× · see citing works
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundAn Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · cited 3× · see citing works
“Recently, MoCoV3 [84], DINO [180], BEiT [85] and MAE [169] have demonstrated that attention-based models are also well suited to visual tasks.”
BackgroundDynamic Convolution: Attention Over Convolution Kernels
Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., & Liu, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01104 · Cited by 1,582 · cited 1× · see citing works
“[116] proposed dynamic convolution, a novel operator design that increases representational power with negligible additional computational cost and does not change the width or depth of the network in parallel with CondConv [174].”
BackgroundSelf-citationRelation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524 · see citing works
FcaNet: Frequency Channel Attention Networks
Qin, Z., Zhang, P., Wu, F. Q., & Li, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00082 · Cited by 1,513 · cited 3× · see citing works
“[57] rethought global information captured from the viewpoint of compression and analysed global average pooling in the frequency domain.”
BackgroundContext Encoding for Semantic Segmentation
Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., & Agrawal, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00747 · Cited by 1,510 · see citing works
Harmonious Attention Network for Person Re-identification
Li, W., Zhu, X., & Gong, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00243 · Cited by 1,497 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · cited 1× · see citing works
“For instance, GoogleNet [178] conforms to the above formula, but does not belong to the attention mechanisms.”
BackgroundFu, J., Zheng, H., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.476 · Cited by 1,459 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
Training Very Deep NetworksPreprint
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,453 · cited 3× · see citing works
“Combine different branches Highway Network [113] NeurIPS2015 Cls linear layer → sigmoid (A) (0, 1) S (I), (II)”
BackgroundInfluentialProceedings of the 2nd International Conference on Neural Information Processing Systems
Touretzky, D. S. (1989) · Cited by 1,413 · see citing works
DRAW: A Recurrent Neural Network For Image GenerationPreprint
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 1,394 · cited 4× · see citing works
“RAM pioneered the use of RNNs for visual attention, and was followed by many other RNN-based methods [21], [35], [88].”
BackgroundMethodInfluentialHan, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392 · cited 4× · see citing works
“Recently, MoCoV3 [84], DINO [180], BEiT [85] and MAE [169] have demonstrated that attention-based models are also well suited to visual tasks.”
BackgroundInfluentialEye movements in natural behavior
Hayhoe, M., & Ballard, D. H. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.02.009 · Cited by 1,321 · cited 1× · see citing works
“M ETHODS for diverting attention to the most important regions of an image and disregarding irrelevant parts are called attention mechanisms; the human visual system uses one [1], [2], [3], [4] to assist in analyzing and understanding complex scenes efficiently and effectively.”
BackgroundAttention Augmented Convolutional Networks
Bello, I., Zoph, B., Le, Q. V., Vaswani, A., & Shlens, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00338 · Cited by 1,320 · cited 2× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
MethodRotate to Attend: Convolutional Triplet Attention Module
Misra, D., Nalamada, T., Arasanipalai, A. U., & Hou, Q. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00318 · Cited by 1,259 · cited 9× · see citing works
“Attention mechanisms have become an indispensable technique in the field of computer vision in the era of deep learning.”
BackgroundInfluentialGoing deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259 · cited 2× · see citing works
“Following ViT, many transformer-based architectures such as PCT [27], IPT [79], T2T-ViT [44], DeepViT [167], SETR [81], PVT [45], CaiT [168], TNT [82], Swin-transformer [46], Query2Label [83], MoCoV3 [84], BEiT [85], SegFormer [86], FuseFormer [169], and MAE [170] have appeared, with excellent results for many kind of…”
BackgroundPSANet: Point-wise Spatial Attention Network for Scene Parsing
Zhao, H., Yi, Z., Liu, S., Shi, J., Loy, C. C., Lin, D., & Jia, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01240-3_17 · Cited by 1,233 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundMethodProceedings of the 24th International Conference on Neural Information Processing Systems
Shawe-Taylor, J. S., Zemel, R. S., Bartlett, P. L., Pereira, F. B., & Weinberger, K. Q. (2011) · Cited by 1,184 · see citing works
SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks
Yang, L., Zhang, R., Li, L., & Xie, X. (2021) · International Conference on Machine Learning · Cited by 1,130 · see citing works
The Dynamic Representation of Scenes
Rensink, R. A. (2000) · Visual Cognition · doi:10.1080/135062800394667 · Cited by 1,079 · see citing works
ParseNet: Looking Wider to See BetterPreprint
Liu, W., Rabinovich, A., & Berg, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04579 · Cited by 1,056 · cited 1× · see citing works
“Due to the localisation of the convolutional operation, CNNs have inherently narrow receptive fields [155, 156], which limits the ability of CNNs to understand scenes globally.”
BackgroundLearning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition
Zheng, H., Fu, J., Mei, T., & Luo, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.557 · Cited by 1,045 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
(2020) · International Conference on Machine Learning · Cited by 1,012 · see citing works
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
Song, S., Lan, C., Xing, J., Zeng, W., & Liu, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11212 · Cited by 1,011 · see citing works
Self-citationExploring Self-Attention for Image Recognition
Zhao, H., Jia, J., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01009 · Cited by 1,002 · cited 4× · see citing works
“PSANet formulates information aggregation as: zi = ∑ j∈Ω(i) F (xi, xj ,∆ij)xj (55) where ∆ij indicates the positional relationship between i and j.”
MethodInfluentialStand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977 · cited 7× · see citing works
“SASA [43] NeurIPS2019 Cls, Det Dot product between query and key -> softmax.”
BackgroundMethodInfluentialBAM: Bottleneck Attention ModulePreprint
Park, J., Woo, S., Lee, J., & Kweon, I. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.06514 · Cited by 951 · cited 4× · see citing works
“[6], [117]) or generate a joint 3-D channel, height, width attention mask directly (e.”
BackgroundInfluentialCoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925 · cited 1× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundStrip Pooling: Rethinking Spatial Pooling for Scene Parsing
Hou, Q., Li, Z., Cheng, M., & Feng, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00406 · Cited by 876 · see citing works
Self-citationAn Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 819 · cited 1× · see citing works
“[141] provided a survey of attention models in deep neural networks which concentrates on their application to natural language processing, while our work focuses on computer vision.”
BackgroundMultiple Object Recognition with Visual Attention
Ba, J., Mnih, V., & Kavukcuoglu, K. (2014) · arXiv · doi:10.48550/arxiv.1412.7755 · Cited by 808 · cited 3× · see citing works
“[88] proposed a deep recurrent network, similar to RAM [31], capable of processing a multi-resolution crop of the input image, called a glimpse, for multiple object recognition task.”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206148 · Cited by 807 · cited 2× · see citing works
“A similar idea can be found in LatentGNN [72], MLP-Mixer [159], and ResMLP [160].”
ResultMulti-context Attention for Human Pose Estimation
Chu, X., Yang, W. P., Ouyang, W., Ma, C., Yuille, A., & Wang, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.601 · Cited by 752 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundAsymmetric Non-Local Neural Networks for Semantic Segmentation
Zhu, Z., Xu, M., Bai, S., Huang, T., & Bai, X. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00068 · Cited by 748 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
MethodExpectation-Maximization Attention Networks for Semantic Segmentation
Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., & Liu, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00926 · Cited by 741 · cited 2× · see citing works
“It was followed by a series of works such as EMANet [40], CCNet [41], HamNet [42] and the Stand-Alone Network [43], which improved speed, quality of results, and generalization capability.”
BackgroundMethodPointASNL: Robust Point Clouds Processing Using Nonlocal Neural Networks With Adaptive Sampling
Yan, X., Zheng, C., Li, Z., Wang, S., & Cui, S. R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00563 · Cited by 709 · see citing works
Relation-Aware Global Attention for Person Re-Identification
Zhang, Z., Lan, C., Zeng, W., Jin, X., & Chen, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00325 · Cited by 701 · cited 11× · see citing works
“Unlike coordinate attention and DANet, which emphasise capturing long-range context, in relation-aware global attention (RGA) [101], Zhang et al. stress the importance of global structural information provided by pairwise relations, and uses it to produce attention maps.”
BackgroundMethodInfluentialRethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 2× · see citing works
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundSet Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., & Teh, Y. W. (2018) · arXiv · doi:10.48550/arxiv.1810.00825 · Cited by 691 · see citing works
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks
Guo, M.-H., Liu, Z.-N., Mu, T., & Hu, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3211006 · Cited by 681 · cited 2× · see citing works
“EANet [75] proposes that self-attention should only consider correlation in a single sample and should ignore potential relationships between different samples.”
BackgroundSelf-citationAn Empirical Study of Spatial Attention Mechanisms in Deep Networks
Zhu, X., Cheng, D., Zhang, Z., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00679 · Cited by 659 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
MethodLocal Relation Networks for Image Recognition
Hu, H., Zhang, Z., Xie, Z., & Lin, S. C.-F. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00356 · Cited by 633 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
End-to-End Comparative Attention Networks for Person Re-Identification
Liu, H., Feng, J., Qi, M., Jiang, J., & Yan, S. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2700762 · Cited by 609 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
ABD-Net: Attentive but Diverse Person Re-Identification
Chen, T., Ding, S., Xie, J., Yuan, Y., Chen, W., Yang, Y., Ren, Z., & Wang, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00844 · Cited by 604 · cited 1× · see citing works
“[130, 101, 123, 86, 96, 17, 84], [108, 40, 61, 166, 148, 152, 135] , [122, 14, 19, 62, 78]”
Action Recognition using Visual AttentionPreprint
Sharma, S., Kiros, R., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04119 · Cited by 593 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Gather-Excite: Exploiting Feature Context in Convolutional Neural NetworksPreprint
Hu, J., Shen, L., Albanie, S., Sun, G., & Vedaldi, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12348 · Cited by 574 · cited 8× · see citing works
“Inspired by SENet, Hu et al. [61] designed GENet to capture long-range spatial contextual information by providing a recalibration function in the spatial domain.”
BackgroundMethodInfluentialCondConv: Conditionally Parameterized Convolutions for Efficient InferencePreprint
Yang, B., Bender, G., Le, Q. V., & Ngiam, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04971 · Cited by 573 · see citing works
Segmentation Transformer: Object-Contextual Representations for Semantic SegmentationPreprint
Yuan, Y., Chen, X., Chen, X., & Wang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11065 · Cited by 548 · cited 2× · see citing works
“OCRNet [73] proposes the concept of object-contextual representation which is a weighted aggregation of all object regions’ representations in the same category, such as a weighted average of all car region representations.”
BackgroundTell Me Where to Look: Guided Attention Inference Network
Li, K., Wu, Z., Peng, K., Ernst, J., & Fu, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00960 · Cited by 546 · see citing works
Graph-Based Global Reasoning Networks
Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., & Kalantidis, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00052 · Cited by 538 · cited 2× · see citing works
“GloRe [71] understands self-attention from a graph learning perspective.”
BackgroundRecalibrating Fully Convolutional Networks With Spatial and Channel “Squeeze and Excitation” Blocks
Roy, A. G., Navab, N., & Wachinger, C. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867261 · Cited by 534 · see citing works
Improving Convolutional Networks With Self-Calibrated Convolutions
Liu, J., Hou, Q., Cheng, M., Wang, C., & Feng, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01011 · Cited by 529 · cited 4× · see citing works
“Motivated by the success of group convolution, Liu et at [86] presented self-calibrated convolution as a means to enlarge the receptive field at each spatial location.”
BackgroundMethodInfluentialSelf-citationGlobal Second-Order Pooling Convolutional Networks
Gao, Z., Xie, J., Wang, Q., & Li, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00314 · Cited by 524 · cited 4× · see citing works
“[54] proposed to improve the squeeze module by using a global second-order pooling (GSoP) block to model high-order statistics while gathering global information.”
BackgroundMethodInfluentialCCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., & Huang, T. S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3007032 · Cited by 524 · see citing works
Pedestrian Alignment Network for Large-scale Person Re-Identification
Zheng, Z., Zheng, L., & Yang, Y. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2873599 · Cited by 521 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
OCNet: Object Context Network for Scene ParsingPreprint
Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., & Wang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00916 · Cited by 520 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundMethodAttention-Aware Compositional Network for Person Re-identification
Xu, J., Zhao, R., Zhu, F., Wang, H., & Ouyang, W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00226 · Cited by 512 · see citing works
Zheng, H., Fu, J., Zha, Z.-J., & Luo, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00515 · Cited by 505 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
VideoLSTM convolves, attends and flows for action recognition
Li, Z., Gavrilyuk, K., Gavves, E., Jain, M., & Snoek, C. G. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.10.011 · Cited by 485 · see citing works
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton DataPreprint
Song, S., Lan, C., Xing, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06067 · Cited by 481 · cited 6× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
BackgroundInfluentialModeling Point Clouds With Self-Attention and Gumbel Subset Sampling
Yang, J., Zhang, Q., Ni, B., Li, L., Liu, J., Zhou, M., & Tian, Q. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00344 · Cited by 465 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
Self-citation(2021) · International Conference on Learning Representations · Cited by 449 · see citing works
Mixed High-Order Attention Network for Person Re-Identification
Chen, B., Deng, W., & Hu, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00046 · Cited by 434 · cited 2× · see citing works
“segmentation [156, 40], face recognition [145, 126], person re-identification [78, 14], action recognition [128, 37], few-show learning [103, 58], medical image processing [99, 47], image generation [46, 159], pose estimation [27], super resolution [30, 164], 3D vision [137, 48], and multi-modal task [116, 141].”
BackgroundDeformable Convolutional NetworksPreprint
Dai, J., Qi, H., Xiong, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06211 · Cited by 428 · cited 6× · see citing works
“[7] proposed deformable convolutional networks (deformable ConvNets) to be invariant to geometric transformations, but they pay attention to the important regions in a different manner.”
BackgroundMethodInfluentialDiversified Visual Attention Networks for Fine-Grained Object Classification
Zhao, B., Wu, X., Feng, J., Peng, Q., & Yan, S. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2648498 · Cited by 424 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification
Wang, C., Zhang, Q., Huang, C., Liu, W., & Wang, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_23 · Cited by 422 · cited 1× · see citing works
“[130, 101, 123, 86, 96, 17, 84], [108, 40, 61, 166, 148, 152, 135] , [122, 14, 19, 62, 78]”
Neural Aggregation Network for Video Face Recognition
Yang, J., Ren, P., Zhang, D., Chen, D., Wen, F., Li, H., & Hua, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.554 · Cited by 411 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundSTAT: Spatial-Temporal Attention Mechanism for Video Captioning
Yan, C. C., Tu, Y., Wang, X., Zhang, Y., Hao, X., Zhang, Y., & Dai, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2924576 · Cited by 402 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
DeepViT: Towards Deeper Vision Transformer
Zhou, D., Kang, B., Jin, X., Yang, L., Lian, X., Jiang, Z., Hou, Q., & Feng, J. (2021) · arXiv · doi:10.48550/arxiv.2103.11886 · Cited by 398 · cited 1× · see citing works
“Following ViT, many transformer-based architectures such as PCT [27], IPT [79], T2T-ViT [44], DeepViT [166], SETR [81], PVT [45], CaiT [167], TNT [82], Swintransformer [46], Query2Label [83], MoCoV3 [84], BEiT [85], SegFormer [86], FuseFormer [168] and MAE [169] have appeared, with excellent results for many kind of v…”
Object-Part Attention Model for Fine-Grained Image Classification
Peng, Y., He, X., & Zhao, J. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2774041 · Cited by 393 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundDiversity Regularized Spatiotemporal Attention for Video-Based Person Re-identification
Li, S., Bąk, S., Carr, P., & Wang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00046 · Cited by 387 · see citing works
TAM: Temporal Adaptive Module for Video Recognition
Liu, Z., Wang, L., Wu, W., Qian, C., & Lü, T. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01345 · Cited by 384 · cited 9× · see citing works
“Combine local attention and global attention TAM [173] Arxiv2020 Action a) local: global spatial average pooling → 1D Convs, b) global: global spatial average pooling → MLP → adaptive convolution (A) (0, 1) S (II), (III)”
BackgroundInfluentialSelf-citationSRM: A Style-Based Recalibration Module for Convolutional Neural Networks
Lee, H., Kim, H., & Nam, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00194 · Cited by 374 · cited 8× · see citing works
“[5], [37], [53], [54], [55], [56], [57], [58] [25], [59], [60] Spatial attention Generate attention mask across spatial domains and use it to select important spatial regions (e.”
BackgroundInfluentialDisentangled Non-local Neural Networks
Yin, M., Yao, Z., Cao, Y., Li, X., Zhang, Z., Lin, S. C.-F., & Hu, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_12 · Cited by 370 · see citing works
Attentional ShapeContextNet for Point Cloud Recognition
Xie, S., Liu, S., Chen, Z., & Tu, Z. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00484 · Cited by 362 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundMethodJointly Attentive Spatial-Temporal Pooling Networks for Video-Based Person Re-identification
Xu, S., Cheng, Y., Gu, K., Yang, Y., Chang, S., & Zhou, P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.507 · Cited by 351 · see citing works
Single Shot Text Detector with Regional Attention
He, P. L., Huang, W., He, T., Zhu, Q., Qiao, Y., & Li, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.331 · Cited by 347 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundEarly and Late Mechanisms of Surround Suppression in Striate Cortex of Macaque
Webb, B. S., Dhruv, N. T., Solomon, S. G., Tailby, C., & Lennie, P. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.3414-05.2005 · Cited by 337 · cited 1× · see citing works
“Motivated by the spatial suppression phenomenon [176], they propose that a neuron which shows suppression effects should be emphasized and define an energy function for each neuron as:”
BackgroundCo-Occurrent Features in Semantic Segmentation
Zhang, H., Zhang, H., Wang, C., & Xie, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00064 · Cited by 312 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
MethodAttentional Pooling for Action RecognitionPreprint
Girdhar, R., & Ramanan, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01467 · Cited by 311 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Gated Channel Transformation for Visual Recognition
Yang, Z., Zhu, L., Wu, Y., & Yang, Y. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr42600.2020.01181 · Cited by 310 · cited 8× · see citing works
“To overcome the above problems, Yang et al. [56] propose the gated channel transformation (GCT) to efficiently collect information while explicitly modeling channel-wise relationships.”
BackgroundMethodInfluentialVOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 309 · cited 2× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundRevisiting Unreasonable Effectiveness of Data in Deep Learning EraPreprint
Sun, C., Shrivastava, A., Singh, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02968 · Cited by 300 · cited 2× · see citing works
“ViT demonstrates that a pure attention-based network can achieve better results than a convolutional neural network especially for large datasets such as JFT-300 [164] and ImageNet-21K [165].”
BackgroundVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276 · see citing works
Global-Local Temporal Representations for Video Person Re-Identification
Li, J., Zhang, S. S., Wang, J., Gao, W., & Tian, Q. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00406 · Cited by 274 · cited 8× · see citing works
“It adopts an adaptive kernel instead of self-attention to capture global contextual information, with lower time complexity than GLTR [172].”
BackgroundMethodInfluentialSelf-citationHierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 249 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
Guan, Q., Huang, Y., Zhong, Z., Zheng, Z., Zheng, L., & Yang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09927 · Cited by 235 · cited 2× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundSTA: Spatial-Temporal Attention for Large-Scale Video-Based Person Re-Identification
Fu, Y., Wang, X., Wei, Y., & Huang, T. S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018287 · Cited by 230 · cited 10× · see citing works
“, [132]), to focus on informative regions.”
BackgroundMethodInfluentialChen, D., Li, H., Xiao, T., Yi, S., & Wang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00128 · Cited by 229 · see citing works
$A^2$-Nets: Double Attention NetworksPreprint
Chen, Y., Kalantidis, Y., Li, J., Yan, S., & Feng, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11579 · Cited by 225 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
MethodRecurrent Spatial-Temporal Attention Network for Action Recognition in Videos
Du, W., Wang, Y., & Qiao, Y. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2778563 · Cited by 218 · cited 7× · see citing works
“[16] introduced spatiotemporal attention to adaptively identify key features in a global way.”
BackgroundInfluentialGCNet: Non-local Networks Meet Squeeze-Excitation Networks and BeyondPreprint
Cao, Y., Xu, J., Lin, S. C.-F., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11492 · Cited by 212 · cited 3× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
BackgroundMLCVNet: Multi-Level Context VoteNet for 3D Object Detection
Xie, Q., Lai, Y., Wu, J., Wang, Z., Zhang, Y., Xu, K., & Wang, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01046 · Cited by 211 · see citing works
Spatial Pyramid Based Graph Reasoning for Semantic Segmentation
Li, X., Yang, Y., Zhao, Q., Shen, T., Lin, Z., & Liu, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00897 · Cited by 201 · see citing works
Masked Autoencoders Are Scalable Vision LearnersPreprint
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06377 · Cited by 200 · cited 4× · see citing works
“Recently, MoCoV3 [84], DINO [180], BEiT [85] and MAE [169] have demonstrated that attention-based models are also well suited to visual tasks.”
BackgroundInfluentialSpatio-temporal Channel Correlation Networks for Action Classification
Diba, A., Fayyaz, M., Sharma, V., Arzani, M. M., Yousefzadeh, R., Gall, J., & Gool, L. V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_18 · Cited by 199 · see citing works
Spatial-Temporal Graph Convolutional Network for Video-Based Person Re-Identification
Yang, J., Zheng, W.-S., Yang, Q., Chen, Y.-C., & Tian, Q. (2020) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr42600.2020.00335 · Cited by 198 · cited 9× · see citing works
“Flattening spatial and temporal dimensions into a sequence, STGCN applies the GCN to capture the spatiotemporal relationships of patches across different frames.”
BackgroundMethodInfluentialQuery2Label: A Simple Transformer Way to Multi-Label ClassificationPreprint
Liu, S., Zhang, L., Yang, X., Su, H., & Zhu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10834 · Cited by 178 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
BackgroundBilinear Attention Networks for Person Retrieval
Fang, P., Zhou, J., Roy, S. K., Petersson, L., & Harandi, M. T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00812 · Cited by 169 · cited 1× · see citing works
“[147] claimed that previous attention models only use first-order information and disregard higher-order statistical information.”
MethodFuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
Liu, R. Z. Z., Deng, H., Huang, Y., Shi, X., Lu, L., Sun, W., Wang, X., Dai, J., & Li, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01378 · Cited by 165 · cited 2× · see citing works
“Following ViT, many transformer-based architectures such as PCT [27], IPT [79], T2T-ViT [44], DeepViT [167], SETR [81], PVT [45], CaiT [168], TNT [82], Swin-transformer [46], Query2Label [83], MoCoV3 [84], BEiT [85], SegFormer [86], FuseFormer [169], and MAE [170] have appeared, with excellent results for many kind of…”
BackgroundYou, H., Feng, Y., Ji, R., & Gao, Y. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240702 · Cited by 161 · see citing works
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial NetworksPreprint
Xu, T., Zhang, P., Huang, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10485 · Cited by 159 · cited 1× · see citing works
“image classification [5], [6], object detection [7], [8], semantic segmentation [9], [10], face recognition [11], [12], person re-identification [13], [14], action recognition [15], [16], few-show learning [17], [18], medical image processing [19], [20], image generation [21], [22], pose estimation [23], super resolut…”
BackgroundSelf-citationCompact Generalized Non-local NetworkPreprint
Yue, K., Sun, M., Yuan, Y., Zhou, F., Ding, E., & Xu, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13125 · Cited by 158 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Zhang, Z., Lan, C., Zeng, W., & Chen, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01042 · Cited by 137 · see citing works
Transformers in computational visual media: A survey
Xu, Y., Wei, H., Lin, M., Deng, Y., Sheng, K., Zhang, M., Tang, F., Dong, W., Huang, F., & Xu, C. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0247-3 · Cited by 137 · cited 2× · see citing works
“Three more specific surveys [141], [142], [143] summarize the development of visual transformers while our paper reviews attention mechanisms in vision more generally, not just selfattention mechanisms.”
BackgroundSecond-Order Non-Local Attention Networks for Person Re-Identification
Bryan, B., Gong, Y., Zhang, Y., & Poellabauer, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00386 · Cited by 129 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Chen, X., Hsieh, C.-J., & Gong, B. (2021) · arXiv · doi:10.48550/arxiv.2106.01548 · Cited by 128 · cited 1× · see citing works
“[184] significantly improved visual transformers by using a novel optimizer, the sharpness-aware minimizer (SAM) [185].”
BackgroundIs Attention Better Than Matrix Decomposition?Preprint
Geng, Z., Guo, M.-H., Chen, H., Li, X., Wei, K., & Lin, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04553 · Cited by 127 · cited 4× · see citing works
“HamNet [42] models capturing global relationships as a low-rank completion problem and designs a series of white-box methods to capture global context using matrix decomposition.”
MethodInfluentialSelf-citationSpSequenceNet: Semantic Segmentation Network on 4D Point Clouds
Shi, H., Lin, G., Wang, H., Hung, T.-Y., & Wang, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00463 · Cited by 106 · cited 1× · see citing works
“[66, 158, 181, 73, 150, 125, 105, 35] [25, 111, 164]”
Interpretable Spatio-Temporal Attention for Video Action Recognition
Meng, L., Zhao, B., Chang, B., Huang, G. M., Sun, W. M., Tung, F., & Sigal, L. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00189 · Cited by 104 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
Multi-Context Attention for Human Pose EstimationPreprint
Chu, X., Yang, W. P., Ouyang, W., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.07432 · Cited by 100 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundAn Empirical Study of Spatial Attention Mechanisms in Deep NetworksPreprint
Zhu, X., Cheng, D., Zhang, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05873 · Cited by 100 · cited 2× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
MethodSurvey on the attention based RNN model and its applications in computer visionPreprint
Wang, F., & Tax, D. M. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06823 · Cited by 99 · cited 1× · see citing works
“[124] present a survey of attention models in computer vision, but it only considers RNN-based attention models, which form just a part of our survey.”
BackgroundYou Look Twice: GaterNet for Dynamic Filter Selection in CNNs
Chen, Z., Li, Y., Samy Bengio, A., & Si, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00939 · Cited by 98 · cited 1× · see citing works
“[5], [37], [53], [54], [55], [56], [57], [58] [25], [59], [60] Spatial attention Generate attention mask across spatial domains and use it to select important spatial regions (e.”
BackgroundSCAN: Self-and-Collaborative Attention Network for Video Person Re-Identification
Zhang, R., Li, J., Sun, H., Ge, Y., Luo, P., Wang, X., & Lin, L. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2911488 · Cited by 93 · see citing works
Hierarchical Pyramid Diverse Attention Networks for Face Recognition
Wang, Q., Wu, T., Zheng, H., & Guo, G. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00835 · Cited by 92 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundBeyond Self-attention: External Attention using Two Linear Layers for Visual TasksPreprint
Guo, M.-H., Liu, Z.-N., Mu, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.02358 · Cited by 83 · cited 2× · see citing works
“EANet [75] proposes that self-attention should only consider correlation in a single sample and should ignore potential relationships between different samples.”
BackgroundSelf-citationLiu, X., Han, Z., Wen, X., Liu, Y.-S., & Zwicker, M. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350960 · Cited by 79 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
Beyond the classical receptive field: The effect of contextual stimuli
Lothar Spillmann, A., Dresp, B., & Tseng, C. (2015) · Journal of Vision · doi:10.1167/15.9.7 · Cited by 75 · cited 1× · see citing works
“Research in the neuroscience community suggests that visual cortical neurons adaptively adjust the sizes of their receptive fields (RFs) according to the input stimulus [174].”
BackgroundPoint TransformerPreprint
Zhao, H., Jiang, L., Jia, J., Torr, P. H. S., & Koltun, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09164 · Cited by 75 · see citing works
VideoLSTM Convolves, Attends and Flows for Action RecognitionPreprint
Li, Z., Gavves, E., Jain, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.01794 · Cited by 63 · see citing works
Learning what and where to attendPreprint
Linsley, D. A., Shiebler, D., Eberhardt, S., & Serre, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08819 · Cited by 62 · see citing works
Single Shot Text Detector with Regional AttentionPreprint
He, P. L., Huang, W., He, T., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00138 · Cited by 58 · cited 1× · see citing works
“, image classification [5, 6], object detection [7, 8], semantic segmentation [9, 10], face recognition [11, 12], person re-identification [13, 14], action recognition [15, 16], few-shot learning [17, 18], medical image processing [19, 20], image generation [21, 22], pose estimation [23], super resolution [24, 25], 3D…”
BackgroundAttentional PointNet for 3D-Object Detection in Point Clouds
Paigwar, A., Erkent, Ö., Wolf, C., & Laugier, C. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00169 · Cited by 55 · cited 1× · see citing works
“[97, 46, 139, 70, 128, 36, 11, 156] , [137, 143, 63, 161, 159, 6, 177, 80] , [67, 180, 10, 23, 24, 163, 155, 151] , [44, 49, 106, 64, 168, 18, 11, 36] , [48, 16, 169, 153, 174, 127, 53, 92] , [89, 21, 5, 134, 65, 171, 2, 109] , [45, 82, 157, 91, 100, 129, 146, 131] , [138, 85, 175, 76, 166, 133, 167, 172] , [41, 1, 17…”
Decoupled Spatial-Temporal Transformer for Video InpaintingPreprint
Liu, R. Z. Z., Deng, H., Huang, Y., Shi, X., Lu, L., Sun, W., Wang, X., Dai, J., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06637 · Cited by 49 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
SRM : A Style-based Recalibration Module for Convolutional Neural NetworksPreprint
Lee, H., Kim, H., & Nam, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10829 · Cited by 42 · cited 8× · see citing works
“[5], [37], [53], [54], [55], [56], [57], [58] [25], [59], [60] Spatial attention Generate attention mask across spatial domains and use it to select important spatial regions (e.”
BackgroundInfluentialFcaNet: Frequency Channel Attention NetworksPreprint
Qin, Z., Zhang, P., Wu, F. Q., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11879 · Cited by 39 · cited 3× · see citing works
“[57] rethought global information captured from the viewpoint of compression and analysed global average pooling in the frequency domain.”
BackgroundCF-SIS: Semantic-Instance Segmentation of 3D Point Clouds by Context Fusion with Self-Attention
Wen, X., Han, Z., Youk, G., & Liu, Y.-S. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413829 · Cited by 33 · cited 1× · see citing works
“[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90], [91], [92], [93]…”
TAM: Temporal Adaptive Module for Video RecognitionPreprint
Liu, Z., Wang, L., Wu, W., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06803 · Cited by 23 · cited 9× · see citing works
“Combine local attention and global attention TAM [173] Arxiv2020 Action a) local: global spatial average pooling → 1D Convs, b) global: global spatial average pooling → MLP → adaptive convolution (A) (0, 1) S (II), (III)”
BackgroundInfluentialSelf-citationGTA: Global Temporal Attention for Video Action Understanding
He, B., Yang, X., Wu, Z., Chen, H., Lim, S.-N., & Shrivastava, A. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.296 · Cited by 23 · see citing works
VOLO: Vision Outlooker for Visual RecognitionPreprint
Yuan, L., Hou, Q., Jiang, Z., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13112 · Cited by 22 · cited 2× · see citing works
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundCan attention enable MLPs to catch up with CNNs?
Guo, M.-H., Liu, Z.-N., Mu, T., Liang, D., Martin, R. R., & Hu, S. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0240-x · Cited by 19 · see citing works
Self-citationREAD: Reciprocal Attention Discriminator for Image-to-Video Re-identification
Shim, M., Ho, H.-I., Kim, J., & Wee, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_20 · Cited by 16 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
GTA: Global Temporal Attention for Video Action UnderstandingPreprint
He, B., Yang, X., Wu, Z., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08510 · Cited by 16 · cited 1× · see citing works
“[130], [132], [133], [134], [135], [136], [137], [138], [139]”
Flagged first, then by citations
10 of the 213 references couldn’t be matched to a known paper.
Research works printed but not linked: 15
- Ferrari, V.; Hebert, M.; Smin- chisescu, C.; Weiss, Y. Eds. Springer Cham, 3-19, 2018.
- Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 213-229, 2020.
- Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 294-310, 2018.
- Zhang, S. Y.; Yan, S. P.; He, X. M. LatentGNN: Learning efficient non-local relations for visual recognition. In: Proceedings of the 36th International Conference on Machine Learning, 7374-7383, 2019.
- Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 191- 207, 2020.
- Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 270-286, 2018.
- Liu, S.; Li, F.; Zhang, H.; Yang, X.; Qi, X.; Su, H.; Zhu, J.; Zhang, L. DAB-DETR: Dynamic anchor boxes are better queries for DETR. arXiv preprint arXiv:2201.12329, 2022. · arXiv
- Guo, M.-H.; Lu, C.-Z.; Liu, Z.-N.; Cheng, M.-M.; Hu, S.-M. Visual attention network. arXiv preprint arXiv:2202.09741, 2022. · arXiv
- You, H. X.; Feng, Y. F.; Ji, R. R.; Gao, Y. PVNet: A joint convolutional network of point cloud and multi- view for 3D shape recognition. In: Proceedings of the 26th ACM International Conference on Multimedia, 1310-1318, 2018.
- Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 384-400, 2018.
- Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 335-350, 2020.
- Han, K.; Wang, Y.; Chen, H.; Chen, X.; Guo, J.; Liu, Z.; Tang, Y.; Xiao, A.; Xu, C.; Xu, Y.; et al. A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020. · arXiv
- Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. · arXiv
- Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 346-361, 2014.
- Foret, P.; Kleiner, A.; Mobahi, H.; Neyshabur, B. Sharpness-aware minimization for efficiently impro- ving generalization. arXiv preprint arXiv:2010.01412, 2020. · arXiv
Cited by
Sorted by citations
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Guo, M.-H., Lu, C.-Z., Liu, Z.-N., Cheng, M., & Hu, S. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0364-2 · Cited by 1,163 · see citing works
“For vision tasks, different channels often represent different objects [43], [65].”
BackgroundInfluentialSelf-citationLarge Selective Kernel Network for Remote Sensing Object Detection
Li, Y., Hou, Q., Zheng, Z., Cheng, M., Yang, J. J., & Li, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01540 · Cited by 956 · see citing works
“The attention mechanism [21] is a simple but effective way to enhance neural representations for various tasks.”
BackgroundSelf-citationBeyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks
Guo, M.-H., Liu, Z.-N., Mu, T., & Hu, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3211006 · Cited by 681 · see citing works
Self-citationConvolutional Neural Networks: A Survey
Krichen, M. (2023) · Computers · doi:10.3390/computers12080151 · Cited by 623 · see citing works
“Attention mechanisms are a powerful tool for improving the performance of CNNs [178–182].”
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 704 · see citing works
Self-citationThe multi-modal fusion in visual question answering: a review of attention mechanisms
Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., & Zheng, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1400 · Cited by 478 · see citing works
HiFuse: Hierarchical multi-scale feature fusion network for medical image classification
Huo, X., Sun, G., Tian, S., Wang, Y., Yu, L., Long, J., Zhang, W., & Li, A. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105534 · Cited by 424 · see citing works
Badgujar, C. M., Poulose, A., & Gan, H. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109090 · Cited by 411 · see citing works
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
Channel prior convolutional attention for medical image segmentation
Huang, H., Chen, Z., Zou, Y. Y., Lu, M., Chen, C., Song, Y., Zhang, H., & Yan, F. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108784 · Cited by 366 · see citing works
Visual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 345 · see citing works
Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review
Cheng, G., Huang, Y., Li, X., Lyu, S., Xu, Z., Zhao, H., Zhao, Q., & Xiang, S. (2024) · Remote Sensing · doi:10.3390/rs16132355 · Cited by 298 · see citing works
YOLOv5-Tassel: Detecting Tassels in RGB UAV Imagery With Improved YOLOv5 Based on Transfer Learning
Liu, W., Quijano, K., & Crawford, M. M. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3206399 · Cited by 271 · see citing works
“Attention mechanism: Inspired by the human visual sys-tem, attention mechanisms have been successfully introduced into computer vision to assist in understanding useful features while de-emphasizing nonrelevant information for complex scenes [45], [46], [47].”
Attention Spiking Neural Networks
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241201 · Cited by 270 · see citing works
“Please refer to [44] for a comprehensive review of attention in CV.”
BackgroundYOLO-MS: Rethinking Multi-Scale Representation Learning for Real-Time Object Detection
Chen, Y., Yuan, X., Wang, J., Wu, R., Li, X., Hou, Q., & Cheng, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3538473 · Cited by 284 · see citing works
“The attention mechanism in Transformer has been proven to be an effective technique for establishing pairwise relationships between a given query Q and a key K [10], [22], [23], [30], [62].”
Self-citationCDLNet: Collaborative Dictionary Learning Network for Remote Sensing Image Scene Classification
Zhao, Y., Liu, J., & Wu, Z. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3336471 · Cited by 244 · see citing works
“Inspired by the human visual sys-tem [16], an attention mechanism [17], [18], [19] has been introduced into computer vision to calculate the importance of different local regions, enabling the model to prioritize critical information in images.”
MethodSelf-citationLarge-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 245 · see citing works
“Many attention models are designed based on this idea, such as the channel attention, spatial attention, temporal attention, branch attention [129].”
BackgroundCollaborative Perception in Autonomous Driving: Methods, Datasets, and Challenges
Han, Y., Zhang, H., Li, H., Jin, Y., Lang, C., & Li, Y. (2023) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2023.3298534 · Cited by 226 · see citing works
BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networks
Cui, H., Dai, W., Zhu, Y., Kan, X., Gu, A. A. C., Lukemire, J., Zhan, L., He, L., Guo, Y., & Yang, C. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3218745 · Cited by 218 · see citing works
“Various fields in deep learning communities such as natural language processing [81] and computer vision [82] have widely benefited from attention mechanisms in terms of model efficiency and accuracy.”
DAE-Former: Dual Attention-Guided Efficient Transformer for Medical Image Segmentation
Azad, R., Arimond, R., Aghdam, E. K., Kazerouni, A., & Merhof, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46005-0_8 · Cited by 212 · see citing works
Attention-based deep meta-transfer learning for few-shot fine-grained fault diagnosis
Li, C., Li, S., Wang, H., Gu, F., & Ball, A. D. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110345 · Cited by 211 · see citing works
EPlus-LLM: A large language model-based computing platform for automated building energy modeling
Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123431 · Cited by 204 · see citing works
LSKNet: A Foundation Lightweight Backbone for Remote Sensing
Li, Y., Li, X., Dai, Y., Hou, Q., Liu, L., Liu, Y., Cheng, M., & Yang, J. Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02247-9 · Cited by 210 · see citing works
“The attention mechanism [67] is a simple but effective way to enhance neural representations for various tasks.”
Self-citationYang, Y. H., Zhang, Y., Cheng, Z., Song, Z., & Tang, C. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107079 · Cited by 192 · see citing works
Bayoudh, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 173 · see citing works
Gao, L., Li, J., Zheng, K., & Jia, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3267890 · Cited by 165 · see citing works
“Similarly, channel attention is to adaptively derive a channelwise attention map to recalibrate the weight of each channel [71], whose idea is highly consistent with SRF.”
Towards An End-to-End Framework for Flow-Guided Video Inpainting
Li, Z., Lu, C.-Z., Qin, J., Guo, C., & Cheng, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01704 · Cited by 154 · see citing works
“Vision Trans-former [15] and its follow-ups [19, 34, 47, 59, 62] achieve an impressive performance on image and video representation learning [9, 13, 35, 43], image generation [41], object detection [2, 65], and many other applications [10, 12, 20, 30].”
BackgroundSelf-citationSpike-based dynamic computing with asynchronous sensing-computing neuromorphic chip
Yao, M., Richter, O., Zhao, G., Qiao, N., Xing, Y., Wang, D., Hu, T., Fang, W., Demirci, T., Marchi, M. D., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-47811-6 · Cited by 152 · see citing works
“Inpractice,functionsf( ⋅ )andg( ⋅ )haveahugedesign space 21,52 (Fig.”
Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding
Yang, Y., Guo, Y., Xiong, J.-Y., Liu, Y., Pan, H., Wang, P., Tong, X., & Guo, B. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450383 · Cited by 180 · see citing works
ISNet: Towards Improving Separability for Remote Sensing Image Change Detection
Cheng, G., Wang, G., & Han, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3174276 · Cited by 129 · see citing works
Detection-Friendly Dehazing: Object Detection in Real-World Hazy Scenes
Li, C., Zhou, H., Liu, Y., Yang, C., Xie, Y., Li, Z., & Zhu, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234976 · Cited by 126 · see citing works
“Attention is treated as a dynamic weight adjustment calculation [47] and attention weights are generated by features in the network.”
BackgroundMulti-Scale Spatiotemporal Feature Fusion Network for Video Saliency Prediction
Zhang, Y., Zhang, T., Wu, C., & Tao, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321394 · Cited by 121 · see citing works
“The attention mechanism [50] is widely used to enhance performance in computer vision tasks.”
BackgroundLong, S., Tan, J., Mao, B., Tang, F., Li, Y., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3526606 · Cited by 127 · see citing works
“(iv) Attention Mechanisms [148], [149], [150]: Although this part belongs to the model layer.”
Artificial intelligence-based diagnosis of Alzheimer's disease with brain MRI images
Yao, Z., Wang, H., Yan, W., Wang, Z. Q., Zhang, W., Wang, Z., & Zhang, G. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110934 · Cited by 131 · see citing works
Dutta, T. K., Nayak, D. R., & Zhang, Y. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105421 · Cited by 123 · see citing works
Large Language Model‐Based Chatbots in Higher Education
Yigci, D., Eryilmaz, M., Yetisen, A. K., Tasoglu, S., & Ozcan, A. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400429 · Cited by 113 · see citing works
IEViT: An enhanced vision transformer architecture for chest X-ray image classification
Okolo, G. I., Katsigiannis, S., & Ramzan, N. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107141 · Cited by 116 · see citing works
“…other architectures [16] have recently gained attention due to their enhanced performance compared to CNNs. Inspired by the success of Transformers in Natural Language Processing (NLP), researchers have recently tried to apply Trans-formers directly to images [21] , using several approaches.”
Masked Diffusion Transformer is a Strong Image Synthesizer
Gao, S., Zhou, P., Cheng, M.-M., & Yan, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02117 · Cited by 91 · see citing works
“Several design improvements, e.g ., adding more attention heads [18], BigGAN [4] residual block, and adaptive group normalization, are proposed in [11] to further enhance the generation ability of the UNet.”
MethodSelf-citationDynamic YOLO for small underwater object detection
Chen, J., & Er, M. J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10788-1 · Cited by 110 · see citing works
“Spatial-aware attention enhances representation capabilities by selectively focusing on crucial regions of the feature map and suppressing unnecessary context, enabling precise feature extraction for object detection (Guo et al. 2022).”
InfluentialPretraining of attention-based deep learning potential model for molecular simulation
Zhang, D., Bi, H., Dai, F., Jiang, W., Liu, X., Zhang, L., & Wang, H. (2024) · npj Computational Materials · doi:10.1038/s41524-024-01278-7 · Cited by 145 · see citing works
“The attention mechanism has achieved great success and played an increasingly important role in CV [57] and NLP [58].”
BackgroundChen, H., He, Y., Zhang, L., Yao, S., Wang, Y., Fang, Y.-M., Liu, Y., & Gao, B. (2023) · International Journal of Digital Earth · doi:10.1080/17538947.2023.2177359 · Cited by 108 · see citing works
“The AM is generally embodied in the form of weights, allocating limited information processing resources to the important parts, thus achieving the e ff ect of improving the performance of deep learning models (Guo et al. 2022; Yang 2020).”
BackgroundDiagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review
Shoeibi, A., Khodatars, M., Jafari, M., Ghassemi, N., Moridian, P., Alizadehsani, R., Ling, S. H., Khosravi, A., Alinejad‐Rokny, H., Lam, H., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.12.010 · Cited by 106 · see citing works
MaizeNet: A Deep Learning Approach for Effective Recognition of Maize Plant Leaf Diseases
Masood, M., Nawaz, M., Nazir, T., Javed, A., Alkanhel, R. I., Elmannai, H., Dhahbi, S., & Bourouis, S. (2023) · IEEE Access · doi:10.1109/access.2023.3280260 · Cited by 117 · see citing works
“tion learning capabilities from the input during training [73].”
BackgroundDeep learning-based semantic segmentation of remote sensing images: a review
Lv, J., Shen, Q., Lv, M., Li, Y., Shi, L., & Zhang, P. (2023) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2023.1201125 · Cited by 104 · see citing works
Jiang, P., Li, X., Shen, H., Chen, Y., Wang, L., Chen, H., Feng, J., & Liu, J. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10588-z · Cited by 111 · see citing works
Self-citationESKNet: An enhanced adaptive selection kernel convolution for ultrasound breast tumors segmentation
Chen, G., Zhou, L., Zhang, J., Yin, X., Cui, L., & Dai, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123265 · Cited by 105 · see citing works
Shen, Y., Tang, B., Li, B., Tan, Q., & Wu, Y. (2022) · Measurement · doi:10.1016/j.measurement.2022.111803 · Cited by 96 · see citing works
Sharma, P., Nayak, D. R., Balabantaray, B. K., Tanveer, M., & Nayak, R. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.006 · Cited by 99 · see citing works
A Review of Digital Twin Applications in Civil and Infrastructure Emergency Management
Cheng, R., Hou, L., & Xu, S. (2023) · Buildings · doi:10.3390/buildings13051143 · Cited by 94 · see citing works
Engineering Semantic Communication: A Survey
Wheeler, D., & Natarajan, B. (2023) · IEEE Access · doi:10.1109/access.2023.3243065 · Cited by 104 · see citing works
“The availability of massive amounts of data and advanced algorithms have enabled the practical implementation of ML in many domains, including NLP [65], computer vision [66], and others.”
BackgroundImproved Mask R-CNN Multi-Target Detection and Segmentation for Autonomous Driving in Complex Scenes
Fang, S., Zhang, B., & Hu, J. (2023) · Sensors · doi:10.3390/s23083853 · Cited by 91 · see citing works
“The attention mechanism is a signal processing mechanism that mimics the human brain, and the strategy is well adapted and gainful for computer vision tasks [31].”
BackgroundXue, Y., Wen, C., Wang, Z., Liu, W., & Chen, G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111205 · Cited by 91 · see citing works
Automated building layout generation using deep learning and graph algorithms
Wang, L., Liu, J., Yan, Z., Cheng, G., Hu, H., Hu, J., & Huang, X. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105036 · Cited by 90 · see citing works
Self-citationSkeletal Fracture Detection with Deep Learning: A Comprehensive Review
Su, Z., Adam, A., Nasrudin, M. F., Ayob, M. B., & Punganan, G. (2023) · Diagnostics · doi:10.3390/diagnostics13203245 · Cited by 89 · see citing works
“The is because atrous convolution can help extract a rich feature map and attention mechanisms can help focus on important features [70].”
BackgroundYOLOv8-C2f-Faster-EMA: An Improved Underwater Trash Detection Model Based on YOLOv8
Zhu, J. S., Hu, T., Zheng, L., Zhou, N., Ge, H., & Hong, Z. (2024) · Sensors · doi:10.3390/s24082483 · Cited by 98 · see citing works
Evolution of Surgical Robot Systems Enhanced by Artificial Intelligence: A Review
Liu, Y., Wu, X., Sang, Y., Zhao, C., Wang, Y., Shi, B., & Fan, Y. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202300268 · Cited by 105 · see citing works
An Improved Apple Object Detection Method Based on Lightweight YOLOv4 in Complex Backgrounds
Zhang, C., Kang, F., & Wang, Y. (2022) · Remote Sensing · doi:10.3390/rs14174150 · Cited by 85 · see citing works
“Attention mechanisms are often used to focus a model on more important information, thus improving their efficiency and accuracy It has been widely used in CNNs to improve their performance [55].”
BackgroundMultistep ahead prediction of temperature and humidity in solar greenhouse based on FAM-LSTM model
Yang, Y., Gao, P., Sun, Z., Wang, H., Lu, M., Liu, Y., & Hu, J. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108261 · Cited by 82 · see citing works
Emerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82 · see citing works
“Attention methods can be classified into five categories [55]: channel attention, spatial attention, temporal attention, branch channel, and hybrid attention.”
MethodZhang, Y., Ding, K., Hui, J.-Z., Lv, J.-X., Zhou, X.-L., & Zheng, P. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101792 · Cited by 80 · see citing works
Li, X., Xu, F., Liu, F., Tong, Y., Lyu, X., & Zhou, J. W. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3339291 · Cited by 82 · see citing works
“The transformer was designed based on the self-AM, which can attend to complete sequences thereby learning long-range dependencies [30], [31], [32].”
MethodWang, H., Xu, Y., He, Y., Cai, Y., Chen, L., Li, Y., Sotelo, M. Á., & Li, Z. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3196954 · Cited by 78 · see citing works
Multi-Task Learning With Multi-Query Transformer for Dense Prediction
Xu, Y., Li, X., Yuan, H., Yang, Y., & Zhang, L. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3292995 · Cited by 79 · see citing works
“And project them into the feature space, where the converter encoder computes the queries, keys and values, to generate the final result [12].”
BackgroundLightweight transformer-driven multi-scale trapezoidal attention network for saliency detection
Usman, M. T., Khan, H., Rida, I., & Koo, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110917 · Cited by 78 · see citing works
Yao, Z., Wang, Z., Huang, J., Xu, N., Cui, X., & Wu, T. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.175407 · Cited by 77 · see citing works
MAE-YOLOv8-based small object detection of green crisp plum in real complex orchard environments
Liu, Q., Lv, J., & Zhang, C. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109458 · Cited by 77 · see citing works
Cui, L., Jing, X., Wang, Y., Huan, Y., Xu, Y., & Zhang, Q. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3225150 · Cited by 76 · see citing works
“In recent years, several transformer-based vision models have been proposed for different computer vision tasks, such as target classification [28], object detection [29], and semantic segmentation [30], [31], [32].”
MethodpFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning
Guo, T., Guo, S., & Wang, J.-X. (2023) · The Web Conference · doi:10.1145/3543507.3583518 · Cited by 120 · see citing works
“To analyze the efect of the number of shots, we vary the shots in [1, 2, 4, 8, 16].”
MethodLiu, J., Hao, J., Lin, H., Pan, W., Yang, J., Feng, Y., Wang, G., Li, J., Jin, Z., Zhao, Z., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100825 · Cited by 75 · see citing works
Self-citationHu, W., Li, X., Li, C., Li, R., Jiang, T., Sun, H., Huang, X., Grzegorzek, M., & Li, X. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107034 · Cited by 82 · see citing works
Advances in attention mechanisms for medical image segmentation
Zhang, J., Chen, X., Yang, B., Guan, Q., Chen, Q., Chen, J., Wu, Q., Xie, Y., & Xia, Y. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2024.100721 · Cited by 143 · see citing works
Yang, Y., Miao, Z., Hua, Z., Wang, B., & Wu, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3351277 · Cited by 73 · see citing works
“Attention mechanism is a typical method to improve the performance of deep learning algorithms [49], [50].”
MethodData Augmentation Method for Plant Leaf Disease Recognition
Min, B., Kim, T., Shin, D., & Shin, D. (2023) · Applied Sciences · doi:10.3390/app13031465 · Cited by 72 · see citing works
Xiong, X., He, M., Li, T., Zheng, G., Xu, W., Fan, X., & Zhang, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3367415 · Cited by 75 · see citing works
“The intention of the attention mechanism is to prompt the original network to prioritize crucial information during the learning process [36].”
BackgroundShen, J., Liu, N., Sun, H., Wu, S., Liang, Z., Han, L., Zhang, Y., & Li, D. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3642410 · Cited by 72 · see citing works
“The attention mechanism in neural networks is a resource allocation scheme that allocates computational resources to more important tasks in situations where computational capacity is limited, while addressing the issue of information overload [46].”
Chen, Y., Kang, Y., Hao, H., Xie, X., Zeng, J., Xu, T., Li, C., Tan, Y., & Fang, L. (2022) · Advanced Functional Materials · doi:10.1002/adfm.202209781 · Cited by 70 · see citing works
Self-citationA review of distributed solar forecasting with remote sensing and deep learning
Chu, Y., Wang, Y., Yang, D., Chen, S., & Li, M. (2024) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2024.114391 · Cited by 71 · see citing works
Giri, N., Roy, R. S., & Cheng, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102536 · Cited by 59 · see citing works
Machine Learning for Cataract Classification/Grading on Ophthalmic Imaging Modalities: A Survey
Zhang, X.-Q., Hu, Y., Xiao, Z., Fang, J., Higashita, R., & Liu, J. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1329-0 · Cited by 69 · see citing works
“In computer vision, attention can be classified into five categories: channel, spatial, temporal attention, branch attention, and attention combinations such as channel & spatial attention [54].”
BackgroundSelf-citationZhang, Z., Bao, L., Xiang, S., Xie, G., & Gao, R. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3409072 · Cited by 69 · see citing works
“Attention mechanisms [39] have gained significant attention in various visual tasks, including image fusion [40], [41], detection [42], [43], and semantic segmentation [44], highlighting their importance in the field.”
BackgroundAn overview: Attention mechanisms in multi-agent reinforcement learning
Hu, K., Xu, K., Xia, Q., Li, M., Song, Z., Song, L., & Sun, N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128015 · Cited by 69 · see citing works
Zhong, T., Qin, C., Shi, G., Zhang, Z., Tao, J., & Liu, C. (2024) · Science China Technological Sciences · doi:10.1007/s11431-024-2734-x · Cited by 69 · see citing works
Fu, R., Xiao, D., Navon, I. M., Fang, F., Yang, L., Wang, C., & Cheng, S. L. (2023) · International Journal for Numerical Methods in Engineering · doi:10.1002/nme.7240 · Cited by 67 · see citing works
A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?
Gonçalves, T., Rio-Torto, I., Teixeira, L. F., & Cardoso, J. S. (2022) · IEEE Access · doi:10.1109/access.2022.3206449 · Cited by 81 · see citing works
“A similar approach was employed in Guo et al. [2021] for retinal vessel segmentation. This methodology was extended in Jin et al. [2020], which employs a 3D U-Net-based architecture with residual blocks and an attention module to extract volumes of interest in liver and brain CT imaging data and perform tumor segmenta…”
MethodInfluentialMineral Prospectivity Mapping Using Deep Self-Attention Model
Yin, B., Zuo, R., & Sun, S. (2022) · Natural Resources Research · doi:10.1007/s11053-022-10142-8 · Cited by 66 · see citing works
“…indispensable component for constructing deep learning models in various artificial intelligence fields, such as computer vision (Wang & Tax, 2016; Guo et al., 2022), natural language processing (Hu, 2019; Galassi et al., 2020), and speech recognition (Chorowski et al., 2015; Bahdanau et al,…”
BackgroundMethodAldakhil, L. A., Alhasson, H. F., & Alharbi, S. S. (2024) · Diagnostics · doi:10.3390/diagnostics14131402 · Cited by 65 · see citing works
“These mechanisms can be characterized as dynamic selection processes that adaptively adjust weights based on the significance of different features [23].”
BackgroundROULETTE: A neural attention multi-output model for explainable Network Intrusion Detection
Andresini, G., Appice, A., Caforio, F. P., Malerba, D., & Vessio, G. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117144 · Cited by 64 · see citing works
Cross-lingual font style transfer with full-domain convolutional attention
Zhao, H., Ji, T.-L., Rosin, P. L., Lai, Y., Meng, W., & Wang, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110709 · Cited by 65 · see citing works
“It is used for many vision tasks, such as image classification, object detection, semantic segmentation, etc. Attention can also be divided into four basic categories: channel attention, spatial attention, temporal attention, and branching attention [17].”
Scale-Context Perceptive Network for Crowd Counting and Localization in Smart City System
Zhai, W., Gao, M., Guo, X., Li, Q., & Jeon, G. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3268226 · Cited by 63 · see citing works
“The attention mechanism [18], [19], [20], [21] has been widely implemented in object counting by adjusting parameters to emphasize the foreground and weaken the background.”
BackgroundMultimodal Agent AI: A Survey of Recent Advances and Future Directions
Sun, Y., Sun, H., Ma, J., Zhang, P., & Huang, X. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-4802-8 · Cited by 69 · see citing works
Aung, P. S., Nguyen, L. X., Tun, Y. K., Han, Z., & Hong, C. S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3329893 · Cited by 62 · see citing works
“Attention mechanisms have been widely used in natural language processing, computer vision, and other domains to enhance model performance [37], [38].”
BackgroundMD-YOLO: Surface Defect Detector for Industrial Complex Environments
Zheng, H., Chen, X., Cheng, H., Du, Y., & Jiang, Z. (2024) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2024.108170 · Cited by 68 · see citing works
Zhou, F., Chen, Y., & Liu, J. (2023) · Remote Sensing · doi:10.3390/rs15051395 · Cited by 67 · see citing works
Self-citationSun, C., Zhang, R., Lu, Y., Cui, Y., Deng, Z., Cao, D., & Khajepour, A. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3321309 · Cited by 61 · see citing works
“Additionally, graph-based methods [136], [140] and attention-based fusion [141], [142], [143] offer powerful tools for merging features and automatically identifying their relationships.”
MethodRobust Optical and SAR Image Matching Using Attention-Enhanced Structural Features
Ye, Y., Yang, C., Gong, G., Yang, P., Quan, D., & Li, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3366247 · Cited by 62 · see citing works
Deep learning approaches for seizure video analysis: A review
Ahmedt‐Aristizabal, D., Armin, M. A., Hayder, Z., Garcia‐Cairasco, N., Petersson, L., Fookes, C., Denman, S., & McGonigal, A. (2024) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2024.109735 · Cited by 63 · see citing works
Deep Learning Frontiers in 3D Object Detection: A Comprehensive Review for Autonomous Driving
Pravallika, A., Hashmi, M. F., & Gupta, A. D. (2024) · IEEE Access · doi:10.1109/access.2024.3456893 · Cited by 65 · see citing works
“There are five sorts of attention methods: channel attention, spatial attention, temporal attention, branch attention, and hybrid attention [214].”
Yu, X., & Li, D. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3351755 · Cited by 60 · see citing works
“The attention mechanism [19] has evolved into an increasingly significant component of computer vision over the last decade.”
BackgroundHuang, Z., Qin, Y., Lin, X., Liu, T., Feng, Z., & Liu, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3217763 · Cited by 59 · see citing works
“However, note that deep learning attention mechanisms generally involve convolution layers, fully connected layers, parameterized activation functions, etc [37], [38].”
BackgroundMethod
- Authors
- Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M.
- Affiliations
- Tsinghua University (CN)
- TKLNDST, College of Computer Science, Nankai University, Tianjin 300350, China
- Nankai University (CN)
- Cardiff University (GB)
- Corresponding author
- Hu, Shi-Min
- ORCID
- Venue
- Computational Visual Media
- Publisher
- Tsinghua University Press
- Published
- 2022-09
- Volume
- 8
- Issue
- 3
- Pages
- 331-368
- ISSN
- 2096-0433 (ISSN-L)
- 2096-0662
- OpenAlex
- W3212386989
- Cited by
- 2,566
- References
- 213
- Citations by year
- 2026: 404 · 2025: 687 · 2024: 739 · 2023: 576 · 2022: 159 · 2021: 1
- FWCI
- 176.50
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Computational Visual Media (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62132012, 61521002)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- attention mechanism, visual attention, channel attention, spatial attention, temporal attention, image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multimodal tasks, self-supervised learning, human visual system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
