The DOI matches its Crossref record; 203 of the 213 references it cites were found in a registry or an index.
Attention mechanisms in computer vision: A survey
Open accessAbstract
Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multimodal tasks, and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention, and branch attention; a related repository https://github.com/MenghaoGuo/Awesome-Vision-Attentions is dedicated to collecting related work. We also suggest future directions for attention mechanism research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The evolution of LMs has led to the proliferation of various attention mechanisms (Guo et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Guo et al. 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 6×
“ResNeSt [115] also adopts this attention mechanism to improve the CNN backbone in a more general way, giving excellent results on ResNet [146] and ResNeXt [176].”
BackgroundMethodResultInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9×
“Given an input feature map X and the previous time step LSTM hidden state ht−1 ∈ Rd, a spatial attention mechanism pays more attention to the semantically useful regions, guided by LSTM hidden state ht−1.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“SGD [104] and Adam [72] are well-suited for optimizing convolutional neural networks.”
BackgroundImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“ViT demonstrates that a pure attention-based network can achieve better results than a convolutional neural network especially for large datasets such as JFT-300 [164] and ImageNet-21K [165].”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Transformers have had great success in natural language processing [33], [38], [149], [150], [152], [161].”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 4×
“Recently, various pure deep self-attention networks (visual transformers) [27], [34], [44], [45], [46], [47], [48], [49] have appeared, showing the huge potential of attention-based models.”
BackgroundInfluentialSelf-citationCBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Large-scale attention-based pre-trained models have had great success in natural language processing [85], [179].”
Background
Research works printed but not linked: 6
- Zhang, S. Y.; Yan, S. P.; He, X. M. LatentGNN: Learning efficient non-local relations for visual recognition. In: Proceedings of the 36th International Conference on Machine Learning, 7374–7383, 2019.
- Liu, S.; Li, F.; Zhang, H.; Yang, X.; Qi, X.; Su, H.; Zhu, J.; Zhang, L. DAB-DETR: Dynamic anchor boxes are better queries for DETR. arXiv preprint arXiv:2201.12329, 2022. · arXiv
- Guo, M.-H.; Lu, C.-Z.; Liu, Z.-N.; Cheng, M.-M.; Hu, S.-M. Visual attention network. arXiv preprint arXiv:2202.09741, 2022. · arXiv
- Han, K.; Wang, Y.; Chen, H.; Chen, X.; Guo, J.; Liu, Z.; Tang, Y.; Xiao, A.; Xu, C.; Xu, Y.; et al. A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020. · arXiv
- Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. · arXiv
- Foret, P.; Kleiner, A.; Mobahi, H.; Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020. · arXiv
Cited by
Sorted by citations
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,506
Guo, M.-H., Lu, C.-Z., Liu, Z.-N., Cheng, M., & Hu, S. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0364-2 · Cited by 1,235
“For vision tasks, different channels often represent different objects [43], [65].”
BackgroundInfluentialSelf-citationLarge Selective Kernel Network for Remote Sensing Object Detection
Li, Y., Hou, Q., Zheng, Z., Cheng, M., Yang, J. J., & Li, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01540 · Cited by 1,022
“The attention mechanism [21] is a simple but effective way to enhance neural representations for various tasks.”
BackgroundSelf-citationBeyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks
Guo, M.-H., Liu, Z.-N., Mu, T., & Hu, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3211006 · Cited by 703
Self-citationConvolutional Neural Networks: A Survey
Krichen, M. (2023) · Computers · doi:10.3390/computers12080151 · Cited by 623
“Attention mechanisms are a powerful tool for improving the performance of CNNs [178–182].”
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 964
Self-citationThe multi-modal fusion in visual question answering: a review of attention mechanisms
Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., & Zheng, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1400 · Cited by 492
HiFuse: Hierarchical multi-scale feature fusion network for medical image classification
Huo, X., Sun, G., Tian, S., Wang, Y., Yu, L., Long, J., Zhang, W., & Li, A. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105534 · Cited by 469
Badgujar, C. M., Poulose, A., & Gan, H. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109090 · Cited by 411
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Channel prior convolutional attention for medical image segmentation
Huang, H., Chen, Z., Zou, Y. Y., Lu, M., Chen, C., Song, Y., Zhang, H., & Yan, F. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108784 · Cited by 376
Visual attention methods in deep learning: An in-depth survey
Hassanin, M., Anwar, S., Radwan, I., Khan, F. S., & Mian, A. S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102417 · Cited by 364
Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Review
Cheng, G., Huang, Y., Li, X., Lyu, S., Xu, Z., Zhao, H., Zhao, Q., & Xiang, S. (2024) · Remote Sensing · doi:10.3390/rs16132355 · Cited by 334
YOLOv5-Tassel: Detecting Tassels in RGB UAV Imagery With Improved YOLOv5 Based on Transfer Learning
Liu, W., Quijano, K., & Crawford, M. M. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3206399 · Cited by 275
“Attention mechanism: Inspired by the human visual sys-tem, attention mechanisms have been successfully introduced into computer vision to assist in understanding useful features while de-emphasizing nonrelevant information for complex scenes [45], [46], [47].”
Attention Spiking Neural Networks
Yao, M., Zhao, G., Zhang, H., Hu, Y., Deng, L., Tian, Y., Xu, B., & Li, G. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3241201 · Cited by 298
“Please refer to [44] for a comprehensive review of attention in CV.”
BackgroundYOLO-MS: Rethinking Multi-Scale Representation Learning for Real-Time Object Detection
Chen, Y., Yuan, X., Wang, J., Wu, R., Li, X., Hou, Q., & Cheng, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3538473 · Cited by 311
“The attention mechanism in Transformer has been proven to be an effective technique for establishing pairwise relationships between a given query Q and a key K [10], [22], [23], [30], [62].”
Self-citationCDLNet: Collaborative Dictionary Learning Network for Remote Sensing Image Scene Classification
Zhao, Y., Liu, J., & Wu, Z. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3336471 · Cited by 244
“Inspired by the human visual sys-tem [16], an attention mechanism [17], [18], [19] has been introduced into computer vision to calculate the importance of different local regions, enabling the model to prioritize critical information in images.”
MethodSelf-citationLarge-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 268
“Many attention models are designed based on this idea, such as the channel attention, spatial attention, temporal attention, branch attention [129].”
BackgroundCollaborative Perception in Autonomous Driving: Methods, Datasets, and Challenges
Han, Y., Zhang, H., Li, H., Jin, Y., Lang, C., & Li, Y. (2023) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2023.3298534 · Cited by 254
BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networks
Cui, H., Dai, W., Zhu, Y., Kan, X., Gu, A. A. C., Lukemire, J., Zhan, L., He, L., Guo, Y., & Yang, C. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3218745 · Cited by 218
“Various fields in deep learning communities such as natural language processing [81] and computer vision [82] have widely benefited from attention mechanisms in terms of model efficiency and accuracy.”
DAE-Former: Dual Attention-Guided Efficient Transformer for Medical Image Segmentation
Azad, R., Arimond, R., Aghdam, E. K., Kazerouni, A., & Merhof, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46005-0_8 · Cited by 212
Attention-based deep meta-transfer learning for few-shot fine-grained fault diagnosis
Li, C., Li, S., Wang, H., Gu, F., & Ball, A. D. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110345 · Cited by 211
EPlus-LLM: A large language model-based computing platform for automated building energy modeling
Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123431 · Cited by 204
LSKNet: A Foundation Lightweight Backbone for Remote Sensing
Li, Y., Li, X., Dai, Y., Hou, Q., Liu, L., Liu, Y., Cheng, M., & Yang, J. Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02247-9 · Cited by 237
“The attention mechanism [67] is a simple but effective way to enhance neural representations for various tasks.”
Self-citationYang, Y. H., Zhang, Y., Cheng, Z., Song, Z., & Tang, C. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107079 · Cited by 197
Bayoudh, K. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102217 · Cited by 180
Gao, L., Li, J., Zheng, K., & Jia, X. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3267890 · Cited by 165
“Similarly, channel attention is to adaptively derive a channelwise attention map to recalibrate the weight of each channel [71], whose idea is highly consistent with SRF.”
Towards An End-to-End Framework for Flow-Guided Video Inpainting
Li, Z., Lu, C.-Z., Qin, J., Guo, C., & Cheng, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01704 · Cited by 154
“Vision Trans-former [15] and its follow-ups [19, 34, 47, 59, 62] achieve an impressive performance on image and video representation learning [9, 13, 35, 43], image generation [41], object detection [2, 65], and many other applications [10, 12, 20, 30].”
BackgroundSelf-citationSpike-based dynamic computing with asynchronous sensing-computing neuromorphic chip
Yao, M., Richter, O., Zhao, G., Qiao, N., Xing, Y., Wang, D., Hu, T., Fang, W., Demirci, T., Marchi, M. D., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-47811-6 · Cited by 179
“Inpractice,functionsf( ⋅ )andg( ⋅ )haveahugedesign space 21,52 (Fig.”
Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding
Yang, Y., Guo, Y., Xiong, J.-Y., Liu, Y., Pan, H., Wang, P., Tong, X., & Guo, B. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450383 · Cited by 233
ISNet: Towards Improving Separability for Remote Sensing Image Change Detection
Cheng, G., Wang, G., & Han, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3174276 · Cited by 130
Detection-Friendly Dehazing: Object Detection in Real-World Hazy Scenes
Li, C., Zhou, H., Liu, Y., Yang, C., Xie, Y., Li, Z., & Zhu, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234976 · Cited by 132
“Attention is treated as a dynamic weight adjustment calculation [47] and attention weights are generated by features in the network.”
BackgroundMulti-Scale Spatiotemporal Feature Fusion Network for Video Saliency Prediction
Zhang, Y., Zhang, T., Wu, C., & Tao, R. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321394 · Cited by 121
“The attention mechanism [50] is widely used to enhance performance in computer vision tasks.”
BackgroundLong, S., Tan, J., Mao, B., Tang, F., Li, Y., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3526606 · Cited by 139
“(iv) Attention Mechanisms [148], [149], [150]: Although this part belongs to the model layer.”
Artificial intelligence-based diagnosis of Alzheimer's disease with brain MRI images
Yao, Z., Wang, H., Yan, W., Wang, Z. Q., Zhang, W., Wang, Z., & Zhang, G. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110934 · Cited by 135
Dutta, T. K., Nayak, D. R., & Zhang, Y. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105421 · Cited by 123
Large Language Model‐Based Chatbots in Higher Education
Yigci, D., Eryilmaz, M., Yetisen, A. K., Tasoglu, S., & Ozcan, A. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400429 · Cited by 113
IEViT: An enhanced vision transformer architecture for chest X-ray image classification
Okolo, G. I., Katsigiannis, S., & Ramzan, N. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107141 · Cited by 121
“…other architectures [16] have recently gained attention due to their enhanced performance compared to CNNs. Inspired by the success of Transformers in Natural Language Processing (NLP), researchers have recently tried to apply Trans-formers directly to images [21] , using several approaches.”
Masked Diffusion Transformer is a Strong Image Synthesizer
Gao, S., Zhou, P., Cheng, M.-M., & Yan, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02117 · Cited by 91
“Several design improvements, e.g ., adding more attention heads [18], BigGAN [4] residual block, and adaptive group normalization, are proposed in [11] to further enhance the generation ability of the UNet.”
MethodSelf-citationDynamic YOLO for small underwater object detection
Chen, J., & Er, M. J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10788-1 · Cited by 110
“Spatial-aware attention enhances representation capabilities by selectively focusing on crucial regions of the feature map and suppressing unnecessary context, enabling precise feature extraction for object detection (Guo et al. 2022).”
InfluentialPretraining of attention-based deep learning potential model for molecular simulation
Zhang, D., Bi, H., Dai, F., Jiang, W., Liu, X., Zhang, L., & Wang, H. (2024) · npj Computational Materials · doi:10.1038/s41524-024-01278-7 · Cited by 155
“The attention mechanism has achieved great success and played an increasingly important role in CV [57] and NLP [58].”
BackgroundChen, H., He, Y., Zhang, L., Yao, S., Wang, Y., Fang, Y.-M., Liu, Y., & Gao, B. (2023) · International Journal of Digital Earth · doi:10.1080/17538947.2023.2177359 · Cited by 108
“The AM is generally embodied in the form of weights, allocating limited information processing resources to the important parts, thus achieving the e ff ect of improving the performance of deep learning models (Guo et al. 2022; Yang 2020).”
BackgroundDiagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review
Shoeibi, A., Khodatars, M., Jafari, M., Ghassemi, N., Moridian, P., Alizadehsani, R., Ling, S. H., Khosravi, A., Alinejad‐Rokny, H., Lam, H., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.12.010 · Cited by 110
MaizeNet: A Deep Learning Approach for Effective Recognition of Maize Plant Leaf Diseases
Masood, M., Nawaz, M., Nazir, T., Javed, A., Alkanhel, R. I., Elmannai, H., Dhahbi, S., & Bourouis, S. (2023) · IEEE Access · doi:10.1109/access.2023.3280260 · Cited by 124
“tion learning capabilities from the input during training [73].”
BackgroundDeep learning-based semantic segmentation of remote sensing images: a review
Lv, J., Shen, Q., Lv, M., Li, Y., Shi, L., & Zhang, P. (2023) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2023.1201125 · Cited by 104
Jiang, P., Li, X., Shen, H., Chen, Y., Wang, L., Chen, H., Feng, J., & Liu, J. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10588-z · Cited by 120
Self-citationESKNet: An enhanced adaptive selection kernel convolution for ultrasound breast tumors segmentation
Chen, G., Zhou, L., Zhang, J., Yin, X., Cui, L., & Dai, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123265 · Cited by 105
Shen, Y., Tang, B., Li, B., Tan, Q., & Wu, Y. (2022) · Measurement · doi:10.1016/j.measurement.2022.111803 · Cited by 98
Sharma, P., Nayak, D. R., Balabantaray, B. K., Tanveer, M., & Nayak, R. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.006 · Cited by 104
A Review of Digital Twin Applications in Civil and Infrastructure Emergency Management
Cheng, R., Hou, L., & Xu, S. (2023) · Buildings · doi:10.3390/buildings13051143 · Cited by 94
Engineering Semantic Communication: A Survey
Wheeler, D., & Natarajan, B. (2023) · IEEE Access · doi:10.1109/access.2023.3243065 · Cited by 115
“The availability of massive amounts of data and advanced algorithms have enabled the practical implementation of ML in many domains, including NLP [65], computer vision [66], and others.”
BackgroundImproved Mask R-CNN Multi-Target Detection and Segmentation for Autonomous Driving in Complex Scenes
Fang, S., Zhang, B., & Hu, J. (2023) · Sensors · doi:10.3390/s23083853 · Cited by 91
“The attention mechanism is a signal processing mechanism that mimics the human brain, and the strategy is well adapted and gainful for computer vision tasks [31].”
BackgroundXue, Y., Wen, C., Wang, Z., Liu, W., & Chen, G. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111205 · Cited by 91
Automated building layout generation using deep learning and graph algorithms
Wang, L., Liu, J., Yan, Z., Cheng, G., Hu, H., Hu, J., & Huang, X. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105036 · Cited by 90
Self-citationSkeletal Fracture Detection with Deep Learning: A Comprehensive Review
Su, Z., Adam, A., Nasrudin, M. F., Ayob, M. B., & Punganan, G. (2023) · Diagnostics · doi:10.3390/diagnostics13203245 · Cited by 103
“The is because atrous convolution can help extract a rich feature map and attention mechanisms can help focus on important features [70].”
BackgroundYOLOv8-C2f-Faster-EMA: An Improved Underwater Trash Detection Model Based on YOLOv8
Zhu, J. S., Hu, T., Zheng, L., Zhou, N., Ge, H., & Hong, Z. (2024) · Sensors · doi:10.3390/s24082483 · Cited by 103
Evolution of Surgical Robot Systems Enhanced by Artificial Intelligence: A Review
Liu, Y., Wu, X., Sang, Y., Zhao, C., Wang, Y., Shi, B., & Fan, Y. (2024) · Advanced Intelligent Systems · doi:10.1002/aisy.202300268 · Cited by 112
An Improved Apple Object Detection Method Based on Lightweight YOLOv4 in Complex Backgrounds
Zhang, C., Kang, F., & Wang, Y. (2022) · Remote Sensing · doi:10.3390/rs14174150 · Cited by 91
“Attention mechanisms are often used to focus a model on more important information, thus improving their efficiency and accuracy It has been widely used in CNNs to improve their performance [55].”
BackgroundMultistep ahead prediction of temperature and humidity in solar greenhouse based on FAM-LSTM model
Yang, Y., Gao, P., Sun, Z., Wang, H., Lu, M., Liu, Y., & Hu, J. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108261 · Cited by 82
Emerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82
“Attention methods can be classified into five categories [55]: channel attention, spatial attention, temporal attention, branch channel, and hybrid attention.”
MethodZhang, Y., Ding, K., Hui, J.-Z., Lv, J.-X., Zhou, X.-L., & Zheng, P. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101792 · Cited by 80
Li, X., Xu, F., Liu, F., Tong, Y., Lyu, X., & Zhou, J. W. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3339291 · Cited by 82
“The transformer was designed based on the self-AM, which can attend to complete sequences thereby learning long-range dependencies [30], [31], [32].”
MethodWang, H., Xu, Y., He, Y., Cai, Y., Chen, L., Li, Y., Sotelo, M. Á., & Li, Z. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3196954 · Cited by 80
Multi-Task Learning With Multi-Query Transformer for Dense Prediction
Xu, Y., Li, X., Yuan, H., Yang, Y., & Zhang, L. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3292995 · Cited by 79
“And project them into the feature space, where the converter encoder computes the queries, keys and values, to generate the final result [12].”
BackgroundLightweight transformer-driven multi-scale trapezoidal attention network for saliency detection
Usman, M. T., Khan, H., Rida, I., & Koo, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110917 · Cited by 78
Yao, Z., Wang, Z., Huang, J., Xu, N., Cui, X., & Wu, T. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.175407 · Cited by 77
MAE-YOLOv8-based small object detection of green crisp plum in real complex orchard environments
Liu, Q., Lv, J., & Zhang, C. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109458 · Cited by 85
Cui, L., Jing, X., Wang, Y., Huan, Y., Xu, Y., & Zhang, Q. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3225150 · Cited by 76
“In recent years, several transformer-based vision models have been proposed for different computer vision tasks, such as target classification [28], object detection [29], and semantic segmentation [30], [31], [32].”
MethodpFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning
Guo, T., Guo, S., & Wang, J.-X. (2023) · The Web Conference · doi:10.1145/3543507.3583518 · Cited by 120
“To analyze the efect of the number of shots, we vary the shots in [1, 2, 4, 8, 16].”
MethodLiu, J., Hao, J., Lin, H., Pan, W., Yang, J., Feng, Y., Wang, G., Li, J., Jin, Z., Zhao, Z., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100825 · Cited by 75
Self-citationHu, W., Li, X., Li, C., Li, R., Jiang, T., Sun, H., Huang, X., Grzegorzek, M., & Li, X. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107034 · Cited by 86
Advances in attention mechanisms for medical image segmentation
Zhang, J., Chen, X., Yang, B., Guan, Q., Chen, Q., Chen, J., Wu, Q., Xie, Y., & Xia, Y. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2024.100721 · Cited by 143
Yang, Y., Miao, Z., Hua, Z., Wang, B., & Wu, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3351277 · Cited by 73
“Attention mechanism is a typical method to improve the performance of deep learning algorithms [49], [50].”
MethodData Augmentation Method for Plant Leaf Disease Recognition
Min, B., Kim, T., Shin, D., & Shin, D. (2023) · Applied Sciences · doi:10.3390/app13031465 · Cited by 87
Xiong, X., He, M., Li, T., Zheng, G., Xu, W., Fan, X., & Zhang, Y. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3367415 · Cited by 76
“The intention of the attention mechanism is to prompt the original network to prioritize crucial information during the learning process [36].”
BackgroundShen, J., Liu, N., Sun, H., Wu, S., Liang, Z., Han, L., Zhang, Y., & Li, D. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3642410 · Cited by 81
“The attention mechanism in neural networks is a resource allocation scheme that allocates computational resources to more important tasks in situations where computational capacity is limited, while addressing the issue of information overload [46].”
Chen, Y., Kang, Y., Hao, H., Xie, X., Zeng, J., Xu, T., Li, C., Tan, Y., & Fang, L. (2022) · Advanced Functional Materials · doi:10.1002/adfm.202209781 · Cited by 70
Self-citationA review of distributed solar forecasting with remote sensing and deep learning
Chu, Y., Wang, Y., Yang, D., Chen, S., & Li, M. (2024) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2024.114391 · Cited by 87
Giri, N., Roy, R. S., & Cheng, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102536 · Cited by 59
Machine Learning for Cataract Classification/Grading on Ophthalmic Imaging Modalities: A Survey
Zhang, X.-Q., Hu, Y., Xiao, Z., Fang, J., Higashita, R., & Liu, J. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1329-0 · Cited by 89
“In computer vision, attention can be classified into five categories: channel, spatial, temporal attention, branch attention, and attention combinations such as channel & spatial attention [54].”
BackgroundSelf-citationZhang, Z., Bao, L., Xiang, S., Xie, G., & Gao, R. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3409072 · Cited by 69
“Attention mechanisms [39] have gained significant attention in various visual tasks, including image fusion [40], [41], detection [42], [43], and semantic segmentation [44], highlighting their importance in the field.”
BackgroundAn overview: Attention mechanisms in multi-agent reinforcement learning
Hu, K., Xu, K., Xia, Q., Li, M., Song, Z., Song, L., & Sun, N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128015 · Cited by 69
Zhong, T., Qin, C., Shi, G., Zhang, Z., Tao, J., & Liu, C. (2024) · Science China Technological Sciences · doi:10.1007/s11431-024-2734-x · Cited by 68
Fu, R., Xiao, D., Navon, I. M., Fang, F., Yang, L., Wang, C., & Cheng, S. L. (2023) · International Journal for Numerical Methods in Engineering · doi:10.1002/nme.7240 · Cited by 71
A Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?
Gonçalves, T., Rio-Torto, I., Teixeira, L. F., & Cardoso, J. S. (2022) · IEEE Access · doi:10.1109/access.2022.3206449 · Cited by 83
“A similar approach was employed in Guo et al. [2021] for retinal vessel segmentation. This methodology was extended in Jin et al. [2020], which employs a 3D U-Net-based architecture with residual blocks and an attention module to extract volumes of interest in liver and brain CT imaging data and perform tumor segmenta…”
MethodInfluentialMineral Prospectivity Mapping Using Deep Self-Attention Model
Yin, B., Zuo, R., & Sun, S. (2022) · Natural Resources Research · doi:10.1007/s11053-022-10142-8 · Cited by 66
“…indispensable component for constructing deep learning models in various artificial intelligence fields, such as computer vision (Wang & Tax, 2016; Guo et al., 2022), natural language processing (Hu, 2019; Galassi et al., 2020), and speech recognition (Chorowski et al., 2015; Bahdanau et al,…”
BackgroundMethodAldakhil, L. A., Alhasson, H. F., & Alharbi, S. S. (2024) · Diagnostics · doi:10.3390/diagnostics14131402 · Cited by 73
“These mechanisms can be characterized as dynamic selection processes that adaptively adjust weights based on the significance of different features [23].”
BackgroundROULETTE: A neural attention multi-output model for explainable Network Intrusion Detection
Andresini, G., Appice, A., Caforio, F. P., Malerba, D., & Vessio, G. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117144 · Cited by 69
Cross-lingual font style transfer with full-domain convolutional attention
Zhao, H., Ji, T.-L., Rosin, P. L., Lai, Y., Meng, W., & Wang, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110709 · Cited by 65
“It is used for many vision tasks, such as image classification, object detection, semantic segmentation, etc. Attention can also be divided into four basic categories: channel attention, spatial attention, temporal attention, and branching attention [17].”
Scale-Context Perceptive Network for Crowd Counting and Localization in Smart City System
Zhai, W., Gao, M., Guo, X., Li, Q., & Jeon, G. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3268226 · Cited by 63
“The attention mechanism [18], [19], [20], [21] has been widely implemented in object counting by adjusting parameters to emphasize the foreground and weaken the background.”
BackgroundMultimodal Agent AI: A Survey of Recent Advances and Future Directions
Sun, Y., Sun, H., Ma, J., Zhang, P., & Huang, X. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-4802-8 · Cited by 72
Aung, P. S., Nguyen, L. X., Tun, Y. K., Han, Z., & Hong, C. S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3329893 · Cited by 62
“Attention mechanisms have been widely used in natural language processing, computer vision, and other domains to enhance model performance [37], [38].”
BackgroundMD-YOLO: Surface Defect Detector for Industrial Complex Environments
Zheng, H., Chen, X., Cheng, H., Du, Y., & Jiang, Z. (2024) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2024.108170 · Cited by 68
Zhou, F., Chen, Y., & Liu, J. (2023) · Remote Sensing · doi:10.3390/rs15051395 · Cited by 70
Self-citationSun, C., Zhang, R., Lu, Y., Cui, Y., Deng, Z., Cao, D., & Khajepour, A. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3321309 · Cited by 66
“Additionally, graph-based methods [136], [140] and attention-based fusion [141], [142], [143] offer powerful tools for merging features and automatically identifying their relationships.”
MethodRobust Optical and SAR Image Matching Using Attention-Enhanced Structural Features
Ye, Y., Yang, C., Gong, G., Yang, P., Quan, D., & Li, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3366247 · Cited by 62
Deep learning approaches for seizure video analysis: A review
Ahmedt‐Aristizabal, D., Armin, M. A., Hayder, Z., Garcia‐Cairasco, N., Petersson, L., Fookes, C., Denman, S., & McGonigal, A. (2024) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2024.109735 · Cited by 65
Deep Learning Frontiers in 3D Object Detection: A Comprehensive Review for Autonomous Driving
Pravallika, A., Hashmi, M. F., & Gupta, A. D. (2024) · IEEE Access · doi:10.1109/access.2024.3456893 · Cited by 69
“There are five sorts of attention methods: channel attention, spatial attention, temporal attention, branch attention, and hybrid attention [214].”
Yu, X., & Li, D. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3351755 · Cited by 60
“The attention mechanism [19] has evolved into an increasingly significant component of computer vision over the last decade.”
BackgroundHuang, Z., Qin, Y., Lin, X., Liu, T., Feng, Z., & Liu, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3217763 · Cited by 59
“However, note that deep learning attention mechanisms generally involve convolution layers, fully connected layers, parameterized activation functions, etc [37], [38].”
BackgroundMethod
Metadata
- Authors
- Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M.
- Affiliations
- Tsinghua University (CN)
- TKLNDST, College of Computer Science, Nankai University, Tianjin 300350, China
- Nankai University (CN)
- Cardiff University (GB)
- Corresponding author
- Hu, Shi-Min
- ORCID
- Venue
- Computational Visual Media
- Publisher
- Tsinghua University Press
- Published
- 2022-09
- Volume
- 8
- Issue
- 3
- Pages
- 331-368
- ISSN
- 2096-0433 (ISSN-L)
- 2096-0662
- OpenAlex
- W3212386989
- Cited by
- 2,566
- References
- 213
- Citations by year
- 2026: 404 · 2025: 687 · 2024: 739 · 2023: 576 · 2022: 159
- FWCI
- 176.50
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Computational Visual Media (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62132012, 61521002)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- attention mechanism, visual attention, channel attention, spatial attention, temporal attention, image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multimodal tasks, self-supervised learning, human visual system
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
