Understanding Self-attention Mechanism via Dynamical System Perspective
Open accessAbstract
The self-attention mechanism (SAM) is widely used in various fields of artificial intelligence and has successfully boosted the performance of different models. However, current explanations of this mechanism are mainly based on intuitions and experiences, while there still lacks direct modeling for how the SAM helps performance. To mitigate this issue, in this paper, based on the dynamical system perspective of the residual neural network, we first show that the intrinsic stiffness phenomenon (SP) in the high-precision solution of ordinary differential equations (ODEs) also widely exists in high-performance neural networks (NN). Thus the ability of NN to measure SP at the feature level is necessary to obtain high performance and is an important factor in the difficulty of training NN. Similar to the adaptive step-size method which is effective in solving stiff ODEs, we show that the SAM is also a stiffness-aware step size adaptor that can enhance the model's representational ability to measure intrinsic SP by refining the estimation of stiffness information and generating adaptive attention values, which provides a new understanding about why and how the SAM can benefit the model performance. This novel perspective can also explain the lottery ticket hypothesis in SAM, design new quantitative metrics of representational ability, and inspire a new theoretic-inspired approach, StepNet. Extensive experiments on several popular benchmarks demonstrate that StepNet can extract fine-grained stiffness information and measure SP accurately, leading to significant improvements in various visual tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The self-attention mechanism enables the model to compute pairwise interactions between all patch tokens, capturing both short-range and long-range dependencies, which are crucial in HAR for distinguishing activities with overlapping or variable-length motions.
“The self-attention mechanism enables the model to compute pairwise interactions between all patch tokens, allowing it to capture both short-range and long-range dependencies which are crucial in HAR for distinguishing activities with overlapping or variable-length motions.”
From A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI (2025) · cites this paper as [43]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks
Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01155 · Cited by 9,641 · cited 5× · see citing works
“Introduction The self-attention mechanism (SAM) [43, 16, 17, 9, 45, 4] is widely used in various artificial intelligence fields and has successfully improved the models’ performance in a number of vision tasks, including image classification [24, 64, 49], object detection [39, 60, 27], instance seg-”
BackgroundMethodInfluentialImage Super-Resolution Using Very Deep Residual Channel Attention Networks
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_18 · Cited by 6,003 · see citing works
A NEW MEASURE OF RANK CORRELATION
Kendall, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 5,988 · see citing works
Matching Networks for One Shot LearningPreprint
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04080 · Cited by 4,874 · see citing works
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · see citing works
UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation
Huang, H., Lin, L., Tong, R., Hu, H., Zhang, Q., Iwamoto, Y., Han, X., Chen, Y., & Wu, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053405 · Cited by 3,247 · see citing works
Self-citationNumerical Methods for Ordinary Differential Equations
Butcher, J. C. (2016) · Wiley eBooks · doi:10.1002/9781119121534 · Cited by 3,020 · see citing works
Pruning Filters for Efficient ConvNets
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016) · arXiv · doi:10.48550/arxiv.1608.08710 · Cited by 2,928 · see citing works
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,760 · see citing works
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566 · cited 1× · see citing works
“The self-attention mechanism (SAM) [41, 15, 16, 9, 43, 4] is widely used in various artificial intelligence fields and has successfully improved the models’ performance in a number of vision tasks, including image classification [22, 60, 47], object detection [37, 56, 25], instance seg-* co-first…”
A New Measure of Rank Correlation
Kendall, M. G. (1938) · Biometrika · doi:10.2307/2332226 · Cited by 2,346 · see citing works
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,242 · see citing works
Neural ordinary differential equations
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018) · arXiv · doi:10.48550/arxiv.1806.07366 · Cited by 2,097 · see citing works
FcaNet: Frequency Channel Attention Networks
Qin, Z., Zhang, P., Wu, F. Q., & Li, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00082 · Cited by 1,513 · see citing works
BAM: Bottleneck Attention ModulePreprint
Park, J., Woo, S., Lee, J., & Kweon, I. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.06514 · Cited by 951 · see citing works
Single Image Super-Resolution via a Holistic Attention Network
Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., & Shen, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_12 · Cited by 849 · see citing works
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep ArchitecturesPreprint
Hu, H., Peng, R., Tai, Y., & Tang, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.03250 · Cited by 792 · see citing works
A Proposal on Machine Learning via Dynamical Systems
Weinan, E. (2017) · Communications in Mathematics and Statistics · doi:10.1007/s40304-017-0103-z · Cited by 606 · see citing works
Fehlberg, E. (1969) · NASA Technical Reports Server (NASA) · Cited by 499 · see citing works
Research Design and Statistical Analysis
Myers, J. L., Well, A. D., & Lorch, R. F. (2013) · Routledge eBooks · doi:10.4324/9780203726631 · Cited by 476 · see citing works
SRM: A Style-Based Recalibration Module for Convolutional Neural Networks
Lee, H., Kim, H., & Nam, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00194 · Cited by 374 · see citing works
End-to-End Instance Segmentation with Recurrent Attention
Ren, M., & Zemel, R. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.39 · Cited by 309 · see citing works
Visformer: The Vision-friendly Transformer
Chen, Z., Xie, L., Niu, J., Liu, X., Wei, L., & Tian, Q. C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00063 · Cited by 271 · see citing works
Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional NetworksPreprint
Li, X., Hu, X., & Yang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09646 · Cited by 241 · see citing works
Lu, Y., Zhong, A., Li, Q., & Dong, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10121 · Cited by 235 · see citing works
Towards Robust Vision Transformer
Mao, X., Qi, G., Chen, Y., Li, X., Duan, R., Ye, S., He, Y., & Xue, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01173 · Cited by 224 · see citing works
Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition
Chen, T., Lin, L., Chen, R., Hui, X., & Wu, H. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3025814 · Cited by 195 · see citing works
Self-citationLightweight Image Super-Resolution With Expectation-Maximization Attention Mechanism
Zhu, X., Guo, K., Ren, S., Hu, B., Hu, M., & Fang, H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3078436 · Cited by 186 · see citing works
Channel-level acceleration of deep face representations
Polyak, A., & Wolf, L. (2015) · IEEE Access · doi:10.1109/access.2015.2494536 · Cited by 171 · see citing works
Chen, T., Pu, T., Wu, H., Xie, Y., Liu, L., & Lin, L. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3131222 · Cited by 150 · see citing works
Self-citationGlobal Sparse Momentum SGD for Pruning Very Deep Neural Networks
Ding, X., Ding, G., Zhou, X., Guo, Y., Han, J., & Liu, J. (2019) · Warwick Research Archive Portal (University of Warwick) · doi:10.48550/arxiv.1909.12778 · Cited by 145 · see citing works
Introduction to Calculus and Analysis
Courant, R., & John, F. (1989) · Springer eBooks · doi:10.1007/978-1-4613-8958-3 · Cited by 135 · see citing works
Symplectic Recurrent Neural NetworksPreprint
Chen, Z., Zhang, J., Arjovsky, M., & Bottou, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.13334 · Cited by 124 · see citing works
Efficient Image Super-Resolution Using Vast-Receptive-Field Attention
Zhou, L. Y., Cai, H., Gu, J., Li, Z., Liu, Y., Chen, X., Qiao, Y., & Dong, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25063-7_16 · Cited by 121 · see citing works
Multi-level Residual Networks from Dynamical Systems ViewPreprint
Chang, B., Meng, L., Haber, E., Tung, F., & Begert, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10348 · Cited by 116 · see citing works
Multi-scale feature fusion residual network for Single Image Super-Resolution
Qin, J., Huang, Y., & Wen, W. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.10.076 · Cited by 111 · see citing works
Self-citationHuang, W., & Leimkuhler, B. (1997) · SIAM Journal on Scientific Computing · doi:10.1137/s1064827595284658 · Cited by 110 · see citing works
Numerical Solution of Ordinary Differential Equations
Shampine, L. F. (2018) · Routledge eBooks · doi:10.1201/9780203745328 · Cited by 78 · see citing works
A New Attention Mechanism to Classify Multivariate Time Series
Hao, Y., & Cao, H. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/277 · Cited by 64 · see citing works
Convolutional neural networks combined with Runge–Kutta methods
Zhu, M., Chang, B., & Fu, C. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07785-2 · Cited by 59 · see citing works
Instance Enhancement Batch Normalization: An Adaptive Regulator of Batch Noise
Liang, S., Huang, Z., Liang, M., & Yang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5917 · Cited by 53 · see citing works
Self-citationCAP: Context-Aware Pruning for Semantic Segmentation
He, W. P., Wu, M., Liang, M., & Lam, S.-K. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00100 · Cited by 45 · see citing works
Self-citationDIANet: Dense-and-Implicit Attention Network
Huang, Z., Liang, S., Liang, M., & Yang, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5842 · Cited by 43 · see citing works
Self-citationSupervised Edge Attention Network for Accurate Image Instance Segmentation
Chen, X., Lian, Y., Jiao, L., Wang, H., Gao, Y., & Lingling, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58583-9_37 · Cited by 43 · see citing works
RS-DARTS: A Convolutional Neural Architecture Search for Remote Sensing Image Scene Classification
Zhang, Z., Liu, S., Zhang, Y., & Chen, W. (2021) · Remote Sensing · doi:10.3390/rs14010141 · Cited by 39 · see citing works
Rethinking the Pruning Criteria for Convolutional Neural Network
Huang, Z., Wang, X., & Luo, P. (2021) · neural information processing systems · Cited by 38 · see citing works
Self-citationAttention Inspiring Receptive-Fields Network for Learning Invariant Representations
Yang, L., Song, Q., Wu, Y., & Hu, M. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2873722 · Cited by 36 · see citing works
Continuous-in-Depth Neural NetworksPreprint
Queiruga, A. F., Erichson, N. B., Taylor, D., & Mahoney, M. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02389 · Cited by 24 · see citing works
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency EncodingPreprint
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06097 · Cited by 19 · see citing works
Blending Pruning Criteria for Convolutional Neural Networks
He, W. P., Huang, Z., Liang, M., Liang, S., & Yang, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86380-7_1 · Cited by 11 · see citing works
Self-citationConvolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel PruningPreprint
Huang, Z., Shao, W., Wang, X., Lin, L., & Luo, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11627 · Cited by 10 · see citing works
Self-citationEfficient Attention Network: Accelerate Attention by Searching Where to PlugPreprint
Huang, Z., Liang, S., Liang, M., Wei, H., & Yang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14058 · Cited by 9 · see citing works
Self-citationThe Lottery Ticket Hypothesis for Self-attention in Convolutional Neural NetworkPreprint
Huang, Z., Liang, S., Liang, M., He, W. P., Yang, H., & Lin, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07858 · Cited by 9 · see citing works
Self-citationA generic shared attention mechanism for various backbone neural networks
Huang, Z., Liang, S., & Liang, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128697 · Cited by 8 · see citing works
Self-citationA Dynamical System Perspective for Lipschitz Neural NetworksPreprint
Meunier, L., Delattre, B., Araujo, A., & Allauzen, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12690 · Cited by 8 · see citing works
Efficient-Receptive Field Block with Group Spatial Attention Mechanism for Object Detection
Zhang, J., Zhao, Z., & Su, F. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412772 · Cited by 6 · see citing works
Switchable Self-attention ModulePreprint
Zhong, S., Wen, W., & Qin, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.05680 · Cited by 4 · see citing works
Self-citationA Generic Shared Attention Mechanism for Various Backbone Neural NetworksPreprint
Huang, Z., Liang, S., Liang, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.16101 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Rasool, A., Shahzad, M. I., Aslam, H., Chan, V., & Arshad, M. A. (2025) · AI · doi:10.3390/ai6030056 · Cited by 61 · see citing works
Liao, M., Tang, H., Li, X., Vijayakumar, P., Arya, V., & Gupta, B. B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102401 · Cited by 30 · see citing works
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-26297-2 · Cited by 26 · see citing works
Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima
Zhong, S., Huang, Z., Li, D., Wen, W., Qin, J., & Lin, L. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645553 · Cited by 20 · see citing works
Self-citationShao, X., Xu, L., Sun, G., Yao, W., Wu, L., & Santina, C. D. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3590174 · Cited by 19 · see citing works
CurveFormer++: 3D Lane Detection by Curve Propagation With Temporal Curve Queries and Attention
Bai, Y., Chen, Z., Liang, P., Song, B., & Cheng, E. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3560337 · Cited by 19 · see citing works
Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer
Shehzadi, T., Ifza, I., Liwicki, M., Stricker, D., & Afzal, M. Z. (2026) · Sensors · doi:10.3390/s26010310 · Cited by 18 · see citing works
A facial expression recognition network using hybrid feature extraction
Song, D., & Liu, C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0312359 · Cited by 15 · see citing works
Optimization of Concentrated Solar Power Subsystems with a focus on green certificate acquisition
Wu, B., Wang, X., Wang, B., Xie, Y., Qi, S., Sun, W., Huang, Q., & Ma, X. (2025) · Solar Energy · doi:10.1016/j.solener.2025.113242 · Cited by 12 · see citing works
A dynamic routing-guided interpretable framework for salt–solvent chemistry
Wang, Z., & You, F. (2026) · Nature Computational Science · doi:10.1038/s43588-026-00955-5 · Cited by 9 · see citing works
A generic shared attention mechanism for various backbone neural networks
Huang, Z., Liang, S., & Liang, M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128697 · Cited by 8 · see citing works
Self-citationLearning Dynamic Batch-Graph Representation for Deep Representation Learning
Wang, X., Jiang, B., Wang, X., & Luo, B. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02175-8 · Cited by 6 · see citing works
Miao, H., Zhang, J., Jiang, Y., & Zhou, J. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-02092-4 · Cited by 5 · see citing works
Zhao, W., Zhong, S., Liu, Y., Wen, W., Qin, J., Liang, M., & Huang, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714840 · Cited by 4 · see citing works
Self-citationZhang, H., Chen, J., Liao, B., Wu, F., & Bi, X.-A. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00625-y · Cited by 4 · see citing works
An Aerial Target Recognition Algorithm Based on Self-Attention and LSTM
Liang, F., Xin, C., He, S., Song, Z., & Lü, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.055326 · Cited by 4 · see citing works
Yield prediction method for regenerated rice based on hyperspectral image and attention mechanisms
Hu, T., Liu, Z., Hu, R., Zeng, L., Deng, K., Dong, H., Li, M., & Deng, Y. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.100804 · Cited by 4 · see citing works
Enhancing Wind Power Forecasting Accuracy Based on OPESC-Optimized CNN-BiLSTM-SA Model
Wang, L., & Zhang, D. (2025) · Mathematics · doi:10.3390/math13132174 · Cited by 3 · see citing works
Wen, X., & Fu, X. (2024) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625500835 · Cited by 2 · see citing works
Liu, J., Li, J., & Ren, Y. (2025) · Remote Sensing · doi:10.3390/rs17173122 · Cited by 2 · see citing works
DEEPAM: Toward Deeper Attention Module in Residual Convolutional Neural Networks
Zhong, S., Wen, W., Qin, J., & Huang, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72332-2_26 · Cited by 2 · see citing works
Self-citationImproved Neural Network Algorithm Combining Adaptive Gradient Clipping and Self-Attention Mechanism
Huang, S., Xing, J., & Li, Y. (2024) · International Symposium on Big Data and Artificial Intelligence (ISBDAI) · doi:10.1145/3723366.3723369 · Cited by 2 · see citing works
Stripe Observation Guided Inference Cost-Free Attention Mechanism
Huang, Z., Zhong, S., Wen, W., Qin, J., & Lin, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_6 · Cited by 1 · see citing works
Self-citationCustomizable scenario generation for clean energy bases
Wu, B., Wang, X., Guan, L., & Li, P. (2025) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2025.111261 · Cited by 1 · see citing works
Zhao, M., Wang, L., & Huang, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-49575-z · Cited by 1 · see citing works
Flat Local Minima for Continual Learning on Semantic Segmentation
Huang, Z., Liang, M., Liang, S., & Zhong, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2054-8_29 · Cited by 1 · see citing works
Self-citationShawly, T., & Alsheikhy, A. A. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.061497 · Cited by 1 · see citing works
Lottery Ticket Hypothesis for Attention Mechanism in Residual Convolutional Neural Network*
Huang, Z., Liang, S., Liang, M., He, W. P., Yang, H., & Lin, L. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688205
Self-citationIrt: An Ink-Aware Transformer with Cross-Scale Attention for Fine-Grained Painting Crack RestorationPreprint
He, X., & Hea, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5256464
Liu, J., Hong, Y., Hu, B., Chen, S., Deng, J., Feng, C., & jie, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5257901
Zhang, D., Cheng, H. D., Zhang, J., & Gao, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5380392
H-FEX: A symbolic learning method for Hamiltonian systems
Lai, J., Liang, S., & Wang, C. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109248
A YOLOv5 Model with Hybrid Attention Mechanism for Transparent Product Detection in E-Commerce
张, 伟. (2025) · E-Commerce Letters · doi:10.12677/ecl.2025.14103364
AGDAformer: Agent-Guidance Dual Attention Transformer for Climate-Aware Crop Yield Prediction
Yi, Z., Fang, Z., Xu, R., Wang, J., & He, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228801
Deep learning-based investigation of chloroplast translation regulatory sequences
Abbasi-Vineh, M. A., Ingvarsson, P. K., & Farrokhi, N. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1698951
A Comparative Study of Remaining Useful Life Prediction for NASA Turbofan Engines
Swain, A., Wadhvani, R., & Rasool, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-10783-1_20
Wei, J. (2025) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml67980.2025.11333678
CTMCNet: An Efficient Deep Learning Framework for Sub-Nyquist Modulation Recognition
Liu, Y., & Li, S. (2025) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact67584.2025.11345037
Natrayan, L. (2026) · International Conference on Cognitive Computing in Engineering, Communications, Sciences and Biomedical Health Informatics (IC3ECSBHI) · doi:10.1109/ic3ecsbhi67834.2026.11469153
Sentence Extraction Framework with High Relevance and Divergence for Document Summarization
Xue, H., Li, B., Ma, D., Lv, X., & Wang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-3827-0_32
An Attentive Multi-Scale Approach for Semantic Segmentation
Neves, V. G., Gomes, I. P., Reyna, A. H., Amaral, W. M., & Wolf, D. F. (2026) · Brazilian Conference on Robotics (CROS) · doi:10.1109/cros69211.2026.11565769
Khandelwal, S., Gupta, P., & Bhardwaj, J. (2025) · IEEE International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA) · doi:10.1109/icccmla66092.2025.11581160
Wang, Z., & You, F. (2026) · Science Advances · doi:10.1126/sciadv.aee4959
The Ethics of Using Generative AI Tools in Jurisprudence: Problems of Normative Legal Regulation
Antsiferova, N. A., & Борисовна, Ш. А. (2026) · Legal Order and Legal Values · doi:10.23947/2949-1843-2026-4-3-17-23
- Authors
- Huang, Z., Liang, M., Qin, J., Zhong, S., & Lin, L.
- Affiliations
- Sun Yat-sen University (CN)
- Northwestern University (US)
- Guangdong University of Technology (CN)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 1412-1422
- OpenAlex
- W4390874069
- Cited by
- 44
- References
- 66
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 11
- FWCI
- 4.71
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- self-attention, ResNet, stiff ordinary differential equations, adaptive step size, adaptive attention, representational ability, lottery ticket hypothesis, visual task
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
