The DOI matches its Crossref record; 112 of the 112 references it cites were found in a registry or an index.
An Attentive Survey of Attention Models
Open accessAbstract
Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy that groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.10903 · Cited by 12,107
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1174 · Cited by 4,712
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Nadaraya, E. A. (1964) · Theory of Probability & Its Applications · doi:10.1137/1109020 · Cited by 3,462
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353
Heterogeneous Graph Attention Network
Wang, X., Ji, H., Shi, C., Wang, B., Ye, Y., Cui, P., & Yu, P. S. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313562 · Cited by 3,191
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Self-Attentive Sequential Recommendation
Kang, W.-C., & McAuley, J. (2018) · 2018 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2018.00035 · Cited by 3,011
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Watson, G. S. (1964) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 2,661
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
KGAT: Knowledge Graph Attention Network for Recommendation
Wang, X., He, X., Cao, Y., Liu, M., & Chua, T.-S. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330989 · Cited by 2,147
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,043
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743
GMAN: A Graph Multi-Attention Network for Traffic Prediction
Zheng, C., Fan, X., Wang, C., & Qi, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5477 · Cited by 1,730
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 1,511
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198
Jain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,179
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K., & Schmid, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,172
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146
Interactive Attention Networks for Aspect-Level Sentiment Classification
Ma, D., Li, S., Zhang, X., & Wang, H. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/568 · Cited by 1,069
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
Aspect Level Sentiment Classification with Deep Memory Network
Tang, D., Qin, B., & Liu, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1021 · Cited by 1,041
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035
(2020) · International Conference on Machine Learning · Cited by 1,012
Exploring Self-Attention for Image Recognition
Zhao, H., Jia, J., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01009 · Cited by 1,002
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977
Deep Modular Co-Attention Networks for Visual Question Answering
Yu, Z., Yu, J., Cui, Y., Tao, D., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00644 · Cited by 964
Abstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1012 · Cited by 881
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., & Chen, L.-C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_7 · Cited by 809
DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., & Zhang, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11941 · Cited by 779
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 10×
“Clark et al. [2019] perform analysis of Transformer-based (pre-trained) language model called Bidirectional Encoder Representations (BERT) to determine that particular heads correspond well to particular relations such as direct objects of verbs, determiners of nouns, objects of prepositions, and so on. Further, first…”
BackgroundMethodResultInfluentialSerrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690
Relational Graph Attention Network for Aspect-based Sentiment Analysis
Wang, K., Shen, W., Yang, Y., Quan, X., & Wang, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.295 · Cited by 665
Attention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 635
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM
Ma, Y., Peng, H., & Cambria, E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12048 · Cited by 592
Ma, F., Chitta, R., Zhou, J., You, Q., Sun, T., & Gao, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098088 · Cited by 576
NAIS: Neural Attentive Item Similarity Model for Recommendation
He, X., He, Z., Song, J., Liu, Z., Jiang, Y.-G., & Chua, T.-S. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2831682 · Cited by 569
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1151 · Cited by 482
(2021) · International Conference on Learning Representations · Cited by 449
From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
Martins, A. F. T., & Astudillo, R. F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02068 · Cited by 431
Coupled Multi-Layer Attentions for Co-Extraction of Aspect and Opinion Terms
Wang, W., Pan, S. J., Dahlmeier, D., & Xiao, X. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10974 · Cited by 425
Understanding Neural Networks through Representation Erasure
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08220 · Cited by 412
Graph-to-Sequence Learning using Gated Graph Neural Networks
Beck, D., Haffari, G., & Cohn, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1026 · Cited by 380
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J. (2024) · TIB Data Manager · doi:10.57702/uhdmfu1r · Cited by 357
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks
Cho, K., Courville, A., & Bengio, Y. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477044 · Cited by 338
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · arXiv · doi:10.48550/arxiv.1901.09451 · Cited by 337
Graph Classification using Structural Attention
Lee, J. B., Rossi, R., & Kong, X. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219980 · Cited by 280
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, E., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1032 · Cited by 279
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
Tang, G., Müller, M., Rios, A., & Sennrich, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1458 · Cited by 273
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.574 · Cited by 232
Attention Models in Graphs: A Survey
Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., & Koh, E. (2019) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3363574 · Cited by 231
Sparse Sequence-to-Sequence Models
Peters, B., Niculae, V., & Martins, A. F. T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1146 · Cited by 196
DAN: Deep Attention Neural Network for News Recommendation
Zhu, Q., Zhou, X., Song, Z., Tan, J., & Guo, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015973 · Cited by 192
Watch Your Step: Learning Node Embeddings via Graph AttentionPreprint
Abu-El-Haija, S., Perozzi, B., Al‐Rfou, R., & Alemi, A. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.09599 · Cited by 181
DETERRENT: Knowledge Guided Graph Attention Network for Detecting Healthcare Misinformation
Cui, L., Seo, H., Tabar, M., Ma, F., Wang, S., & Lee, D. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403092 · Cited by 163
A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation
Zhu, F., Wang, Y., Chen, C., Liu, G., & Zheng, X. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/415 · Cited by 158
Deeper Attention to Abusive User Content Moderation
Pavlopoulos, J., Malakasiotis, P., & Androutsopoulos, I. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1117 · Cited by 151
SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and DecompositionPreprint
Lin, Z., Wu, Y., Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang, J., & Ahn, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.02407 · Cited by 145
Survey on the attention based RNN model and its applications in computer visionPreprint
Wang, F., & Tax, D. M. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06823 · Cited by 138
Neural Machine Translation With GRU-Gated Attention Model
Zhang, B., Xiong, D., Xie, J., & Su, J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2957276 · Cited by 135
Dynamic Meta-Embeddings for Improved Sentence Representations
Kiela, D., Wang, C., & Cho, K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1176 · Cited by 132
Monotonic Multihead AttentionPreprint
Ma, X., Pino, J., Cross, J., Puzon, L., & Gu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12406 · Cited by 117
Improved Knowledge Distillation via Teacher Assistant
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., & Ghasemzadeh, H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5963 · Cited by 101 · cited 2×
“Moreover, they observed that introducing a teacher assistant (a concept introduced in Mirzadeh et al. [2020]) also helps the distillation of large pre-trained Transformer models. Similarly, Touvron et al. [2020] employed a teacher-student strategy specific to transformers ensuring that the student learns from the teac…”
BackgroundA Hierarchical Attention Model for Social Contextual Image Recommendation
Wu, L., Chen, L., Hong, R., Fu, Y., Xie, X., & Wang, M. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2913394 · Cited by 101
Towards Transparent and Explainable Attention Models
Mohankumar, A. K., Nema, P., Narasimhan, S., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.387 · Cited by 91
GAKE: Graph Aware Knowledge Embedding
Feng, J., Huang, M., Yang, Y., & Zhu, X. (2016) · International Conference on Computational Linguistics · Cited by 77
Interactive Visualization and Manipulation of Attention-based Neural Machine Translation
Lee, J., Shin, J.-H., & Kim, J. (2017) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d17-2021 · Cited by 77
Self-Attention Guided Copy Mechanism for Abstractive Summarization
Xu, S., Li, H., Yuan, P., Wu, Y., He, X., & Zhou, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.125 · Cited by 74
Self-Attention: A Better Building Block for Sentiment Analysis Neural Network Classifiers
Ambartsoumian, A., & Popowich, F. P. (2018) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/w18-6219 · Cited by 70
Hierarchical User and Item Representation with Three-Tier Attention for Recommendation
Wu, C., Wu, F., Liu, J., & Huang, Y. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1180 · Cited by 46
Liu, S.-S., Li, T., Li, Z.-M., Srikumar, V., Pascucci, V., & Bremer, P. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2007 · Cited by 39
Sparse and Continuous Attention MechanismsPreprint
Martins, A. F. T., Farinhas, A., Treviso, M. V., Niculae, V., Aguiar, P. M. Q., & Figueiredo, M. A. T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07214 · Cited by 26
A Genre-Aware Attention Model to Improve the Likability Prediction of Books
Maharjan, S., Montes, M., Gonzalez, F. A., & Solorio, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1375 · Cited by 23
Stacked Self-Attention Networks for Visual Question Answering
Sun, Q., & Fu, Y. (2019) · International Conference on Multimedia Retrieval · doi:10.1145/3323873.3325044 · Cited by 22
Ma, B., Zhang, J., Xia, Y., & Tao, D. (2020) · neural information processing systems · Cited by 22
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Deng, Z., Peng, H., Xia, C., Li, J., He, L., & Yu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.555 · Cited by 19
Generative Attention Networks for Multi-Agent Behavioral Modeling
Li, G., Jiang, B., Zhu, H., Che, Z., & Liu, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6209 · Cited by 19
Compositional De-Attention Networks
Tay, Y., Luu, A. T., Zhang, A., Wang, S., & Hui, S. C. (2019) · neural information processing systems · Cited by 18
Wang, Y., Li, J., Lyu, M. R.-T., & King, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.268 · Cited by 18
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive ConnectionPreprint
Li, X., Meng, Y., Zhou, M., Han, Q., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09833 · Cited by 17
Towards Understanding Attention-Based Speech Recognition Models
Qin, C.-X., & Qu, D. (2020) · IEEE Access · doi:10.1109/access.2020.2970758 · Cited by 7
Policy learning with partial observation and mechanical constraints for multi-person modelingPreprint
Fujii, K., Takeishi, N., Kawahara, Y., et al. (2020) · arXiv (Cornell University) · Cited by 7
Social explorative attention based recommendation for content distribution platforms
Xiao, W., Zhao, H., Fang, T., Song, Y., Zheng, V. W., & Yang, Q. (2021) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-020-00729-1 · Cited by 6
Research works printed but not linked: 42
- 10.5555/3327546.3327592 · doi:10.5555/3327546.3327592
- Lei Jimmy Ba , Volodymyr Mnih , and Koray Kavukcuoglu . 2014 . Multiple object recognition with visual attention . In Proceedings of the International Conference on Learning Representations. Lei Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2014. Multiple object recognition with visual attention. In Proceedings of the International Conference on Learning Representations.
- Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2015 . Neural machine translation by jointly learning to align and translate . In Proceedings of the 3rd International Conference on Learning Representations. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.
- Chung-Cheng Chiu and Colin Raffel. 2017. Monotonic chunkwise attention. arXiv:1712.05382. Retrieved from https://arxiv.org/abs/1712.05382. Chung-Cheng Chiu and Colin Raffel. 2017. Monotonic chunkwise attention. arXiv:1712.05382. Retrieved from https://arxiv.org/abs/1712.05382. · arXiv
- 10.5555/2969239.2969304 · doi:10.5555/2969239.2969304
- Mostafa Dehghani , Stephan Gouws , Oriol Vinyals , Jakob Uszkoreit , and Lukasz Kaiser . 2019 . Universal transformers . In Proceedings of the 7th International Conference on Learning Representations. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In Proceedings of the 7th International Conference on Learning Representations.
- Alex Graves Greg Wayne and Ivo Danihelka. 2014b. Neural turing machines. arXiv:1410.5401. Retrieved from https://arxiv.org/abs/1410.5401. Alex Graves Greg Wayne and Ivo Danihelka. 2014b. Neural turing machines. arXiv:1410.5401. Retrieved from https://arxiv.org/abs/1410.5401. · arXiv
- 10.5555/3045118.3045274 · doi:10.5555/3045118.3045274
- 10.5555/2969239.2969428 · doi:10.5555/2969239.2969428
- Saumya Jetley , Nicholas A. Lord , Namhoon Lee , and Philip Torr . 2018 . Learn to pay attention . In Proceedings of the International Conference on Learning Representations. Saumya Jetley, Nicholas A. Lord, Namhoon Lee, and Philip Torr. 2018. Learn to pay attention. In Proceedings of the International Conference on Learning Representations.
- Salman Khan , Muzammal Naseer , Munawar Hayat , Syed Waqas Zamir , Fahad Shahbaz Khan, and Mubarak Shah. 2021 . Transformers in vision: A survey. arXiv:2101.01169. Retrieved from https://arxiv.org/abs/2101.01169. Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey. arXiv:2101.01169. Retrieved from https://arxiv.org/abs/2101.01169. · arXiv
- 10.5555/3045390.3045536 · doi:10.5555/3045390.3045536
- Yang Li , Lukasz Kaiser , Samy Bengio , and Si Si . 2019 a. Area attention . In Proceedings of the International Conference on Machine Learning. PMLR, 3846–3855 . Yang Li, Lukasz Kaiser, Samy Bengio, and Si Si. 2019a. Area attention. In Proceedings of the International Conference on Machine Learning. PMLR, 3846–3855.
- 10.5555/3454287.3454289 · doi:10.5555/3454287.3454289
- 10.5555/3157096.3157129 · doi:10.5555/3157096.3157129
- Thang Luong , Hieu Pham , and Christopher D. Manning . 2015b. Effective approaches to attention-based neural machine translation . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1412–1421 . Thang Luong, Hieu Pham, and Christopher D. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1412–1421.
- 10.5555/3171837.3171854 · doi:10.5555/3171837.3171854
- 10.5555/3045390.3045561 · doi:10.5555/3045390.3045561
- 10.5555/3454287.3455544 · doi:10.5555/3454287.3455544
- 10.5555/2969033.2969073 · doi:10.5555/2969033.2969073
- Niki Parmar , Ashish Vaswani , Jakob Uszkoreit , Lukasz Kaiser , Noam Shazeer , Alexander Ku , and Dustin Tran . 2018. Image transformer.Proceedings of Machine Learning Research , Vol. 80 . 4055–4064. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer.Proceedings of Machine Learning Research, Vol. 80. 4055–4064.
- 10.1145/3289600.3290609 · doi:10.1145/3289600.3290609
- Alessandro Sordoni Philip Bachman Adam Trischler and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. arXiv:1606.02245. Retrieved from https://arxiv.org/abs/1606.02245. Alessandro Sordoni Philip Bachman Adam Trischler and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. arXiv:1606.02245. Retrieved from https://arxiv.org/abs/1606.02245. · arXiv
- Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski and Armand Joulin. 2019b. Adaptive attention span in transformers. arXiv:1905.07799. Retrieved from https://arxiv.org/abs/1905.07799. Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski and Armand Joulin. 2019b. Adaptive attention span in transformers. arXiv:1905.07799. Retrieved from https://arxiv.org/abs/1905.07799. · arXiv
- 10.5555/2969442.2969512 · doi:10.5555/2969442.2969512
- 10.5555/3454287.3454838 · doi:10.5555/3454287.3454838
- Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2020. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2020. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. · arXiv
- Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. · arXiv
- 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
- 10.5555/2969442.2969540 · doi:10.5555/2969442.2969540
- 10.5555/3298023.3298050 · doi:10.5555/3298023.3298050
- Jason Weston Sumit Chopra and Antoine Bordes. 2014. Memory networks. arXiv:1410.3916. Retrieved from https://arxiv.org/abs/1410.3916. Jason Weston Sumit Chopra and Antoine Bordes. 2014. Memory networks. arXiv:1410.3916. Retrieved from https://arxiv.org/abs/1410.3916. · arXiv
- 10.5555/3045390.3045643 · doi:10.5555/3045390.3045643
- Kelvin Xu , Jimmy Ba , Ryan Kiros , Kyunghyun Cho , Aaron Courville , Ruslan Salakhudinov , Rich Zemel , and Yoshua Bengio . 2015 . Proceedings of Machine Learning Research , Vol. 37 . 2048–2057. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Proceedings of Machine Learning Research, Vol. 37. 2048–2057.
- 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804
- 10.5555/3304222.3304315 · doi:10.5555/3304222.3304315
- 10.5555/3304222.3304315 · doi:10.5555/3304222.3304315
- Amir Zadeh , Paul Pu Liang , Soujanya Poria , Prateek Vij , Erik Cambria , and Louis-Philippe Morency . 2018 . Multi-attention recurrent network for human communication comprehension . In Proceedings of the AAAI Conference on Artificial Intelligence. 5642–5649 . Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018. Multi-attention recurrent network for human communication comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence. 5642–5649.
- Ruochi Zhang , Yuesong Zou , and Jian Ma . 2020 . Hyper-SAGNN: A self-attention based graph neural network for hypergraphs . In Proceedings of the International Conference on Learning Representations. Ruochi Zhang, Yuesong Zou, and Jian Ma. 2020. Hyper-SAGNN: A self-attention based graph neural network for hypergraphs. In Proceedings of the International Conference on Learning Representations.
- Shenjian Zhao and Zhihua Zhang . 2018. Attention-via-attention neural machine translation . In Association for the Advancement of Artificial Intelligence. Shenjian Zhao and Zhihua Zhang. 2018. Attention-via-attention neural machine translation. In Association for the Advancement of Artificial Intelligence.
- 10.5555/3172077.3172382 · doi:10.5555/3172077.3172382
- Chang Zhou , Jinze Bai , Junshuai Song , Xiaofei Liu , Zhengchao Zhao , Xiusi Chen , and Jun Gao . 2018 . ATRank: An attention-based user behavior modeling framework for recommendation . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, 4564–4571 . Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018. ATRank: An attention-based user behavior modeling framework for recommendation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, 4564–4571.
Cited by
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
A review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00815-1 · Cited by 2,678
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566
“[141] provided a survey of attention models in deep neural networks which concentrates on their application to natural language processing, while our work focuses on computer vision.”
BackgroundTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
Metadata
- Authors
- Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R.
- Venue
- ACM Transactions on Intelligent Systems and Technology
- Published
- 2021-10-22
- Volume
- 12
- Issue
- 5
- Pages
- 1-32
- ISSN
- 2157-6904 (ISSN-L)
- 2157-6912
- OpenAlex
- W2941531368
- Cited by
- 689
- References
- 112
- FWCI
- 42.59
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.