The DOI matches its Crossref record; 174 of the 174 references it cites were found in a registry or an index.
Attention in Natural Language Processing
Open accessAbstract
Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 1×
“The encoder is a bidirectional recurrent neural network (BiRNN) [133] that computes an annotation term hi for every Fig.”
BackgroundLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 1×
“This is especially interesting since it could overcome a well-known shortcoming of RNNs: their limited ability of modeling long-range dependences [140].”
BackgroundEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
(2011) · Cited by 6,993
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1174 · Cited by 4,712
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00781 · Cited by 3,151
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
Attention-based LSTM for Aspect-level Sentiment Classification
Wang, Y., Huang, M., zhu, X., & Zhao, L. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1058 · Cited by 2,450
Self-Normalizing Neural Networks
Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.02515 · Cited by 2,267
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,043
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
Jain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,975
A Structured Self-attentive Sentence Embedding
Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.03130 · Cited by 1,743 · cited 1×
“Other self-attentive approaches, such as [52] and [100], are characterized by the complete absence of any query term q, which results in simplified compatibility functions (see Section IV-B).”
BackgroundBuilding high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1×
“In the context of a multilayer neural architecture, it is fair to assume that the deepest levels will compute the most abstract features [146], [153].”
BackgroundIntroduction to Statistical Relational Learning
(2007) · doi:10.7551/mitpress/7432.001.0001 · Cited by 1,643
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Weight Uncertainty in Neural Networks
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1505.05424 · Cited by 1,342
Weight Uncertainty in Neural Network
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,335
A Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · International Conference on Machine Learning · Cited by 1,285
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1053 · Cited by 1,172
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,129
Interactive Attention Networks for Aspect-Level Sentiment Classification
Ma, D., Li, S., Zhang, X., & Wang, H. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/568 · Cited by 1,069
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 2×
“[142] is that the most relevant keys are the”
BackgroundAspect Level Sentiment Classification with Deep Memory Network
Tang, D., Qin, B., & Liu, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1021 · Cited by 1,041
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 3×
“Wiegreffe and Pinter [9] pushed the discussion further, providing experiments that demonstrate that creating an effective global adversarial attention model is much more difficult than creating a local one and that attention weights may contain information regarding feature importance.”
BackgroundNeural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821 · cited 1×
“What this survey does not offer is a comprehensive account of all the neural architectures for NLP (for an excellent overview, see [131]) or of all the neural architectures for NLP that uses an attention mechanism.”
DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., & Zhang, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11941 · Cited by 779
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Deep Biaffine Attention for Neural Dependency Parsing
Atmani, M., & Lafourcade, M. (2016) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.24 · Cited by 693
Modeling Coverage for Neural Machine Translation
Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1008 · Cited by 692
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690
Effective Approaches to Attention-based Neural Machine Translation
Luong, M.-T., Pham, H., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.04025 · Cited by 655
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM
Ma, Y., Peng, H., & Cambria, E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12048 · Cited by 592
Reasoning about Entailment with Neural Attention
Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiský, T., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06664 · Cited by 582
Multi-grained Attention Network for Aspect-Level Sentiment Classification
Fan, F., Feng, Y., & Zhao, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1380 · Cited by 520
Deep Biaffine Attention for Neural Dependency ParsingPreprint
Dozat, T., & Manning, C. D. (2016) · arXiv (Cornell University) · Cited by 515
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1151 · Cited by 482
Constituency Parsing with a Self-Attentive Encoder
Kitaev, N., & Klein, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1249 · Cited by 476
Attend and Diagnose: Clinical Time Series Analysis Using Attention Models
Song, H., Rajan, D., Thiagarajan, J., & Spanias, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11635 · Cited by 468
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 467
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 467
Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., & Hinton, G. (2014) · arXiv · doi:10.48550/arxiv.1412.7449 · Cited by 464
Learning to combine foveal glimpses with a third-order Boltzmann machine
Larochelle, H., & Hinton, G. E. (2010) · neural information processing systems · Cited by 436
From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
Martins, A. F. T., & Astudillo, R. F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02068 · Cited by 431
An Unsupervised Neural Attention Model for Aspect Extraction
He, R., Lee, W. S., Ng, H. T., & Dahlmeier, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1036 · Cited by 408
Learning Natural Language Inference with LSTM
Wang, S., & Jiang, J. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1170 · Cited by 408
Linguistically-Informed Self-Attention for Semantic Role Labeling
Strubell, E., Verga, P., Andor, D., Weiss, D., & McCallum, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1548 · Cited by 406
Using “annotator rationales” to improve machine learning for text categorization
Zaidan, O. F., Eisner, J. M., & Piatko, C. (2007) · Cited by 393
Kim, Y., Denton, C., Hoang, L., & Rush, A. M. (2017) · arXiv · doi:10.48550/arxiv.1702.00887 · Cited by 381 · cited 1×
“For example, structured attention networks [47] exploit neural conditional random fields to model the (conditional) attention distribution.”
BackgroundAdaptive Co-attention Network for Named Entity Recognition in Tweets
Zhang, Q., Fu, J., Liu, X., & Huang, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11962 · Cited by 381
Human‐centered artificial intelligence and machine learning
Riedl, M. O. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.117 · Cited by 371
Sequential Recommender System based on Hierarchical Attention Networks
Ying, H., Zhuang, F., Zhang, F., Liu, Y., Xu, G., Xie, X., Xiong, H., & Wu, J. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/546 · Cited by 367
Gated-Attention Readers for Text Comprehension
Dhingra, B., Liu, H., Yang, Z., Cohen, W., & Salakhutdinov, R. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1168 · Cited by 366
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 362
Hao, Y., Zhang, Y., Liu, K., He, S., Liu, Z., Wu, H., & Zhao, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1021 · Cited by 359
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J. (2024) · TIB Data Manager · doi:10.57702/uhdmfu1r · Cited by 357
Deep Semantic Role Labeling With Self-Attention
Tan, Z., Wang, M., Xie, J., Chen, Y., & Shi, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11928 · Cited by 306
dos Santos, C. N., Tan, M., Xiang, B., & Zhou, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03609 · Cited by 300
Does String-Based Neural MT Learn Source Syntax?
Shi, X., Padhi, I., & Knight, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1159 · Cited by 300
Text Understanding with the Attention Sum Reader Network
Kadlec, R., Schmid, M., Bajgar, O., & Kleindienst, J. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1086 · Cited by 295
Aspect Term Extraction with History Attention and Selective Transformation
Li, X., Bing, L., Li, P., Lam, W., & Yang, Z. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/583 · Cited by 284
Context-Aware Neural Machine Translation Learns Anaphora Resolution
Voita, E., Serdyukov, P., Sennrich, R., & Titov, I. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1117 · Cited by 279
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures
Tang, G., Müller, M., Rios, A., & Sennrich, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1458 · Cited by 273
ReasoNet: Learning to Stop Reading in Machine Comprehension
Shen, Y., Huang, P.-S., Gao, J., & Chen, W. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098177 · Cited by 262
Rationale-Augmented Convolutional Neural Networks for Text Classification
Zhang, Y., Marshall, I., & Wallace, B. C. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1076 · Cited by 249
Attention Interpretability Across NLP TasksPreprint
Vashishth, S., Upadhyay, S., Tomar, G. S., & Faruqui, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11218 · Cited by 243
Temporal Attention-Augmented Bilinear Network for Financial Time-Series Data Analysis
Tran, D. T., Iosifidis, A., Kanniainen, J., & Gabbouj, M. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2869225 · Cited by 238
Cross-topic Argument Mining from Heterogeneous Sources
Stab, C., Miller, T., Schiller, B., Rai, P., & Gurevych, I. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1402 · Cited by 237
Effective Attention Modeling for Aspect-Level Sentiment Classification
He, R., Lee, W. S., Ng, H. T., & Dahlmeier, D. (2018) · International Conference on Computational Linguistics · Cited by 234
Attention Models in Graphs: A Survey
Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., & Koh, E. (2019) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3363574 · Cited by 231
Attention-Based Transactional Context Embedding for Next-Item Recommendation
Wang, S., Hu, L., Cao, L., Huang, X., Lian, D., & Liu, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11851 · Cited by 215
Gated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11832 · Cited by 200
Modeling Localness for Self-Attention Networks
Yang, B., Tu, Z., Wong, D. F., Meng, F., Chao, L. S., & Zhang, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1475 · Cited by 196
Inner Attention based Recurrent Neural Networks for Answer Selection
Wang, B., Liu, K., & Zhao, J. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1122 · Cited by 194
Multi-Head Attention with Disagreement Regularization
Li, J., Tu, Z., Yang, B., Lyu, M. R., & Zhang, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1317 · Cited by 190
A Question Answering Approach for Emotion Cause Extraction
Gui, L., Hu, J., He, Y., Xu, R., Qin, L., & Du, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1167 · Cited by 178
Attention-over-Attention Neural Networks for Reading Comprehension
Cui, Y., Chen, Z., Wei, S., Wang, S., Liu, T., & Hu, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1055 · Cited by 171
Attention, Learn to Solve Routing Problems!
Kool, W., van Hoof, H., & Welling, M. (2018) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1803.08475 · Cited by 171
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
Shen, T., Zhou, T., Long, G., Jiang, J. B., Wang, S., & Zhang, C. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/604 · Cited by 166
Training Deeper Neural Machine Translation Models with Transparent Attention
Bapna, A., Chen, M., Fırat, O., Cao, Y. J., & Wu, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1338 · Cited by 165
Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding
Wang, P., Han, J., Li, C., & Pan, R. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017152 · Cited by 165
Hierarchical Attention Transfer Network for Cross-Domain Sentiment Classification
Li, Z., Wei, Y., Zhang, Y., & Yang, Q. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12055 · Cited by 160
Self-Attentional Acoustic Models
Sperber, M., Niehues, J., Neubig, G., Stüker, S., & Waibel, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1910 · Cited by 159
Dynamic Meta-Embeddings for Improved Sentence Representations
Kiela, D., Wang, C., & Cho, K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1176 · Cited by 157
Aspect-level Sentiment Classification with HEAT (HiErarchical ATtention) Network
Cheng, J., Zhao, S., Zhang, J., King, I., Zhang, X., & Wang, H. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133037 · Cited by 153
Deeper Attention to Abusive User Content Moderation
Pavlopoulos, J., Malakasiotis, P., & Androutsopoulos, I. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1117 · Cited by 151
Neural Machine Translation with Supervised Attention
Liu, L., Utiyama, M., Finch, A., & Sumita, E. (2016) · arXiv · doi:10.48550/arxiv.1609.04186 · Cited by 146
Coverage Embedding Models for Neural Machine Translation
Mi, H., Sankaran, B., Wang, Z., & Ittycheriah, A. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1096 · Cited by 144
Deriving Machine Attention from Human Rationales
Bao, Y., Chang, S., Yu, M., & Barzilay, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1216 · Cited by 138
Please Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138
Tay, Y., Luu, A. T., & Hui, S. C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1185 · Cited by 134
Supervised Attentions for Neural Machine Translation
Mi, H., Wang, Z., & Ittycheriah, A. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1249 · Cited by 134
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Li, H., Zhu, J., Liu, T., Zhang, J., & Zong, C. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/577 · Cited by 131
Kang, G., Zheng, L., Yan, Y., & Yang, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_25 · Cited by 127
Augmenting Neural Networks with First-order Logic
Li, T., & Srikumar, V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1028 · Cited by 122
Incorporating Glosses into Neural Word Sense Disambiguation
Luo, F., Liu, T., Xia, Q., Chang, B., & Sui, Z. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1230 · Cited by 120
A Co-Attention Neural Network Model for Emotion Cause Analysis with Emotional Context Awareness
Li, X., Song, K., Shi, F., Wang, D., & Zhang, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1506 · Cited by 114
Iterative Alternating Neural Attention for Machine ReadingPreprint
Sordoni, A., Bachman, P., Trischler, A., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02245 · Cited by 112
Habernal, I., & Gurevych, I. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1129 · Cited by 110
Exploring Human-Like Attention Supervision in Visual Question Answering
Qiao, T., Dong, J., & Xu, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12272 · Cited by 100
Syntax-Directed Attention for Neural Machine Translation
Chen, K., Wang, R., Utiyama, M., Sumita, E., & Zhao, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11910 · Cited by 99
Consensus Attention-based Neural Networks for Chinese Reading ComprehensionPreprint
Cui, Y., Liu, T., Chen, Z., Wang, S., & Hu, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02250 · Cited by 94
Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction
Du, J., Han, J., Way, A., & Wan, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1245 · Cited by 86
Gated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1706.07230 · Cited by 82
Yu, J., Marujo, L., Jiang, J., Karuturi, P., & Brendel, W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1137 · Cited by 81
Interactive Visualization and Manipulation of Attention-based Neural Machine Translation
Lee, J., Shin, J.-H., & Kim, J. (2017) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d17-2021 · Cited by 77
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Song, J., Zeng, P., Gao, L., & Shen, H. T. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/126 · Cited by 77
Zhang, R., dos Santos, C. N., Yasunaga, M., Xiang, B., & Radev, D. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2017 · Cited by 77
Attention-Guided Answer Distillation for Machine Reading Comprehension
Hu, M., Peng, Y., Wei, F., Huang, Z., Li, D., Yang, N., & Zhou, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1232 · Cited by 77
How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures
Domhan, T. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1167 · Cited by 74
Enhancing Sentence Embedding with Generalized PoolingPreprint
Chen, Q., Ling, Z.-H., & Zhu, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09828 · Cited by 74
Hierarchical Attention Flow for Multiple-Choice Reading Comprehension
Zhu, H., Wei, F., Qin, B., & Liu, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12040 · Cited by 73
Learning What Data to LearnPreprint
Yang, F., Tian, F., Qin, T., Bian, J., & Liu, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.08635 · Cited by 72
Interactive Attention for Neural Machine TranslationPreprint
Meng, F., Lu, Z., Li, H., & Liu, Q. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.05011 · Cited by 71
Importance of Self-Attention for Sentiment Analysis
Letarte, G., Paradis, F., Giguère, P., & Laviolette, F. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5429 · Cited by 70
Convolution-Based Neural Attention With Applications to Sentiment Classification
Du, J., Gui, L., He, Y., Xu, R., & Wang, X. (2019) · IEEE Access · doi:10.1109/access.2019.2900335 · Cited by 70
Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs
Shen, Y., Deng, Y., Yang, M., Li, Y., Du, N., Fan, W., & Lei, K. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210081 · Cited by 67
Word n-gram attention models for sentence similarity and inference
López-Gazpio, I., Maritxalar, M., Lapata, M., & Agirre, E. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.04.054 · Cited by 65
Poggi, M., & Mattoccia, S. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.61 · Cited by 62
Attention Models in Graphs: A Survey
Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., & Koh, E. (2018) · arXiv · doi:10.48550/arxiv.1807.07984 · Cited by 59
Sparse and Constrained Attention for Neural Machine Translation
Malaviya, C., Ferreira, P., & Martins, A. F. T. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2059 · Cited by 58
Enhanced Aspect Level Sentiment Classification with Auxiliary Memory.
Zhu, P., & Qian, T. (2018) · International Conference on Computational Linguistics · Cited by 57
Improving Attention Model Based on Cognition Grounded Data for Sentiment Analysis
Long, Y., Xiang, R., Lu, Q., Huang, C., & Li, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2903056 · Cited by 55
Deep Metric Learning by Online Soft Mining and Class-Aware Attention
Wang, X., Yang, H., Kodirov, E., Hu, G., & Robertson, N. M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015361 · Cited by 55
Uncertainty-Aware Attention for Reliable Interpretation and Prediction
Huh, J. (2019) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 52
Multi-focus Attention Network for Efficient Deep Reinforcement LearningPreprint
Choi, J., Lee, B., & Zhang, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.04603 · Cited by 50
Interpretable Emoji Prediction via Label-Wise Attention LSTMs
Barbieri, F., Espinosa-Anke, L., Camacho-Collados, J., Schockaert, S., & Saggion, H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1508 · Cited by 48
Hermitian Co-Attention Networks for Text Matching in Asymmetrical Domains
Tay, Y., Luu, A. T., & Hui, S. C. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/615 · Cited by 46
Hierarchical User and Item Representation with Three-Tier Attention for Recommendation
Wu, C., Wu, F., Liu, J., & Huang, Y. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1180 · Cited by 46
Multi-Input Attention for Unsupervised OCR Correction
Dong, R., & Smith, D. A. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1220 · Cited by 44
Completely Heterogeneous Transfer Learning with Attention - What And What Not To Transfer
Moon, S., & Carbonell, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/349 · Cited by 43
Liu, S.-S., Li, T., Li, Z.-M., Srikumar, V., Pascucci, V., & Bremer, P. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2007 · Cited by 39
A Question-Focused Multi-Factor Attention Network for Question Answering
Kundu, S., & Ng, H. T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12065 · Cited by 33
Mention and Entity Description Co-Attention for Entity Disambiguation
Nie, F., Cao, Y., Wang, J., Lin, C.-Y., & Pan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12043 · Cited by 32
Learning What's Easy: Fully Differentiable Neural Easy-First Taggers
Martins, A. F. T., & Kreutzer, J. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1036 · Cited by 28
Word Attention for Sequence to Sequence Text Understanding
Wu, L., Tian, F., Li, Z., Lai, J., & Liu, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11971 · Cited by 28
A Genre-Aware Attention Model to Improve the Likability Prediction of Books
Maharjan, S., Montes, M., Gonzalez, F. A., & Solorio, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1375 · Cited by 27
Lauscher, A., Glavaš, G., Ponzetto, S. P., & Eckert, K. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1370 · Cited by 27
Attention-via-Attention Neural Machine Translation
Zhao, S., & Zhang, Z. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11254 · Cited by 26
Neural Cross-Lingual Coreference Resolution And Its Application To Entity Linking
Kundu, G., Sil, A., Florian, R., & Hamza, W. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2063 · Cited by 26
Lin, J., Sun, X., Ren, X., Li, M., & Su, Q. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1331 · Cited by 24
Attentive Gated Lexicon Reader with Contrastive Contextual Co-Attention for Sentiment Classification
Tay, Y., Luu, A. T., Hui, S. C., & Su, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1381 · Cited by 23
An Unsupervised Model With Attention Autoencoders for Question Retrieval
Zhang, M., & Wu, Y. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11926 · Cited by 22
WECA: A WordNet-Encoded Collocation-Attention Network for Homographic Pun Recognition
Diao, Y., Lin, H., Wu, D., Yang, L., Xu, K., Yang, Z., Wang, J., Zhang, S., Xu, B., & Zhang, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1272 · Cited by 16
Deterministic Attention for Sequence-to-Sequence Constituent Parsing
Ma, C., Liu, L., Tamura, A., Zhao, T., & Sumita, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10967 · Cited by 15
Decoding-History-Based Adaptive Control of Attention for Neural Machine TranslationPreprint
Lin, J., Ma, S., Su, Q., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01812 · Cited by 14
Hybrid Neural Attention for Agreement/Disagreement Inference in Online Debates
Chen, D., Du, J., Bing, L., & Xu, R. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1069 · Cited by 14
Improving Sequence-to-Sequence Constituency Parsing
Liu, L., Zhu, M., & Shi, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11917 · Cited by 13
Spliethöver, M., Klaff, J., & Heuer, H. (2019) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/w19-4509 · Cited by 12
A Question-Focused Multi-Factor Attention Network for Question AnsweringPreprint
Kundu, S., & Ng, H. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.08290 · Cited by 10
Supervised Domain Enablement Attention for Personalized Domain Classification
Kim, J.-K., & Kim, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1106 · Cited by 8
Attend to Knowledge: Memory-Enhanced Attention Network for Image Captioning
Chen, H., Ding, G., Lin, Z., Guo, Y., & Han, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00563-4_16 · Cited by 6
Specificity-Driven Cascading Approach for Unsupervised Sentiment Modification
Yang, P., Lin, J., Xu, J., Xie, J., Su, Q., & Sun, X. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1553 · Cited by 3
Cited by
An Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 689
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., & He, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 549
Attention mechanism in neural networks: where it comes and where it goes
Soydaner, D. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07366-3 · Cited by 414
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Classification of Remote Sensing Images Using EfficientNet-B3 CNN Model With Attention
Alhichri, H., Alswayed, A. S., Bazi, Y., Ammour, N., & Alajlan, N. A. (2021) · IEEE Access · doi:10.1109/access.2021.3051085 · Cited by 357
“The attention mechanism (the idea of focusing on spe-cific parts of the input) has been applied in deep learning for speech recognition [30], Natural Language processing [31], multimodal reasoning and matching [32], object detection [33], and image recognition [34]–[36].”
Background
Metadata
- Authors
- Galassi, A., Lippi, M., & Torroni, P.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2021-10
- Volume
- 32
- Issue
- 10
- Pages
- 4291-4308
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W3031696893
- Cited by
- 635
- References
- 174
- FWCI
- 31.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.