The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Attention mechanism in neural networks: where it comes and where it goes
Open accessAbstract
A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 1×
“The first part, encoder, is a bidirectional RNN (BiRNN) [47] that takes word vectors as input.”
BackgroundNeural Mechanisms of Selective Visual Attention
Desimone, R., & Duncan, J. (1995) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.18.030195.001205 · Cited by 8,388
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 3×
“Here, score is referred as a content-based function, and three different alternatives are considered [51].”
BackgroundThe attention system of the human brain
Posner, M. I., & Petersen, S. E. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.000325 · Cited by 8,070
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“This architecture is an extension of the neural network [49] that encodes an image into a compact representation, followed by an RNN that generates a corresponding sentence.”
BackgroundProceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
Fukushima, K. (1980) · Biological Cybernetics · doi:10.1007/bf00344251 · Cited by 5,759
A fast algorithm for particle simulations
Greengard, L., & Rokhlin, V (1987) · Journal of Computational Physics · doi:10.1016/0021-9991(87)90140-9 · Cited by 4,868
Some Mathematical Notes on Three-Mode Factor Analysis
Tucker, L. R. (1966) · Psychometrika · doi:10.1007/bf02289464 · Cited by 3,492
The attention system of the human brain: 20 years after
Petersen, S. E., & Posner, M. I. (2012) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062111-150525 · Cited by 3,439
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · cited 1×
“In order to obtain better representations, SpanBERT is proposed as a pre-training method [151].”
MethodSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10529 · Cited by 1,827
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Bundesen, C. (1990) · Psychological Review · doi:10.1037//0033-295x.97.4.523 · Cited by 1,704
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435
The role of visual attention in saccadic eye movements
Hoffman, J. E., & Subramaniam, B. (1995) · Perception & Psychophysics · doi:10.3758/bf03206794 · Cited by 1,432
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4828 · Cited by 1,373
A Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296
Attention Augmented Convolutional Networks
Bello, I., Zoph, B., Le, Q., Vaswani, A., & Shlens, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00338 · Cited by 1,205
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1053 · Cited by 1,172
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146
Modeling visual attention via selective tuning
Tsotsos, J. K., Culhane, S. M., Kei Wai, W. Y., Lai, Y., Davis, N., & Nuflo, F. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(95)00025-9 · Cited by 1,143
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
Yin, W., Schütze, H., Xiang, B., & Zhou, B. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00097 · Cited by 924
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1470 · Cited by 904
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_28 · Cited by 727
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“Additionally, various approaches have been presented in Hierarchical Visual Transformer [183], Long-Short Transformer (Transformer-LS) [184], Perceiver [185], and Performer [186].”
MethodAn Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 689
A coherent computational approach to model bottom-up visual attention
Le Meur, O., Le Callet, P., Barba, D., & Thoreau, D. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.86 · Cited by 642
Attention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 635
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1441 · Cited by 631
Scanpaths in saccadic eye movements while viewing and recognizing patterns
Noton, D., & Stark, L. (1971) · Vision Research · doi:10.1016/0042-6989(71)90213-6 · Cited by 539
Efficient Content-Based Sparse Attention with Routing Transformers
Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509
Decompositions of a Higher-Order Tensor in Block Terms—Part II: Definitions and Uniqueness
De Lathauwer, L. (2008) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/070690729 · Cited by 479
Constituency Parsing with a Self-Attentive Encoder
Kitaev, N., & Klein, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1249 · Cited by 476
Orienting of attention and eye movements
Sheliga, B. M., Riggio, L., & Rizzolatti, G. (1994) · Experimental Brain Research · doi:10.1007/bf00233988 · Cited by 442
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442
Context-free attentional operators: The generalized symmetry transform
Reisfeld, D., Wolfson, H., & Yeshurun, Y. (1995) · International Journal of Computer Vision · doi:10.1007/bf01418978 · Cited by 414
BERT for Coreference Resolution: Baselines and Analysis
Joshi, M., Levy, O., Zettlemoyer, L., & Weld, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1588 · Cited by 312
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, E., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1032 · Cited by 279
Spatial attention and eye movements
Sheliga, B. M., Riggio, L., & Rizzolatti, G. (1995) · Experimental Brain Research · doi:10.1007/bf00240962 · Cited by 237
Scalable Vision Transformers with Hierarchical Pooling
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00043 · Cited by 234
Attention Models in Graphs: A Survey
Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., & Koh, E. (2019) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3363574 · Cited by 231
A model of attention-guided visual perception and recognition
Rybak, I. A., Gusakova, V. I., Golovan, A. V., Podladchikova, L. N., & Shevtsova, N. A. (1998) · Vision Research · doi:10.1016/s0042-6989(98)00020-0 · Cited by 220
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1223 · Cited by 198
Modeling Localness for Self-Attention Networks
Yang, B., Tu, Z., Wong, D. F., Meng, F., Chao, L. S., & Zhang, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1475 · Cited by 196
Neural network model for selective attention in visual pattern recognition and associative recall
Fukushima, K. (1987) · Applied Optics · doi:10.1364/ao.26.004985 · Cited by 176
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
Shen, T., Zhou, T., Long, G., Jiang, J. B., Wang, S., & Chengqi Zhang, B. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/604 · Cited by 166
A Time-Restricted Self-Attention Layer for ASR
Povey, D., Hadian, H., Ghahremani, P., Li, K., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462497 · Cited by 162
Self-Attentional Acoustic Models
Sperber, M., Niehues, J., Neubig, G., Stüker, S., & Waibel, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1910 · Cited by 159
Selective Attention for Context-aware Neural Machine Translation
Maruf, S., Martins, A. F. T., & Haffari, G. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1313 · Cited by 159
Salah, A. A., Alpaydın, E., & Akarun, L. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.990146 · Cited by 140
Cross-Target Stance Classification with Self-Attention Networks
Xu, C., Paris, C. L., Nepal, ., & Sparks, R. S. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2123 · Cited by 136
Modeling Recurrence for Transformer
Hao, J., Wang, X., Yang, B., Wang, L., Zhang, J., & Tu, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1122 · Cited by 96
Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction
Du, J., Han, J., Way, A., & Wan, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1245 · Cited by 86
Scene analysis with saccadic eye movements: top-down and bottom-up modeling
Schill, K., Umkehrer, E., Beinlich, S., Krieger, G., & Zetzsche, C. (2001) · Journal of Electronic Imaging · doi:10.1117/1.1329627 · Cited by 85
Deep Convolutional Neural Networks with Layer-Wise Context Expansion and Attention
Yu, D., Xiong, W., Droppo, J., Stolcke, A., Ye, G., Li, J., & Zweig, G. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-251 · Cited by 82
LEARNING TO GENERATE ARTIFICIAL FOVEA TRAJECTORIES FOR TARGET DETECTION
Schmidhuber, J., & Huber, R. (1991) · International Journal of Neural Systems · doi:10.1142/s012906579100011x · Cited by 69
Q-learning of sequential attention for visual object recognition from informative local descriptors
Paletta, L., Fritz, G., & Seifert, C. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102433 · Cited by 60
Recognition and segmentation of connected characters with selective attention
Fukushima, K., & Imagawa, T. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80071-1 · Cited by 48
Spatio-Temporal Attention Models for Grounded Video Captioning
Zanfir, M., Marinoiu, E., & Sminchisescu, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54190-7_7 · Cited by 42
SCAN: A Scalable Model of Attentional Selection
Postma, E. O., van den Herik, H. J., & Hudson, P. T. W. (1997) · Neural Networks · doi:10.1016/s0893-6080(97)00034-8 · Cited by 41
A prototype for data-driven visual attention
Culhane, S. M., & Tsotsos, J. K. (2003) · IAPR International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.1992.201502 · Cited by 21
Miau, F., & Itti, L. (2005) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2001.1019059 · Cited by 21
An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation
Shu, R., & Nakayama, H. (2017) · doi:10.18653/v1/w17-3201 · Cited by 8
Object Recognition Based on Human Saccadic Behaviour
Keller, J. G., Rogers, S. K., Kabrisky, M., & Oxley, M. E. (1999) · Pattern Analysis and Applications · doi:10.1007/s100440050033 · Cited by 7
Deep Supervised Hashing by Classification for Image Retrieval
Luo, X., Guo, Y., Ma, Z., Zhong, H., Li, T., Ju, W., Chen, C., & Deng, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92273-3_1 · Cited by 5
Research works printed but not linked: 88
- Goodfellow I, Bengio Y, Courville A (2016) Deep learning. The MIT Press
- Noton D, Stark L (1971) Eye movements and visual perception. Sci Am 224(6):34
- Alpaydın E (1995) Selective attention for handwritten digit recognition. Adv Neural Inf Process Syst 8:771–777
- Ahmad S (1991) VISIT: a neural model of covert visual attention. Adv Neural Inf Process Syst 4:420–427
- Brown T et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
- Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30:5998–6008
- Milanese R et al (1994) Integration of bottom-up and top-down cues for visual attention using non-linear relaxation. In: IEEE computer society conference on computer vision and pattern recoginition, Seattle, WA, pp 781–785 · doi:10.1109/cvpr.1994.323898
- Zhang W et al (2006) A computational model of eye movements during object class detection. Adv Neural Inf Process Syst 19:1609–1616
- Walther D et al (2002) Attentional selection for object recognition—A gentle way. In: Bu¨lthoff HH, Wallraven C, Lee SW, Poggio TA (eds) International workshop on biologically motivated computer vision. Springer, Berlin, Heidelberg, pp 472–479
- Gould S et al (2007) Peripheral-foveal vision for real-time object recognition and tracking in video. In: International joint conference on artificial intelligence (IJCAI) pp 2115–2121
- Larochelle H, Hinton G (2010) Learning to combine foveal glimpses with a third-order Boltzmann machine. Adv Neural Inf Process Syst 23:1243–1251
- Mnih V et al (2014) Recurrent models of visual attention. Adv Neural Inf Process Syst 27:2204–2212
- Stollenga M et al (2014) Deep networks with internal selective attention through feedback connections. Adv Neural Inf Process Syst 27:3545–3553
- Sutskever I, Vinyals O, Le Q (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27:3104–3112
- Xu K et al (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057
- Lu J et al (2016) Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems. 29
- Sukhbaatar S et al (2015) End-to-end memory networks. Adv Neural Inf Process Syst 28:2440–2448
- Chorowski J et al (2015) Attention-based models for speech recognition. Adv Neural Inf Process Syst 28:577–585
- Chen K et al (2015) ABC-CNN: An attention based convolutional neural network for visual question answering, arXiv preprint arXiv:1511.05960 · arXiv
- Sorokin I et al (2015) Deep attention recurrent Q-network, arXiv preprint arXiv:1512.01693 · arXiv
- Ba J et al (2015) Learning wake-sleep recurrent attention models. Adv Neural Inf Process Syst 28:2593–2601
- Gregor K et al (2015) DRAW: a recurrent neural network for image generation. In: International conference on machine learning, pp 1462–1471
- Mansimov E et al (2016) Generating images from captions with attention. In: International conference on learning representations
- Reed S et al (2016) Learning what and where to draw. Adv Neural Inf Process Syst 29:217–225
- Kerg G et al (2020) Untangling tradeoffs between recurrence and self-attention in neural networks. Advances in Neural Information Processing Systems, 33
- Lin Z et al (2017) A structured self-attentive sentence embedding. In: International conference on learning representations
- Paulus R, Xiong C, Socher R (2018) A deep reinforced model for abstractive summarization. In: International conference on learning representations
- Vyas A et al (2020) Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33
- Kaiser L et al (2017) One model to learn them all. arXiv preprint arXiv:1706.05137 · arXiv
- Ramachandran P et al (2019) Stand-alone self-attention in vision models. Adv Neural Inf Process Syst 32:68–80
- Goodfellow I et al (2014) Generative adversarial networks. Adv Neural Inf Process Syst 27:2672–2680
- Zhang H et al (2019) Self-attention generative adversarial networks. In: International conference on machine learning, pp 7354–7363
- Yu A et al (2018) QANet: combining local convolution with global self-attention for reading comprehension. In: International conference on learning representations
- Zhang J et al (2018) Ga AN: gated attention networks for learning on large and spatiotemporal graphs. In: Conference on uncertainty in artificial intelligence
- Al-Rfou R et al (2019) Character-level language modeling with deeper self-attention. AAAI Conf Artif Intell 33:3159
- Li X et al (2020) SAC: accelerating and structuring self-attention via sparse adaptive connection. Advances in Neural Information Processing Systems, 33
- Yang B et al (2019) Context-aware self-attention networks. AAAI Conf Artif Intell 33:387
- Shen T et al (2018) Di SAN: directional self-attention network for RNN/CNN-free language understanding. In: AAAI Conference on artificial intelligence, pp 5446–5455
- Shen T et al (2018) Bi-directional block self-attention for fast and memory-efficient sequence modeling. In: International conference on learning representations
- Jernite Y et al (2017) Variable computation in recurrent neural networks. In: International conference on learning representations
- Shiv V, Quirk C (2019) Novel positional encodings to enable tree-based transformers. Adv Neural Inf Process Syst 32:12081–12091
- Hoshen Y (2017) VAIN: Attentional Multi-agent predictive modeling, Advances in Neural Information Processing Systems, 30, Long Beach, CA
- Hu S et al (2021) UPDeT: universal multi-agent reinforcement learning via policy decoupling with transformers. In: International conference on learning representations
- Wu S et al (2020) Adversarial sparse transformer for time series forecasting. Advances in Neural Information Processing Systems, 33
- Doersch C, Gupta A, Zisserman A (2020) Cross Transformers: spatially-aware few-shot transfer. Adv Neural Inf Process Syst 33:21981–21993
- Liu X et al (2020) Learning to encode position for transformer with continuous dynamical model. In: International conference on machine learning, pp 6327–6335
- Hudson D, Zitnick L (2021) Generative adversial transformers. In: International conference on machine learning, pp 4487–4499
- Radford A et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
- Dehghani M et al (2019) Universal transformers. In: International conference on learning representations
- Ma X et al (2019) A Tensorized Transformer for Language Modeling. Adv Neural Inf Process Syst 32:2232–2242
- Devlin J et al (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proc of NAACL-HLT 2019:4171–4186
- Taylor W (1953) Cloze procedure: a new tool for measuring readability. J Bull 30(4):415
- Wang W et al (2020) MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33
- McCarley J, Chakravarti R, Sil A (2020) Structured pruning of a BERT-based question answering model, arXiv preprint arXiv:1910.06360 · arXiv
- Gong L et al (2019) Efficient training of BERT by progressively stacking. In: International conference on machine learning, pp 2337–2346
- Lan Z et al (2020) ALBERT: a lite BERT for self-supervised learning of language representations. In: International conference on learning representations
- Liu Y et al (2019) Ro BERTa: a robustly optimized BERT pretraining approach, arXiv preprint arXiv:1907.11692 · arXiv
- He P et al (2021) DeBERTa: decoding-enhanced BERT with disentangled attention. In: International conference on learning representations
- Sanh V et al (2019) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In: The 5th workshop on energy efficient machine learning and cognitive computing—NeurIPS
- Wang W et al (2020) Struct BERT: incorporating language structures into pre-training for deep language understanding. In: International conference on learning representations
- Shen S et al (2020) Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. AAAI Conf Artif Intell 34:8815
- Prakash P et al (2021) Rare BERT: transformer architecture for rare disease patient identification using administrative claims. AAAI Conf Artif Intell 35:453
- Wu Z et al (2020) Lite transformer with long-short range attention. In: International conference on learning representations UK
- Mehta S et al (2021) DeLighT: deep and light-weight transformer. In: International conference on learning representations
- Tay Y et al (2021) HyperGrid transformers: towards a single model for multiple tasks. In: International conference on learning representations
- Yun S et al (2018) Graph transformer networks. In: International conference on learning representations
- Rong Y et al (2020) Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33
- J. Yang, et al.,(2021) GraphFormers:GNN-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34
- ZhaoC et al (2020) Transformer-XH: multi-evidence reasoning with extra hop attention. In: International conference on learning representations
- You R et al (2019) AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in Neural Information Processing Systems, 32
- Fan X et al (2020) Bayesian attention modules. Advances in Neural Information Processing Systems, 33
- Brunner G et al (2020) On identifiability in transformers. In: International conference on learning representations
- Dosovitskiy A et al (2021) An image is worth $$16 \times 16$$ words: transformers for image recognition at scale. In: International conference on learning representations
- Nguyen T et al (2021) FMMformer: efficient and flexible transformer via decomposed near-field and far-field attention. Advances in Neural Information Processing Systems, 34
- Lee J et al (2019) Set transformer: a framework for attention-based permutation-invariant neural networks. In: International conference on machine learning pp 3744–3753
- Child R et al (2019) Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509 · arXiv
- Peng H et al (2021) Random feature attention. In: International conference on learning representations UK
- Chen Y et al (2021) Skyformer: remodel self-attention with Gaussian kernel and Nyström method. Advances in Neural Information Processing Systems, 34
- Zaheer M et al (2020) Big bird: transformers for longer sequences. Advances in Neural Information Processing Systems, 33
- Huang CZ et al (2019) Music transformer: generating music with long-term structure. In: International conference on learning representations UK
- Lu J et al (2021) SOFT: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34
- Zhu C et al (2021) Long-short transformer: efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34
- Jaegle A et al (2021) Perceiver: general perception with iterative attention. In: International conference on machine learning, pp 4651–4664
- El-Nouby A et al. (2021) XCiT: cross-covariance image transformers. Advances in neural information processing systems, 34
- Yu Q et al (2021) Glance-and-gaze vision transformer. Advances in Neural Information Processing Systems, 34
- Zeng Z et al (2021) You only sample (almost) once: linear cost self-attention via Bernoulli sampling. In: International conference on machine learning, pp 12,321–12,332
- Shen Z et al (2021) Efficient attention: attention with linear complexities. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 3531–3539
- Ma X et al (2021) Luna: linear unified nested attention. Advances in Neural Information Processing Systems, 34
Cited by
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 487
Mishra, L. N., & Senapati, B. (2025) · IEEE Access · doi:10.1109/access.2025.3552592 · Cited by 35
Moveh, S., Merchán-Cruz, E. A., Abuhussain, M., Dodo, Y. A., Alhumaid, S., & Alhamami, A. H. (2025) · Energies · doi:10.3390/en18061468 · Cited by 27
Mahali, M. I., Leu, J.-S., Avian, C., Darmawan, J. T., Faisal, M., Putro, N. A. S., & Prakosa, S. W. (2025) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2025.106599 · Cited by 1
Metadata
- Authors
- Soydaner, D.
- Venue
- Neural Computing and Applications
- Published
- 2022-05-25
- Volume
- 34
- Issue
- 16
- Pages
- 13371-13385
- ISSN
- 0941-0643 (ISSN-L)
- 1433-3058
- OpenAlex
- W4225141805
- Cited by
- 414
- References
- 83
- FWCI
- 24.28
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.