The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Attention mechanism in neural networks: where it comes and where it goes
Open accessAbstract
A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Neural Mechanisms of Selective Visual Attention
Desimone, R., & Duncan, J. (1995) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.18.030195.001205 · Cited by 8,388
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 3×
“Here, score is referred as a content-based function, and three different alternatives are considered [51].”
BackgroundThe attention system of the human brain
Posner, M. I., & Petersen, S. E. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.000325 · Cited by 8,070
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Research works printed but not linked: 88
- Goodfellow I, Bengio Y, Courville A (2016) Deep learning. The MIT Press
- Noton D, Stark L (1971) Eye movements and visual perception. Sci Am 224(6):34
- Alpaydın E (1995) Selective attention for handwritten digit recognition. Adv Neural Inf Process Syst 8:771–777
- Ahmad S (1991) VISIT: a neural model of covert visual attention. Adv Neural Inf Process Syst 4:420–427
- Brown T et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
- Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30:5998–6008
- Milanese R et al (1994) Integration of bottom-up and top-down cues for visual attention using non-linear relaxation. In: IEEE computer society conference on computer vision and pattern recoginition, Seattle, WA, pp 781–785 · doi:10.1109/cvpr.1994.323898
- Zhang W et al (2006) A computational model of eye movements during object class detection. Adv Neural Inf Process Syst 19:1609–1616
- Walther D et al (2002) Attentional selection for object recognition—A gentle way. In: Bu¨lthoff HH, Wallraven C, Lee SW, Poggio TA (eds) International workshop on biologically motivated computer vision. Springer, Berlin, Heidelberg, pp 472–479
- Gould S et al (2007) Peripheral-foveal vision for real-time object recognition and tracking in video. In: International joint conference on artificial intelligence (IJCAI) pp 2115–2121
- Larochelle H, Hinton G (2010) Learning to combine foveal glimpses with a third-order Boltzmann machine. Adv Neural Inf Process Syst 23:1243–1251
- Mnih V et al (2014) Recurrent models of visual attention. Adv Neural Inf Process Syst 27:2204–2212
- Stollenga M et al (2014) Deep networks with internal selective attention through feedback connections. Adv Neural Inf Process Syst 27:3545–3553
- Sutskever I, Vinyals O, Le Q (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27:3104–3112
- Xu K et al (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057
- Lu J et al (2016) Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems. 29
- Sukhbaatar S et al (2015) End-to-end memory networks. Adv Neural Inf Process Syst 28:2440–2448
- Chorowski J et al (2015) Attention-based models for speech recognition. Adv Neural Inf Process Syst 28:577–585
- Chen K et al (2015) ABC-CNN: An attention based convolutional neural network for visual question answering, arXiv preprint arXiv:1511.05960 · arXiv
- Sorokin I et al (2015) Deep attention recurrent Q-network, arXiv preprint arXiv:1512.01693 · arXiv
- Ba J et al (2015) Learning wake-sleep recurrent attention models. Adv Neural Inf Process Syst 28:2593–2601
- Gregor K et al (2015) DRAW: a recurrent neural network for image generation. In: International conference on machine learning, pp 1462–1471
- Mansimov E et al (2016) Generating images from captions with attention. In: International conference on learning representations
- Reed S et al (2016) Learning what and where to draw. Adv Neural Inf Process Syst 29:217–225
- Kerg G et al (2020) Untangling tradeoffs between recurrence and self-attention in neural networks. Advances in Neural Information Processing Systems, 33
- Lin Z et al (2017) A structured self-attentive sentence embedding. In: International conference on learning representations
- Paulus R, Xiong C, Socher R (2018) A deep reinforced model for abstractive summarization. In: International conference on learning representations
- Vyas A et al (2020) Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33
- Kaiser L et al (2017) One model to learn them all. arXiv preprint arXiv:1706.05137 · arXiv
- Ramachandran P et al (2019) Stand-alone self-attention in vision models. Adv Neural Inf Process Syst 32:68–80
- Goodfellow I et al (2014) Generative adversarial networks. Adv Neural Inf Process Syst 27:2672–2680
- Zhang H et al (2019) Self-attention generative adversarial networks. In: International conference on machine learning, pp 7354–7363
- Yu A et al (2018) QANet: combining local convolution with global self-attention for reading comprehension. In: International conference on learning representations
- Zhang J et al (2018) Ga AN: gated attention networks for learning on large and spatiotemporal graphs. In: Conference on uncertainty in artificial intelligence
- Al-Rfou R et al (2019) Character-level language modeling with deeper self-attention. AAAI Conf Artif Intell 33:3159
- Li X et al (2020) SAC: accelerating and structuring self-attention via sparse adaptive connection. Advances in Neural Information Processing Systems, 33
- Yang B et al (2019) Context-aware self-attention networks. AAAI Conf Artif Intell 33:387
- Shen T et al (2018) Di SAN: directional self-attention network for RNN/CNN-free language understanding. In: AAAI Conference on artificial intelligence, pp 5446–5455
- Shen T et al (2018) Bi-directional block self-attention for fast and memory-efficient sequence modeling. In: International conference on learning representations
- Jernite Y et al (2017) Variable computation in recurrent neural networks. In: International conference on learning representations
- Shiv V, Quirk C (2019) Novel positional encodings to enable tree-based transformers. Adv Neural Inf Process Syst 32:12081–12091
- Hoshen Y (2017) VAIN: Attentional Multi-agent predictive modeling, Advances in Neural Information Processing Systems, 30, Long Beach, CA
- Hu S et al (2021) UPDeT: universal multi-agent reinforcement learning via policy decoupling with transformers. In: International conference on learning representations
- Wu S et al (2020) Adversarial sparse transformer for time series forecasting. Advances in Neural Information Processing Systems, 33
- Doersch C, Gupta A, Zisserman A (2020) Cross Transformers: spatially-aware few-shot transfer. Adv Neural Inf Process Syst 33:21981–21993
- Liu X et al (2020) Learning to encode position for transformer with continuous dynamical model. In: International conference on machine learning, pp 6327–6335
- Hudson D, Zitnick L (2021) Generative adversial transformers. In: International conference on machine learning, pp 4487–4499
- Radford A et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
- Dehghani M et al (2019) Universal transformers. In: International conference on learning representations
- Ma X et al (2019) A Tensorized Transformer for Language Modeling. Adv Neural Inf Process Syst 32:2232–2242
- Devlin J et al (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proc of NAACL-HLT 2019:4171–4186
- Taylor W (1953) Cloze procedure: a new tool for measuring readability. J Bull 30(4):415
- Wang W et al (2020) MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33
- McCarley J, Chakravarti R, Sil A (2020) Structured pruning of a BERT-based question answering model, arXiv preprint arXiv:1910.06360 · arXiv
- Gong L et al (2019) Efficient training of BERT by progressively stacking. In: International conference on machine learning, pp 2337–2346
- Lan Z et al (2020) ALBERT: a lite BERT for self-supervised learning of language representations. In: International conference on learning representations
- Liu Y et al (2019) Ro BERTa: a robustly optimized BERT pretraining approach, arXiv preprint arXiv:1907.11692 · arXiv
- He P et al (2021) DeBERTa: decoding-enhanced BERT with disentangled attention. In: International conference on learning representations
- Sanh V et al (2019) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In: The 5th workshop on energy efficient machine learning and cognitive computing—NeurIPS
- Wang W et al (2020) Struct BERT: incorporating language structures into pre-training for deep language understanding. In: International conference on learning representations
- Shen S et al (2020) Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. AAAI Conf Artif Intell 34:8815
- Prakash P et al (2021) Rare BERT: transformer architecture for rare disease patient identification using administrative claims. AAAI Conf Artif Intell 35:453
- Wu Z et al (2020) Lite transformer with long-short range attention. In: International conference on learning representations UK
- Mehta S et al (2021) DeLighT: deep and light-weight transformer. In: International conference on learning representations
- Tay Y et al (2021) HyperGrid transformers: towards a single model for multiple tasks. In: International conference on learning representations
- Yun S et al (2018) Graph transformer networks. In: International conference on learning representations
- Rong Y et al (2020) Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33
- J. Yang, et al.,(2021) GraphFormers:GNN-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34
- ZhaoC et al (2020) Transformer-XH: multi-evidence reasoning with extra hop attention. In: International conference on learning representations
- You R et al (2019) AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in Neural Information Processing Systems, 32
- Fan X et al (2020) Bayesian attention modules. Advances in Neural Information Processing Systems, 33
- Brunner G et al (2020) On identifiability in transformers. In: International conference on learning representations
- Dosovitskiy A et al (2021) An image is worth $$16 \times 16$$ words: transformers for image recognition at scale. In: International conference on learning representations
- Nguyen T et al (2021) FMMformer: efficient and flexible transformer via decomposed near-field and far-field attention. Advances in Neural Information Processing Systems, 34
- Lee J et al (2019) Set transformer: a framework for attention-based permutation-invariant neural networks. In: International conference on machine learning pp 3744–3753
- Child R et al (2019) Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509 · arXiv
- Peng H et al (2021) Random feature attention. In: International conference on learning representations UK
- Chen Y et al (2021) Skyformer: remodel self-attention with Gaussian kernel and Nyström method. Advances in Neural Information Processing Systems, 34
- Zaheer M et al (2020) Big bird: transformers for longer sequences. Advances in Neural Information Processing Systems, 33
- Huang CZ et al (2019) Music transformer: generating music with long-term structure. In: International conference on learning representations UK
- Lu J et al (2021) SOFT: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34
- Zhu C et al (2021) Long-short transformer: efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34
- Jaegle A et al (2021) Perceiver: general perception with iterative attention. In: International conference on machine learning, pp 4651–4664
- El-Nouby A et al. (2021) XCiT: cross-covariance image transformers. Advances in neural information processing systems, 34
- Yu Q et al (2021) Glance-and-gaze vision transformer. Advances in Neural Information Processing Systems, 34
- Zeng Z et al (2021) You only sample (almost) once: linear cost self-attention via Bernoulli sampling. In: International conference on machine learning, pp 12,321–12,332
- Shen Z et al (2021) Efficient attention: attention with linear complexities. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 3531–3539
- Ma X et al (2021) Luna: linear unified nested attention. Advances in Neural Information Processing Systems, 34
Cited by
Mishra, L. N., & Senapati, B. (2025) · IEEE Access · doi:10.1109/access.2025.3552592 · Cited by 35
Moveh, S., Merchán-Cruz, E. A., Abuhussain, M., Dodo, Y. A., Alhumaid, S., & Alhamami, A. H. (2025) · Energies · doi:10.3390/en18061468 · Cited by 27
Mahali, M. I., Leu, J.-S., Avian, C., Darmawan, J. T., Faisal, M., Putro, N. A. S., & Prakosa, S. W. (2025) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2025.106599 · Cited by 1
Metadata
- Authors
- Soydaner, D.
- Venue
- Neural Computing and Applications
- Published
- 2022-05-25
- Volume
- 34
- Issue
- 16
- Pages
- 13371-13385
- ISSN
- 0941-0643 (ISSN-L)
- 1433-3058
- OpenAlex
- W4225141805
- Cited by
- 414
- References
- 81
- FWCI
- 24.28
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Attention mechanism in neural networks: where it comes and where it goes. https://app.cochat.ai/papers/10.1007/s00521-022-07366-3
- The paper
- Soydaner, D. (2022). Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07366-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.