Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 414

The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.

Attention mechanism in neural networks: where it comes and where it goes

Open access
Soydaner, Derya·2022·Neural Computing and Applications

Abstract

A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00521-022-07366-3

Cites

81 of 81 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 81 references

Research works printed but not linked: 88

  1. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. The MIT Press
  2. Noton D, Stark L (1971) Eye movements and visual perception. Sci Am 224(6):34
  3. Alpaydın E (1995) Selective attention for handwritten digit recognition. Adv Neural Inf Process Syst 8:771–777
  4. Ahmad S (1991) VISIT: a neural model of covert visual attention. Adv Neural Inf Process Syst 4:420–427
  5. Brown T et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901
  6. Vaswani A et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30:5998–6008
  7. Milanese R et al (1994) Integration of bottom-up and top-down cues for visual attention using non-linear relaxation. In: IEEE computer society conference on computer vision and pattern recoginition, Seattle, WA, pp 781–785 · doi:10.1109/cvpr.1994.323898
  8. Zhang W et al (2006) A computational model of eye movements during object class detection. Adv Neural Inf Process Syst 19:1609–1616
  9. Walther D et al (2002) Attentional selection for object recognition—A gentle way. In: Bu¨lthoff HH, Wallraven C, Lee SW, Poggio TA (eds) International workshop on biologically motivated computer vision. Springer, Berlin, Heidelberg, pp 472–479
  10. Gould S et al (2007) Peripheral-foveal vision for real-time object recognition and tracking in video. In: International joint conference on artificial intelligence (IJCAI) pp 2115–2121
  11. Larochelle H, Hinton G (2010) Learning to combine foveal glimpses with a third-order Boltzmann machine. Adv Neural Inf Process Syst 23:1243–1251
  12. Mnih V et al (2014) Recurrent models of visual attention. Adv Neural Inf Process Syst 27:2204–2212
  13. Stollenga M et al (2014) Deep networks with internal selective attention through feedback connections. Adv Neural Inf Process Syst 27:3545–3553
  14. Sutskever I, Vinyals O, Le Q (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27:3104–3112
  15. Xu K et al (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057
  16. Lu J et al (2016) Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems. 29
  17. Sukhbaatar S et al (2015) End-to-end memory networks. Adv Neural Inf Process Syst 28:2440–2448
  18. Chorowski J et al (2015) Attention-based models for speech recognition. Adv Neural Inf Process Syst 28:577–585
  19. Chen K et al (2015) ABC-CNN: An attention based convolutional neural network for visual question answering, arXiv preprint arXiv:1511.05960 · arXiv
  20. Sorokin I et al (2015) Deep attention recurrent Q-network, arXiv preprint arXiv:1512.01693 · arXiv
  21. Ba J et al (2015) Learning wake-sleep recurrent attention models. Adv Neural Inf Process Syst 28:2593–2601
  22. Gregor K et al (2015) DRAW: a recurrent neural network for image generation. In: International conference on machine learning, pp 1462–1471
  23. Mansimov E et al (2016) Generating images from captions with attention. In: International conference on learning representations
  24. Reed S et al (2016) Learning what and where to draw. Adv Neural Inf Process Syst 29:217–225
  25. Kerg G et al (2020) Untangling tradeoffs between recurrence and self-attention in neural networks. Advances in Neural Information Processing Systems, 33
  26. Lin Z et al (2017) A structured self-attentive sentence embedding. In: International conference on learning representations
  27. Paulus R, Xiong C, Socher R (2018) A deep reinforced model for abstractive summarization. In: International conference on learning representations
  28. Vyas A et al (2020) Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33
  29. Kaiser L et al (2017) One model to learn them all. arXiv preprint arXiv:1706.05137 · arXiv
  30. Ramachandran P et al (2019) Stand-alone self-attention in vision models. Adv Neural Inf Process Syst 32:68–80
  31. Goodfellow I et al (2014) Generative adversarial networks. Adv Neural Inf Process Syst 27:2672–2680
  32. Zhang H et al (2019) Self-attention generative adversarial networks. In: International conference on machine learning, pp 7354–7363
  33. Yu A et al (2018) QANet: combining local convolution with global self-attention for reading comprehension. In: International conference on learning representations
  34. Zhang J et al (2018) Ga AN: gated attention networks for learning on large and spatiotemporal graphs. In: Conference on uncertainty in artificial intelligence
  35. Al-Rfou R et al (2019) Character-level language modeling with deeper self-attention. AAAI Conf Artif Intell 33:3159
  36. Li X et al (2020) SAC: accelerating and structuring self-attention via sparse adaptive connection. Advances in Neural Information Processing Systems, 33
  37. Yang B et al (2019) Context-aware self-attention networks. AAAI Conf Artif Intell 33:387
  38. Shen T et al (2018) Di SAN: directional self-attention network for RNN/CNN-free language understanding. In: AAAI Conference on artificial intelligence, pp 5446–5455
  39. Shen T et al (2018) Bi-directional block self-attention for fast and memory-efficient sequence modeling. In: International conference on learning representations
  40. Jernite Y et al (2017) Variable computation in recurrent neural networks. In: International conference on learning representations
  41. Shiv V, Quirk C (2019) Novel positional encodings to enable tree-based transformers. Adv Neural Inf Process Syst 32:12081–12091
  42. Hoshen Y (2017) VAIN: Attentional Multi-agent predictive modeling, Advances in Neural Information Processing Systems, 30, Long Beach, CA
  43. Hu S et al (2021) UPDeT: universal multi-agent reinforcement learning via policy decoupling with transformers. In: International conference on learning representations
  44. Wu S et al (2020) Adversarial sparse transformer for time series forecasting. Advances in Neural Information Processing Systems, 33
  45. Doersch C, Gupta A, Zisserman A (2020) Cross Transformers: spatially-aware few-shot transfer. Adv Neural Inf Process Syst 33:21981–21993
  46. Liu X et al (2020) Learning to encode position for transformer with continuous dynamical model. In: International conference on machine learning, pp 6327–6335
  47. Hudson D, Zitnick L (2021) Generative adversial transformers. In: International conference on machine learning, pp 4487–4499
  48. Radford A et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9
  49. Dehghani M et al (2019) Universal transformers. In: International conference on learning representations
  50. Ma X et al (2019) A Tensorized Transformer for Language Modeling. Adv Neural Inf Process Syst 32:2232–2242
  51. Devlin J et al (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proc of NAACL-HLT 2019:4171–4186
  52. Taylor W (1953) Cloze procedure: a new tool for measuring readability. J Bull 30(4):415
  53. Wang W et al (2020) MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33
  54. McCarley J, Chakravarti R, Sil A (2020) Structured pruning of a BERT-based question answering model, arXiv preprint arXiv:1910.06360 · arXiv
  55. Gong L et al (2019) Efficient training of BERT by progressively stacking. In: International conference on machine learning, pp 2337–2346
  56. Lan Z et al (2020) ALBERT: a lite BERT for self-supervised learning of language representations. In: International conference on learning representations
  57. Liu Y et al (2019) Ro BERTa: a robustly optimized BERT pretraining approach, arXiv preprint arXiv:1907.11692 · arXiv
  58. He P et al (2021) DeBERTa: decoding-enhanced BERT with disentangled attention. In: International conference on learning representations
  59. Sanh V et al (2019) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In: The 5th workshop on energy efficient machine learning and cognitive computing—NeurIPS
  60. Wang W et al (2020) Struct BERT: incorporating language structures into pre-training for deep language understanding. In: International conference on learning representations
  61. Shen S et al (2020) Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. AAAI Conf Artif Intell 34:8815
  62. Prakash P et al (2021) Rare BERT: transformer architecture for rare disease patient identification using administrative claims. AAAI Conf Artif Intell 35:453
  63. Wu Z et al (2020) Lite transformer with long-short range attention. In: International conference on learning representations UK
  64. Mehta S et al (2021) DeLighT: deep and light-weight transformer. In: International conference on learning representations
  65. Tay Y et al (2021) HyperGrid transformers: towards a single model for multiple tasks. In: International conference on learning representations
  66. Yun S et al (2018) Graph transformer networks. In: International conference on learning representations
  67. Rong Y et al (2020) Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33
  68. J. Yang, et al.,(2021) GraphFormers:GNN-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34
  69. ZhaoC et al (2020) Transformer-XH: multi-evidence reasoning with extra hop attention. In: International conference on learning representations
  70. You R et al (2019) AttentionXML: label tree-based attention-aware deep model for high-performance extreme multi-label text classification. Advances in Neural Information Processing Systems, 32
  71. Fan X et al (2020) Bayesian attention modules. Advances in Neural Information Processing Systems, 33
  72. Brunner G et al (2020) On identifiability in transformers. In: International conference on learning representations
  73. Dosovitskiy A et al (2021) An image is worth $$16 \times 16$$ words: transformers for image recognition at scale. In: International conference on learning representations
  74. Nguyen T et al (2021) FMMformer: efficient and flexible transformer via decomposed near-field and far-field attention. Advances in Neural Information Processing Systems, 34
  75. Lee J et al (2019) Set transformer: a framework for attention-based permutation-invariant neural networks. In: International conference on machine learning pp 3744–3753
  76. Child R et al (2019) Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509 · arXiv
  77. Peng H et al (2021) Random feature attention. In: International conference on learning representations UK
  78. Chen Y et al (2021) Skyformer: remodel self-attention with Gaussian kernel and Nyström method. Advances in Neural Information Processing Systems, 34
  79. Zaheer M et al (2020) Big bird: transformers for longer sequences. Advances in Neural Information Processing Systems, 33
  80. Huang CZ et al (2019) Music transformer: generating music with long-term structure. In: International conference on learning representations UK
  81. Lu J et al (2021) SOFT: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34
  82. Zhu C et al (2021) Long-short transformer: efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34
  83. Jaegle A et al (2021) Perceiver: general perception with iterative attention. In: International conference on machine learning, pp 4651–4664
  84. El-Nouby A et al. (2021) XCiT: cross-covariance image transformers. Advances in neural information processing systems, 34
  85. Yu Q et al (2021) Glance-and-gaze vision transformer. Advances in Neural Information Processing Systems, 34
  86. Zeng Z et al (2021) You only sample (almost) once: linear cost self-attention via Bernoulli sampling. In: International conference on machine learning, pp 12,321–12,332
  87. Shen Z et al (2021) Efficient attention: attention with linear complexities. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 3531–3539
  88. Ma X et al (2021) Luna: linear unified nested attention. Advances in Neural Information Processing Systems, 34

Cited by

See all 414

Metadata

Authors
Soydaner, D.
Venue
Neural Computing and Applications
Published
2022-05-25
Volume
34
Issue
16
Pages
13371-13385
ISSN
  • 0941-0643 (ISSN-L)
  • 1433-3058
OpenAlex
W4225141805
Cited by
414
References
81
FWCI
24.28
Type
journal article
Language
en
Open access
green
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Attention mechanism in neural networks: where it comes and where it goes. https://app.cochat.ai/papers/10.1007/s00521-022-07366-3
The paper
Soydaner, D. (2022). Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07366-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.