Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 689

The DOI matches its Crossref record; 110 of the 110 references it cites were found in a registry or an index.

An Attentive Survey of Attention Models

Open access
Chaudhari, Sneha, Mithal, Varun, Polatkan, Gungor, Ramanath, Rohan·2021·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/3465055
Open this paper in CoChat — free account

Abstract

Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy that groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3465055

Cites

110 of 110 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 110 references

Research works printed but not linked: 43

  1. 10.5555/3327546.3327592 · doi:10.5555/3327546.3327592
  2. Lei Jimmy Ba , Volodymyr Mnih , and Koray Kavukcuoglu . 2014 . Multiple object recognition with visual attention . In Proceedings of the International Conference on Learning Representations. Lei Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2014. Multiple object recognition with visual attention. In Proceedings of the International Conference on Learning Representations.
  3. Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2015 . Neural machine translation by jointly learning to align and translate . In Proceedings of the 3rd International Conference on Learning Representations. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations.
  4. Chung-Cheng Chiu and Colin Raffel. 2017. Monotonic chunkwise attention. arXiv:1712.05382. Retrieved from https://arxiv.org/abs/1712.05382. Chung-Cheng Chiu and Colin Raffel. 2017. Monotonic chunkwise attention. arXiv:1712.05382. Retrieved from https://arxiv.org/abs/1712.05382. · arXiv
  5. 10.5555/2969239.2969304 · doi:10.5555/2969239.2969304
  6. Mostafa Dehghani , Stephan Gouws , Oriol Vinyals , Jakob Uszkoreit , and Lukasz Kaiser . 2019 . Universal transformers . In Proceedings of the 7th International Conference on Learning Representations. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In Proceedings of the 7th International Conference on Learning Representations.
  7. Alex Graves Greg Wayne and Ivo Danihelka. 2014b. Neural turing machines. arXiv:1410.5401. Retrieved from https://arxiv.org/abs/1410.5401. Alex Graves Greg Wayne and Ivo Danihelka. 2014b. Neural turing machines. arXiv:1410.5401. Retrieved from https://arxiv.org/abs/1410.5401. · arXiv
  8. 10.5555/3045118.3045274 · doi:10.5555/3045118.3045274
  9. 10.5555/2969239.2969428 · doi:10.5555/2969239.2969428
  10. Saumya Jetley , Nicholas A. Lord , Namhoon Lee , and Philip Torr . 2018 . Learn to pay attention . In Proceedings of the International Conference on Learning Representations. Saumya Jetley, Nicholas A. Lord, Namhoon Lee, and Philip Torr. 2018. Learn to pay attention. In Proceedings of the International Conference on Learning Representations.
  11. Salman Khan , Muzammal Naseer , Munawar Hayat , Syed Waqas Zamir , Fahad Shahbaz Khan, and Mubarak Shah. 2021 . Transformers in vision: A survey. arXiv:2101.01169. Retrieved from https://arxiv.org/abs/2101.01169. Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey. arXiv:2101.01169. Retrieved from https://arxiv.org/abs/2101.01169. · arXiv
  12. 10.5555/3045390.3045536 · doi:10.5555/3045390.3045536
  13. Yang Li , Lukasz Kaiser , Samy Bengio , and Si Si . 2019 a. Area attention . In Proceedings of the International Conference on Machine Learning. PMLR, 3846–3855 . Yang Li, Lukasz Kaiser, Samy Bengio, and Si Si. 2019a. Area attention. In Proceedings of the International Conference on Machine Learning. PMLR, 3846–3855.
  14. 10.5555/3454287.3454289 · doi:10.5555/3454287.3454289
  15. 10.5555/3157096.3157129 · doi:10.5555/3157096.3157129
  16. Thang Luong , Hieu Pham , and Christopher D. Manning . 2015b. Effective approaches to attention-based neural machine translation . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1412–1421 . Thang Luong, Hieu Pham, and Christopher D. Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1412–1421.
  17. 10.5555/3171837.3171854 · doi:10.5555/3171837.3171854
  18. 10.5555/3045390.3045561 · doi:10.5555/3045390.3045561
  19. 10.5555/3454287.3455544 · doi:10.5555/3454287.3455544
  20. 10.5555/2969033.2969073 · doi:10.5555/2969033.2969073
  21. Niki Parmar , Ashish Vaswani , Jakob Uszkoreit , Lukasz Kaiser , Noam Shazeer , Alexander Ku , and Dustin Tran . 2018. Image transformer.Proceedings of Machine Learning Research , Vol. 80 . 4055–4064. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer.Proceedings of Machine Learning Research, Vol. 80. 4055–4064.
  22. 10.1145/3289600.3290609 · doi:10.1145/3289600.3290609
  23. Alessandro Sordoni Philip Bachman Adam Trischler and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. arXiv:1606.02245. Retrieved from https://arxiv.org/abs/1606.02245. Alessandro Sordoni Philip Bachman Adam Trischler and Yoshua Bengio. 2016. Iterative alternating neural attention for machine reading. arXiv:1606.02245. Retrieved from https://arxiv.org/abs/1606.02245. · arXiv
  24. Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski and Armand Joulin. 2019b. Adaptive attention span in transformers. arXiv:1905.07799. Retrieved from https://arxiv.org/abs/1905.07799. Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski and Armand Joulin. 2019b. Adaptive attention span in transformers. arXiv:1905.07799. Retrieved from https://arxiv.org/abs/1905.07799. · arXiv
  25. 10.5555/2969442.2969512 · doi:10.5555/2969442.2969512
  26. 10.5555/3454287.3454838 · doi:10.5555/3454287.3454838
  27. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2020. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2020. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. · arXiv
  28. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877. Retrieved from https://arxiv.org/abs/2012.12877. · arXiv
  29. 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
  30. 10.5555/2969442.2969540 · doi:10.5555/2969442.2969540
  31. 10.5555/3298023.3298050 · doi:10.5555/3298023.3298050
  32. Jason Weston Sumit Chopra and Antoine Bordes. 2014. Memory networks. arXiv:1410.3916. Retrieved from https://arxiv.org/abs/1410.3916. Jason Weston Sumit Chopra and Antoine Bordes. 2014. Memory networks. arXiv:1410.3916. Retrieved from https://arxiv.org/abs/1410.3916. · arXiv
  33. 10.5555/3045390.3045643 · doi:10.5555/3045390.3045643
  34. Kelvin Xu , Jimmy Ba , Ryan Kiros , Kyunghyun Cho , Aaron Courville , Ruslan Salakhudinov , Rich Zemel , and Yoshua Bengio . 2015 . Proceedings of Machine Learning Research , Vol. 37 . 2048–2057. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Proceedings of Machine Learning Research, Vol. 37. 2048–2057.
  35. 10.5555/3454287.3454804 · doi:10.5555/3454287.3454804
  36. 10.5555/3304222.3304315 · doi:10.5555/3304222.3304315
  37. 10.5555/3304222.3304315 · doi:10.5555/3304222.3304315
  38. Amir Zadeh , Paul Pu Liang , Soujanya Poria , Prateek Vij , Erik Cambria , and Louis-Philippe Morency . 2018 . Multi-attention recurrent network for human communication comprehension . In Proceedings of the AAAI Conference on Artificial Intelligence. 5642–5649 . Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018. Multi-attention recurrent network for human communication comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence. 5642–5649.
  39. Ruochi Zhang , Yuesong Zou , and Jian Ma . 2020 . Hyper-SAGNN: A self-attention based graph neural network for hypergraphs . In Proceedings of the International Conference on Learning Representations. Ruochi Zhang, Yuesong Zou, and Jian Ma. 2020. Hyper-SAGNN: A self-attention based graph neural network for hypergraphs. In Proceedings of the International Conference on Learning Representations.
  40. Shenjian Zhao and Zhihua Zhang . 2018. Attention-via-attention neural machine translation . In Association for the Advancement of Artificial Intelligence. Shenjian Zhao and Zhihua Zhang. 2018. Attention-via-attention neural machine translation. In Association for the Advancement of Artificial Intelligence.
  41. 10.5555/3172077.3172382 · doi:10.5555/3172077.3172382
  42. Chang Zhou , Jinze Bai , Junshuai Song , Xiaofei Liu , Zhengchao Zhao , Xiusi Chen , and Jun Gao . 2018 . ATRank: An attention-based user behavior modeling framework for recommendation . In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, 4564–4571 . Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018. ATRank: An attention-based user behavior modeling framework for recommendation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, 4564–4571.
  43. Xizhou Zhu , Weijie Su , Lewei Lu , Bin Li , Xiaogang Wang , and Jifeng Dai . 2021 . Deformable DETR: Deformable transformers for end-to-end object detection . In Proceedings of the International Conference on Learning Representations. Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. Deformable DETR: Deformable transformers for end-to-end object detection. In Proceedings of the International Conference on Learning Representations.

Cited by

Metadata

Authors
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R.
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2021-10-22
Volume
12
Issue
5
Pages
1-32
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/3465055
OpenAlex
W2941531368
Cited by
689
References
110
FWCI
42.59
Type
journal article
Language
en
Open access
green
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.