Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Despite efficiency and ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers because the kernel trick makes the model a linear recurrent network that compresses data into matrix-valued states.
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredApproximate attention methods reduce compute complexity by trading off model quality but often do not achieve wall-clock speedup.
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
The Introduction (p. 1) says sparse attention and locality-sensitive hashing reduce computational complexity but do not speed up autoregressive inference. That supports the speed claim only for that setting: the paper also says these methods can be trained efficiently on long sequences, and it does not establish a general trade-off in model quality. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLinear RNNs use a vector-valued memory while linear Transformers use a matrix-valued memory.
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Section 3.3.2 (p. 4) says the linear Transformer stores a matrix as its internal state, supporting the matrix-valued-memory half of the claim. Section 3.4 (p. 5) describes that same Transformer as an RNN with two hidden states; it does not establish that linear RNNs, including state space models, generally use vector-valued memory. (LLM, conf=0.84)
Experimental — accuracy not yet measuredCurrent large language models exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons (the 'goldfish problem').
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
The Introduction (p. 1) says transformers perform impressively on various tasks, but limited context can disrupt temporal coherence and hinder long-term dependencies. It does not assess single-turn versus extended interactions or whether models maintain personalization across them; the 'goldfish problem' is the citing authors’ term. (LLM, conf=0.87)
Experimental — accuracy not yet measuredIn linear Transformer (or FWP) with zero initial memory state, the stored knowledge of a lower level (fast weight) directly affects the output of the model in another level.
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Algorithm 1 (p. 5) initializes the attention state S to zero, updates it with key–value information, and uses S to compute the output. Section 3.4 (p. 5) likewise describes attention memory as a recurrent hidden state. Our copy does not identify that state as a “fast weight” or describe its stored information as directly affecting the output of another level, so the cross-level claim is not fully supported. (LLM, conf=0.81)
Experimental — accuracy not yet measured
Different settings for α_t and φ(·) recover different variants of linear attention with Hebbian rule.
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Section 3.2 (p. 3) says different feature maps φ can yield different linearized attention implementations, including polynomial attention. Section 3.4 (p. 5) casts causal attention as a recurrence. Our copy does not describe varying α_t between fixed, learnable, channel-wise, or input-dependent settings, or characterize the recurrence as a Hebbian rule. (LLM, conf=0.85)
Experimental — accuracy not yet measuredUsing higher-order feature maps on the keys enhances the capacity of a memory.
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Section 3.2 (p. 3) says feature maps are applied to keys, and §3.2.1 (p. 3) discusses a degree-2 polynomial feature map. The paper analyzes its computational cost, but does not say that higher-order maps enhance memory capacity or that this is a commonly used technique. (LLM, conf=0.78)
Experimental — accuracy not yet measuredFor random inputs, the Frobenius norm of the linear attention memory state grows as O(T).
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Section 3.3 (p. 4) defines the linear-attention state as a cumulative quantity, and Algorithm 1 (p. 5) updates S by adding one key–value term per input. These passages support the state’s additive structure, but the paper does not state an expected Frobenius-norm bound for random inputs; its O(N) results concern computation and memory, not the norm of S. (LLM, conf=0.78)
Experimental — accuracy not yet measuredHaving the gates both learnable and input dependent gives the best results for mLSTM, in contrast to other methods.
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Section 5 (p. 8) identifies this paper’s method as a linear transformer, so it can serve as an example of an ‘other method.’ Its reported results compare linear attention with other transformer approaches, not learnable, input-dependent gates or mLSTM. The claimed best results for mLSTM are therefore not supported by this source. (LLM, conf=0.78)
Experimental — accuracy not yet measuredLearned attention mechanisms are designed for performance, not interpretability, and obfuscate the explicit interaction structure between token representations modeled in RN.
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
Modern deep learning architectures seem to be heterogeneous and are based on a combination of a subset of self-attention variants, modern recurrent neural networks, canon layers, global convolutions, and MLP blocks.
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The first generation of modern recurrent architectures are based on Hebbian-like learning rules.
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Optimizing the equivalent associative memory problem with gradient descent and weight decay recovers the original linear attention recurrence.
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Linear attention reduces the quadratic cost of softmax attention to O(T).
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Linear attention's additive update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Linear attention can be implemented recurrently by maintaining and updating a matrix-valued state.
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs.
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
In the online case c=1 and φ(·)=(·), the SWLA recurrence is the same as linear attention.
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length.
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-attention complexity scales as O(n^2) with sequence length n, making unlimited context computationally infeasible even with recent linear attention variants.
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 34
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
A review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379 · see citing works
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · see citing works
LoFTR: Detector-Free Local Feature Matching with Transformers
Sun, J., Shen, Z., Wang, Y., Bao, H., & Zhou, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00881 · Cited by 1,824 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01181 · Cited by 1,458 · see citing works
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
scGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,383 · see citing works
“Although Flash-Attention was adopted, any efficient transformers can be potentially utilized for scGPT as well, such as Transformers with linear complexity (Linformer) [66] and Kernelized Self-Attention (KSA) [67].”
MethodTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925 · see citing works
Efficiently Modeling Long Sequences with Structured State Spaces
Gu, A., Goel, K., & Ré, C. (2021) · arXiv · doi:10.48550/arxiv.2111.00396 · Cited by 805 · see citing works
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 752 · see citing works
Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3116668 · Cited by 735 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
“There is also a long line of research on using dense attention matrices, but defined by low-rank kernels substituting softmax (Katharopoulos et al., 2020; Shen et al., 2018).”
BackgroundMethodInfluentialFNet: Mixing Tokens with Fourier Transforms
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontañón, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.319 · Cited by 575 · see citing works
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Li, R., Zheng, S., Zhang, C., Duan, C., Su, J., Wang, L., & Atkinson, P. M. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3093977 · Cited by 516 · see citing works
Anchor DETR: Query Design for Transformer-Based Detector
Wang, Y., Zhang, X., Yang, T., & Sun, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20158 · Cited by 511 · see citing works
3D Object Detection with Pointformer
Pan, X., Xia, Z., Song, S., Li, L. E., & Huang, G. M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00738 · Cited by 493 · see citing works
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · see citing works
“These methods range from sparse-approximation [53, 77] to low-rank approximation [13, 52, 88], and their combinations [3, 9, 96].”
MethodLi, R., Zheng, S., Zhang, C., Duan, C., Wang, L., & Atkinson, P. M. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.09.005 · Cited by 462 · see citing works
Transforming Model Prediction for Tracking
Mayer, C., Danelljan, M., Bhat, G., Paul, M., Paudel, D. P., Yu, F., & Gool, L. V. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00853 · Cited by 460 · see citing works
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · see citing works
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390 · see citing works
Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376 · see citing works
Autoregressive Visual Tracking
Wei, X., Bai, Y., Zheng, Y., Shi, D., & Gong, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00935 · Cited by 378 · see citing works
Fast Convergence of DETR with Spatially Modulated Co-Attention
Gao, P., Zheng, M., Wang, X., Dai, J., & Li, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00360 · Cited by 386 · see citing works
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
Zhang, P., Dai, X., Yang, J., Xiao, B., Yuan, L., Zhang, L., & Gao, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00299 · Cited by 358 · see citing works
FLatten Transformer: Vision Transformer using Focused Linear Attention
Han, D., Pan, X., Han, Y., Song, S., & Huang, G. M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00548 · Cited by 359 · see citing works
A-ViT: Adaptive Tokens for Efficient Vision Transformer
Yin, H., Vahdat, A., Alvarez, J. M., Mallya, A., Kautz, J., & Molchanov, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01054 · Cited by 360 · see citing works
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction
Cai, H., Li, J., Hu, M., Gan, C., & Han, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01587 · Cited by 356 · see citing works
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration
Li, Y., Fan, Y., Xiang, X., Demandolx, D., Ranjan, R., Timofte, R., & Gool, L. V. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01753 · Cited by 323 · see citing works
Multistage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
Li, R., Zheng, S., Duan, C., Su, J., & Zhang, C. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3063381 · Cited by 300 · see citing works
TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers
Ding, Y., Yuan, W., Zhu, Q., Zhang, H., Liu, X., Wang, Y., & Liu, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00839 · Cited by 287 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286 · see citing works
REGTR: End-to-end Point Cloud Correspondences with Transformers
Yew, Z., & Lee, G. H. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00656 · Cited by 317 · see citing works
Jin, Z. J., Qiu, Y., Zhang, K., Li, H., & Luo, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3559891 · Cited by 279 · see citing works
MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing
Qiu, Y., Zhang, K., Wang, C., Luo, W., Li, H., & Jin, Z. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01176 · Cited by 267 · see citing works
XCiT: Cross-Covariance Image TransformersPreprint
El-Nouby, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Verbeek, J. J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09681 · Cited by 273 · see citing works
Long Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241 · see citing works
MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer
Huang, K., Wu, T.-H., Su, H.-T., & Hsu, W. H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00398 · Cited by 238 · see citing works
NILM applications: Literature review of learning approaches, recent developments and challenges
Angelis, G.-F., Timplalexis, C., Krinidis, S., Ioannidis, D., & Tzovaras, D. K. (2022) · Energy and Buildings · doi:10.1016/j.enbuild.2022.111951 · Cited by 221 · see citing works
Zhang, H., Zhu, Y., Wang, D., Zhang, L., Chen, T., Wang, Z., & Ye, Z. (2024) · Applied Sciences · doi:10.3390/app14135683 · Cited by 221 · see citing works
OnePose: One-Shot Object Pose Estimation without CAD Models
Sun, J., Wang, Z., Zhang, S., He, X., Zhao, H., Zhang, G., & Zhou, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00670 · Cited by 182 · see citing works
ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution
Zhang, M., Zhang, C., Zhang, Q., Guo, J., Gao, X., & Zhang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02109 · Cited by 175 · see citing works
Efficient Visual Tracking with Exemplar Transformers
Blatter, P., Kanakis, M., Danelljan, M., & Gool, L. V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00162 · Cited by 171 · see citing works
Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16091 · Cited by 176 · see citing works
End-to-End Transformer-Based Models in Textual-Based NLP
Rahali, A., & Akhloufi, M. A. (2023) · AI · doi:10.3390/ai4010004 · Cited by 187 · see citing works
Lightweight transformer image feature extraction network
Zheng, W., Lu, S., Yang, Y., Yin, Z., & Yin, L. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1755 · Cited by 167 · see citing works
A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking
Papa, L., Russo, P., Amerini, I., & Zhou, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392941 · Cited by 177 · see citing works
RayMVSNet++: Learning Ray-Based 1D Implicit Fields for Accurate Multi-View Stereo
Shi, Y., Xi, J., Hu, D., Cai, Z., & Xu, K. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3296163 · Cited by 160 · see citing works
CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
Cho, S., Shin, H., Hong, S., Arnab, A., Seo, P. H., & Kim, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00394 · Cited by 165 · see citing works
Zhao, S., Zhu, L., Wang, X., & Yang, Y. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531950 · Cited by 157 · see citing works
DeepConsensus improves the accuracy of sequences with a gap-aware sequence transformer
Baid, G., Cook, D. E., Shafin, K., Yun, T., Llinares-López, F., Berthet, Q., Belyaeva, A., Töpfer, A., Wenger, A. M., Rowell, W. J., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01435-7 · Cited by 147 · see citing works
Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation
Fan, X., Liu, Z., Lian, J., Zhao, W. X., Xie, X., & Wen, J.-R. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462978 · Cited by 152 · see citing works
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146 · see citing works
Scalable Vision Transformers with Hierarchical Pooling
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00043 · Cited by 168 · see citing works
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145 · see citing works
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · see citing works
“To reduce the computational requirement of attention on such long context, there have been numerous methods proposed to approximate attention [2, 3, 4, 8, 9, 14, 19, 20].”
Random Feature AttentionPreprint
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 169 · see citing works
Self-citationEnd-to-End Object Detection with Adaptive Clustering TransformerPreprint
Zheng, M., Gao, P., Zhang, R., Li, K., Wang, X., Li, H., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09315 · Cited by 166 · see citing works
Learned Token Pruning for Transformers
Kim, S., Shen, S., Thorsley, D., Gholami, A., Kwon, W., Hassoun, J., & Keutzer, K. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539260 · Cited by 139 · see citing works
MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
Chen, J., & Ho, C. M. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00086 · Cited by 139 · see citing works
Sparse DETR: Efficient End-to-End Object Detection with Learnable SparsityPreprint
Roh, B., Shin, J., Shin, W., & Kim, S. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2111.14330 · Cited by 329 · see citing works
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137 · see citing works
GENA-LM: a family of open-source foundational DNA language models for long sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O. L., & Burtsev, M. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1310 · Cited by 142 · see citing works
MACU-Net for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
Li, R., Duan, C., Zheng, S., Zhang, C., & Atkinson, P. M. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3052886 · Cited by 136 · see citing works
Semantics lead all: Towards unified image registration and fusion from a semantic perspective
Xie, H., Zhang, Y., Qiu, J., Zhai, X., Liu, X., Yang, Y., Zhao, S., Luo, Y., & Zhong, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101835 · Cited by 134 · see citing works
PnP-DETR: Towards Efficient Visual Analysis with Transformers
Wang, T., Yuan, L., Chen, Y., Feng, J., & Yan, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00462 · Cited by 133 · see citing works
Recurring the Transformer for Video Action Recognition
Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., & Yu, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01367 · Cited by 137 · see citing works
LambdaNetworks: Modeling Long-Range Interactions Without Attention
Bello, I. (2021) · arXiv · doi:10.48550/arxiv.2102.08602 · Cited by 129 · see citing works
Cirstea, R.-G., Guo, C., Yang, B., Kieu, T., Dong, X., & Pan, S. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/277 · Cited by 157 · see citing works
FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer
Liu, Z., Yang, X., Tang, H., Shang, Y., & Han, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00122 · Cited by 126 · see citing works
Selective Structured State-Spaces for Long-Form Video Understanding
Wang, J., Zhu, W., Wang, P., Yu, X., Liu, L. L., Omar, M., & Hamid, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00618 · Cited by 121 · see citing works
Generative artificial intelligence: a historical perspective
He, R., Cao, J., & Tan, T. (2025) · National Science Review · doi:10.1093/nsr/nwaf050 · Cited by 146 · see citing works
Adaptive Frequency Filters As Efficient Global Token Mixers
Huang, Z., Zhang, Z., Lan, C., Zha, Z.-J., Lu, Y. Q., & Guo, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00556 · Cited by 116 · see citing works
Sub-word Level Lip Reading With Visual Attention
Prajwal, K. R., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00510 · Cited by 115 · see citing works
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., & Catanzaro, B. (2021) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2111.13587 · Cited by 195 · see citing works
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111 · see citing works
Theme Transformer: Symbolic Music Generation With Theme-Conditioned Transformer
Shih, Y.-J., Wu, S.-L., Zalkow, F., Müller, M., & Yang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3161851 · Cited by 115 · see citing works
Deep learning for predicting respiratory rate from biosignals
Kumar, A., Ritam, M., Han, L., Guo, S., & Chandra, R. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105338 · Cited by 109 · see citing works
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
Netmamba: Efficient Network Traffic Classification Via Pre-Training Unidirectional Mamba
Wang, T., Xie, X., Wang, W., Wang, C., Zhao, Y., & Cui, Y. (2024) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp61940.2024.10858569 · Cited by 115 · see citing works
Local-Global Context Aware Transformer for Language-Guided Video Segmentation
Liang, C., Wang, W., Zhou, T., Miao, J., Luo, Y., & Yang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3262578 · Cited by 106 · see citing works
Global Self-Attention as a Replacement for Graph Convolution
Hussain, M. S., Zaki, M. J., & Subramanian, D. (2021) · Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539296 · Cited by 215 · see citing works
Compression of Deep Learning Models for Text: A Survey
Gupta, M., & Agrawal, P. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3487045 · Cited by 112 · see citing works
RPConvformer: A novel Transformer-based deep neural networks for traffic flow prediction
Wen, Y., Xu, P., Li, Z., Xu, W. A., & Wang, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119587 · Cited by 105 · see citing works
Is Attention Better Than Matrix Decomposition?Preprint
Geng, Z., Guo, M.-H., Chen, H., Li, X., Wei, K., & Lin, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04553 · Cited by 127 · see citing works
Transformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 148 · see citing works
Fastformer: Additive Attention Can Be All You NeedPreprint
Wu, C., Wu, F., Qi, T., Huang, Y., & Xie, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09084 · Cited by 128 · see citing works
Shen, H., Wang, Z., Zhou, X., Lamantia, M., Yang, K., Chen, P., & Wang, J. (2022) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2022.3157652 · Cited by 94 · see citing works
Focused Attention in Transformers for interpretable classification of retinal images
Playout, C., Duval, R., Boucher, M. C., & Farida Cheriet, A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102608 · Cited by 96 · see citing works
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299 · see citing works
3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition
Wang, L., & Koniusz, P. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00544 · Cited by 89 · see citing works
Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers
Martinez-Gonzalez, A. N., Villamizar, M., & Odobez, J. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00257 · Cited by 90 · see citing works
Transformers for Tabular Data Representation: A Survey of Models and Applications
Badaro, G., Saeed, M., & Papotti, P. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00544 · Cited by 96 · see citing works
Video Background Music Generation with Controllable Music Transformer
Di, S., Jiang, Z., Liu, S., Wang, Z., Zhu, L., He, Z., Liu, H., & Yan, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475195 · Cited by 89 · see citing works
SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation
Xie, B., Cao, J., Xie, J., Khan, F. S., & Pang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00329 · Cited by 92 · see citing works
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
