The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 301–400 of 855
He, J., Xu, H., Li, S., & Yu, Y. F. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3435883 · Cited by 16
Learning Human Mesh Recovery in 3D Scenes
Shen, Z., Cen, Z., Peng, S., Shuai, Q., Bao, H., & Zhou, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01634 · Cited by 17
Not All Memories are Created Equal: Learning to Forget by ExpiringPreprint
Sukhbaatar, S., Ju, D., Poff, S., Roller, S., Szlam, A. D., Weston, J., & Fan, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06548 · Cited by 17
Stable, Fast and Accurate: Kernelized Attention with Relative Positional EncodingPreprint
Luo, S., Li, S., Cai, T., He, D., Peng, D., Zheng, S., Ke, G., Wang, L., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12566 · Cited by 24
Learning to (Learn at Test Time): RNNs with Expressive Hidden States
Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.04620 · Cited by 18
“We now show that among these induced instantiations, the TTT layer with a linear model and batch GD is equivalent to linear attention [44], a widely known RNN layer.”
InfluentialFan, H., Zhu, M., Hu, Y., Feng, H., He, Z., Liu, H., & Liu, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131270 · Cited by 17
A comprehensive review of lightweight deep learning models for edge computing with future directions
Lai, N., Dewi, D. A., Maidin, S. S., Xiao, W., Zhao, S., & Hu, Q. (2026) · Discover Computing · doi:10.1007/s10791-026-10021-3 · Cited by 22
∞-former: Infinite Memory Transformer-former: Infinite Memory Transformer
Martins, P. H., Marinho, Z., & Martins, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.375 · Cited by 15
Differentiable Short-Term Models for Efficient Online Learning and Prediction in Monophonic Music
Bjare, M. R., Lattner, S., & Widmer, G. (2022) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.123 · Cited by 15
Sparse-to-Local-Dense Matching for Geometry-Guided Correspondence Estimation
Li, S., Zhao, Q., & Xia, Z. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3287500 · Cited by 15
A Coarse-to-Fine Feature Match Network Using Transformers for Remote Sensing Image Registration
Liang, C., Dong, Y., Zhao, C., & Sun, Z. (2023) · Remote Sensing · doi:10.3390/rs15133243 · Cited by 15
SPTR: Structure-Preserving Transformer for Unsupervised Indoor Depth Completion
Zhao, L., Zheng, W., Duan, Y., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3312664 · Cited by 15
Autoregressive transformers for data-driven spatiotemporal learning of turbulent flows
Patil, A., Viquerat, J., & Hachem, E. (2023) · APL Machine Learning · doi:10.1063/5.0152212 · Cited by 15
A Review of Trajectory Prediction Methods for the Vulnerable Road User
Schuetz, E., & Flohr, F. B. (2023) · Robotics · doi:10.3390/robotics13010001 · Cited by 20
Softmax-Free Linear Transformers
Lu, J., Zhang, J., Zhu, X., Feng, J., Xiang, T., & Zhang, L. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02035-5 · Cited by 20
Accurate Vision-Enabled UAV Location Using Feature-Enhanced Transformer-Driven Image Matching
Wang, H., Zhou, F., & Wu, Q. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3387495 · Cited by 15
Auto-Spikformer: Spikformer architecture search
Che, K.-W., Zhou, Z., Niu, J., Ma, Z., Fang, W., Chen, Y.-Q., Shen, S., Yuan, L., & Tian, Y. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1372257 · Cited by 18
CR-former: Single-Image Cloud Removal With Focused Taylor Attention
Wu, Y., Deng, Y., Zhou, S., Liu, Y., Huang, W., & Wang, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3506780 · Cited by 15
Combining hybrid DNN-HMM ASR systems with attention-based models using lattice rescoring
Li, Q., Zhang, C., & Woodland, P. C. (2022) · Speech Communication · doi:10.1016/j.specom.2022.12.002 · Cited by 15
Global Attention Improves Graph Networks GeneralizationPreprint
Puny, O., Ben-Hamu, H., & Lipman, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07846 · Cited by 19
MV-YOLO: An Efficient Small Object Detection Framework Based on Mamba
Wu, S., Lu, X.-Y., Guo, C., & Guo, H. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3584955 · Cited by 27
PermuteFormer: Efficient Relative Position Encoding for Long Sequences
Chen, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.828 · Cited by 15
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU
Kahardipraja, P., Madureira, B., & Schlangen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.90 · Cited by 14
Lyu, B., Fan, C., Ming, Y., Zhao, P., & Hu, N. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3245407 · Cited by 14
AutoRecon: Automated 3D Object Discovery and Reconstruction
Wang, Y., He, X., Peng, S., Lin, H., Bao, H., & Zhou, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02048 · Cited by 14
BlinkLinMulT: Transformer-Based Eye Blink Detection
Fodor, Á., Fenech, K., & Lörincz, A. (2023) · Journal of Imaging · doi:10.3390/jimaging9100196 · Cited by 15
ResMatch: Residual Attention Learning for Feature Matching
Deng, Y., Zhang, K., Zhang, S., Li, Y., & Ma, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27915 · Cited by 16
EF-CenterNet: An efficient anchor-free model for UAV-based banana leaf disease detection
Thai, H.-T., Le, K.-H., & Nguyen, N. L.-T. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.109927 · Cited by 14
IAFMVS: Iterative Depth Estimation with Adaptive Features for Multi-View Stereo
Zhao, G., Wei, H., & He, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129682 · Cited by 14
Learning to See Low-Light Images via Feature Domain Adaptation
Yang, Q., Cheng, Q., Yue, H., Zhang, L., Liu, Y., & Yang, J. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3563775 · Cited by 14
UM-Mamba: An efficient U-network with medical visual state space for medical image segmentation
Chen, H., Liu, Q., Fu, Z., & Liu, L. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104436 · Cited by 14
Learning Enriched Feature Descriptor for Image Matching and Visual Measurement
Rao, Y., Ju, Y., Wang, S., Gao, F., Fan, H., & Dong, J. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3249237 · Cited by 16
The NLP Task Effectiveness of Long-Range Transformers
Qin, G., Feng, Y., & Van Durme, B. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.273 · Cited by 14
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Irie, K., Schlag, I., Csordás, R., & Schmidhuber, J. (2021) · arXiv · doi:10.48550/arxiv.2106.06295 · Cited by 14
Edge-augmented Graph Transformers: Global Self-attention is Enough for Graphs.Preprint
Hussain, M. S., Zaki, M. J., & Subramanian, D. (2021) · arXiv (Cornell University) · Cited by 14
I-YOLOv11n: A Lightweight and Efficient Small Target Detection Framework for UAV Aerial Images
Ma, Y., Xi, C., Ma, T., Sun, H. X., Lu, H., Xu, X., & Chen, X. (2025) · Sensors · doi:10.3390/s25154857 · Cited by 20
Żelasko, P., Pappagari, R. R., & Dehak, N. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00420 · Cited by 14
Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation
Themyr, L., Rambour, C., Thome, N., Collins, T., & Hostettler, A. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00324 · Cited by 13
HyperMixer: An MLP-based Low Cost Alternative to Transformers
Mai, F., Pannatier, A., Fehr, F., Chen, H., Marelli, F., Fleuret, F., & Henderson, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.871 · Cited by 17
Self-citationFourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
He, Z., Yang, M., Feng, M., Yin, J., Wang, X., Leng, J., & Lin, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.570 · Cited by 26
IMP: Iterative Matching and Pose Estimation with Adaptive Pooling
Xue, F., Budvytis, I., & Cipolla, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02042 · Cited by 13
Li, Z., Gao, Z., Zhang, G., Liu, J., & Xu, L. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107291 · Cited by 13
Hori, K., Suzuki, K., & Ogata, T. (2024) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii58957.2024.10417267 · Cited by 15
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition
Liu, L., Liu, L., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3363446 · Cited by 13
DSAP: Dynamic Sparse Attention Perception Matcher for Accurate Local Feature Matching
Dai, K., Wang, K., Xie, T., Sun, T., Zhang, J., Kong, Q., Jiang, Z., Li, R., Zhao, L., & Omar, M. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3370781 · Cited by 13
BAFusion: Bidirectional Attention Fusion for 3D Object Detection Based on LiDAR and Camera
Liu, M., Jia, Y., Lyu, Y., Dong, Q., & Yang, Y. (2024) · Sensors · doi:10.3390/s24144718 · Cited by 14
A Cost-Efficient FPGA-Based CNN-Transformer Using Neural ODE
Okubo, I., Sugiura, K., & Matsutani, H. (2024) · IEEE Access · doi:10.1109/access.2024.3480977 · Cited by 14
RTLinearFormer: Semantic segmentation with lightweight linear attentions
Gu, Y., Fu, C., Song, W., Wang, X., & Chen, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129489 · Cited by 16
How Can Self-Attention Networks Recognize Dyck-n Languages?
Ebrahimi, J., Gelda, D., & Zhang, W. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.384 · Cited by 43
Poly-NL: Linear Complexity Non-local Layers With 3rd Order Polynomials
Babiloni, F., Marras, I., Kokkinos, F., Deng, J., Chrysos, G. G., & Zafeiriou, S. P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01035 · Cited by 16
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer
Wang, N., Gan, G., Zhang, P., Zhang, S., Wei, V. J., Liu, Q., & Jiang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.170 · Cited by 13
Wang, S., Zhou, F., Sun, Y., Ma, L., Zhang, J., & Zheng, Y. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00141 · Cited by 13
Integrating local and global correlations with Mamba-Transformer for multi-class anomaly detection
Ma, Z., Li, J., Jiang, K., & Wong, W. K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113740 · Cited by 12
FMAP: Learning robust and accurate local feature matching with anchor points
Dai, K., Xie, T., Wang, K., Jiang, Z., Li, R., & Zhao, L. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121328 · Cited by 12
Std-net: spatio-temporal decomposition network for video demoiréing with sparse transformers
Niu, Y., Xu, R., Lin, Z., & Liu, W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3386604 · Cited by 12
CTOD: Cross-Attentive Task-Alignment for One-Stage Object Detection
Yao, R., Rong, Y., Huang, Q., & Xiong, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3422879 · Cited by 12
Hyperspectral Object Detection Based on Spatial–Spectral Fusion and Visual Mamba
Li, W., Yuan, F., Zhang, H., Lv, Z., & Wu, B. (2024) · Remote Sensing · doi:10.3390/rs16234482 · Cited by 12
Zeng, X., Wan, H., Fan, Z., Yu, X., & Guo, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126220 · Cited by 12
Zhao, W., Shi, X., Ni, F., & Tian, Y. (2025) · Measurement · doi:10.1016/j.measurement.2025.117899 · Cited by 12
Predicting Attention Sparsity in Transformers
Treviso, M., Góis, A., Fernandes, P., Fonseca, E., & Martins, A. F. T. (2022) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2022.spnlp-1.7 · Cited by 17
TT-Net: Tensorized Transformer Network for 3D medical image segmentation
Wang, J., Qu, A., Wang, Q., Zhao, Q., Liu, J., & Wu, Q. (2023) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102234 · Cited by 13
Fast Convergence of DETR with Spatially Modulated Co-AttentionPreprint
Gao, P., Zheng, M., Wang, X., Dai, J., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.07448 · Cited by 12
Linear Transformers Are Secretly Fast Weight Memory SystemsPreprint
Schlag, I., Irie, K., & Schmidhuber, J. (2021) · arXiv (Cornell University) · Cited by 14
Transformer Acceleration with Dynamic Sparse AttentionPreprint
Liu, L., Qu, Z., Chen, Z., Ding, Y.-F., & Xie, Y. (2021) · arXiv.org · doi:10.48550/arxiv.2110.11299 · Cited by 35
OAMatcher: An Overlapping Areas-based Network for Accurate Local Feature MatchingPreprint
Dai, K., Xie, T., Wang, K., Jiang, Z., Li, R., & Zhao, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.05846 · Cited by 12
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
“The Linear Transformer [235] then served as the foundation for synthesising music from these attributes.”
CPS: Full-Song and Style-Conditioned Music Generation with Linear Transformer
Wang, W., Li, X., Jin, C., Di, L., Zhou, Q., & Tie, Y. (2022) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew56448.2022.9859286 · Cited by 12
Linear Complexity Self-Attention with $3^{\text{rd}}$ Order Polynomials
Babiloni, F., Marras, I., Deng, J., Kokkinos, F., Maggioni, M., Chrysos, G. G., Torr, P. H. S., & Zafeiriou, S. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3231971 · Cited by 11
Lu, Y., Gong, M., Li, J., & Ma, J. (2024) · Plants · doi:10.3390/plants13020149 · Cited by 11
Zhang, Z., Yin, J., & Wang, L. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111551 · Cited by 11
DanceComposer: Dance-to-Music Generation Using a Progressive Conditional Music Generator
Liang, X., Li, W., Huang, L., & Gao, C. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3405734 · Cited by 15
Multi-view depth estimation based on multi-feature aggregation for 3D reconstruction
Zhang, C. Y., Liang, L., Zhou, J., & Xu, Y. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.103954 · Cited by 11
EA-MVSNet: Learning Error-Awareness for Enhanced Multi-View Stereo
Gu, W., Xiao, H., Zhao, X., & Kang, W. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3430115 · Cited by 11
Li, Y., Wei, C., Wu, D., Cui, Y., He, P., Zhang, Y., & Wang, R. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3511538 · Cited by 11
Dong, P.-C., Tan, Y.-H., Liu, X., Luo, P., Liu, Y., Liang, L., Zhou, Y., Pang, D., Yung, M.-T., Zhang, D., et al. (2025) · IEEE International Solid-State Circuits Conference · doi:10.1109/isscc49661.2025.10904499 · Cited by 24
FocusMorph: A novel multi-scale fusion network for 3D brain MR image registration
Liu, T., Zhang, Z., Fan, G., Li, N., Xu, C., Li, B., Zhao, G., & Zhou, S. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111761 · Cited by 11
Reinforced Intelligence Through Active Interaction in Real World: A Survey on Embodied AI
Kwon, W., Baek, S., Baek, J., Shin, W., Gwak, M., Park, P., & Lee, S. (2025) · International Journal of Control Automation and Systems · doi:10.1007/s12555-025-0127-1 · Cited by 13
Resource-efficient Hybrid X-formers for Vision
Pranav Jeevan, P., & Sethi, A. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00361 · Cited by 12
Yu, T., Khalitov, R., Cheng, L., & Yang, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00077 · Cited by 11
Maneeroj, S., & Sritrakool, N. (2022) · IEEE Access · doi:10.1109/access.2022.3182390 · Cited by 13
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
Wang, X., Zhou, W., Zhang, Q., Zhou, J., Gao, S., Wang, J., Zhang, M., Gao, X., Chen, Y. W., & Gui, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.35 · Cited by 14
Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer
Wu, Y., Kan, S., Zeng, M., & Li, M. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/493 · Cited by 13
Wang, K., Zhang, X., Wan, F., Chen, R., Zhang, J., Wang, J., & Yang, Y. B. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3322506 · Cited by 11
Chen, D., Zhang, Y., Kundu, S., Li, C., & Beerel, P. A. (2023) · doi:10.1109/iccad57390.2023.10323702 · Cited by 12
Efficient Attentions for Long Document SummarizationPreprint
Huang, L., Cao, S., Parulian, N. N., Ji, H., & Wang, L. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.02112 · Cited by 507
The Piano Inpainting ApplicationPreprint
Hadjeres, G., & Crestel, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05944 · Cited by 42
Scatterbrain: Unifying Sparse and Low-rank Attention ApproximationPreprint
Chen, B., Dao, T., Winsor, E., Song, Z., Rudra, A., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15343 · Cited by 26
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström MethodPreprint
Chen, Y., Zeng, Q., Ji, H., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00035 · Cited by 11
Attention Approximates Sparse Distributed MemoryPreprint
Bricken, T., & Pehlevan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05498 · Cited by 15
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10
“Linear-attention Transformers can evaluate their next-token function in O ( n ) time in their input length, as observed by Katharopoulos et al. (2020).”
BackgroundFeature matching driven background generalization neural networks for surface defect segmentation
Chen, B., Niu, T., Zhang, R., Zhang, H., Lin, Y., & Li, B. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111451 · Cited by 10
Neuromorphic Vision Restoration Network for Advanced Driver Assistance System
Xie, D., Xiao, H., Hu, X., Zhou, Y., Zhou, G., & Duan, S. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3367728 · Cited by 10
Addlesee, A., & Eshghi, A. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1343052 · Cited by 11
Yu, H., Hou, Y., Wang, F., Wang, J., Zhu, J., & Guo, J. (2024) · Sensors · doi:10.3390/s24165220 · Cited by 10
Optimizing Foundation Model Inference on a Many-Tiny-Core Open-Source RISC-V Platform
Potocnik, V., Colagrande, L., Fischer, T., Bertaccini, L., Pagliari, D. J., Burrello, A., & Benini, L. (2024) · IEEE transactions on circuits and systems for artificial intelligence. · doi:10.1109/tcasai.2024.3459412 · Cited by 13
Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural Network
Li, Z., & Ma, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3512352 · Cited by 12
Context-Aware Multi-view Stereo Network for Efficient Edge-Preserving Depth Estimation
Su, W., & Tao, W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02337-8 · Cited by 10
Wei, Z., Qu, S., Li, Z., Shi, Q., & Zhang, C. (2025) · Sensors · doi:10.3390/s25072062 · Cited by 12
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers
Nauen, T. C., Palacio, S., Raue, F., & Dengel, A. R. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00676 · Cited by 20
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
