The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 201–300 of 855
Deep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 49
SimA: Simple Softmax-free Attention for Vision Transformers
Koohpayegani, S. A., & Pirsiavash, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00259 · Cited by 50
Liu, A., & Zhang, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123884 · Cited by 31
When the Edge Meets Transformers: Distributed Inference with Transformer Models
Hu, C., & Li, B. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00017 · Cited by 34
A non-intrusive reduced order model with transformer neural network and its application
Wu, P., Qiu, F., Feng, W., Fang, F., & Pain, C. C. (2022) · Physics of Fluids · doi:10.1063/5.0123185 · Cited by 30
Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers
Wei, C., Duke, B., Jiang, R., Aarabi, P., Taylor, G. W., & Shkurti, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02172 · Cited by 32
Vigour testing for the rice seed with computer vision-based techniques
Qiao, J., Liao, Y., Yin, C., Yang, X., Tú, H. M., Wang, W., & Liu, Y. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1194701 · Cited by 29
Ge, J., Zhang, H., Zuo, L., Xu, L., Jiang, J., Song, M., Ding, Y., Xie, Y., Wu, F., Wang, C., et al. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.12.021 · Cited by 29
Torch-NILM: An Effective Deep Learning Toolkit for Non-Intrusive Load Monitoring in Pytorch
Gkalinikis, N. V., Nalmpantis, C., & Vrakas, D. (2022) · Energies · doi:10.3390/en15072647 · Cited by 29
Deep Residual Weight-Sharing Attention Network With Low-Rank Attention for Visual Question Answering
Qin, B., Hu, H. R., & Zhuang, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3173131 · Cited by 30
MBPTrack: Improving 3D Point Cloud Tracking with Memory networks and Box Priors
Xu, T.-X., Guo, Y.-C., Lai, Y., & Zhang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00909 · Cited by 32
MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images
Huang, J., Yu, H., Yu, K., Navab, N., Ilić, S., & Busam, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00962 · Cited by 37
STAGP: Spatio-Temporal Adaptive Graph Pooling Network for Pedestrian Trajectory Prediction
Liu, Z., He, L., Yuan, L., Lv, K., Zhong, R., & Yao-hua, C. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3346806 · Cited by 27
TransLoc4D: Transformer-Based 4D Radar Place Recognition
Peng, G., Li, H., Zhao, Y., Zhang, J., Wu, Z., Zheng, P., & Wang, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01666 · Cited by 27
Optimizing the Deployment of Tiny Transformers on Low-Power MCUs
Jung, V. J., Burrello, A., Scherer, M., Conti, F., & Benini, L. (2024) · IEEE transactions on computers · doi:10.1109/tc.2024.3500360 · Cited by 30
Neural Fourier Energy Disaggregation
Nalmpantis, C., Gkalinikis, N. V., & Vrakas, D. (2022) · Italian National Conference on Sensors · doi:10.3390/s22020473 · Cited by 22
Finetuning Pretrained Transformers into RNNs
Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.830 · Cited by 79
Self-citationGuide Local Feature Matching by Overlap Estimation
Chen, Y., Huang, D., Xu, S., Liu, J., & Liu, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19913 · Cited by 26
Yin, P., Zhang, D., Han, W., Li, J., & Cheng, J. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3214889 · Cited by 29
Graph Neural Networks for Natural Language Processing: A Survey
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000096 · Cited by 417
Hechen, Z., Huang, W., Yin, L., Xie, W., & Zhao, Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107228 · Cited by 27
Transformer-based sensor fusion for autonomous vehicles: A comprehensive review
Abdulmaksoud, A., & Ahmed, R. (2025) · IEEE Access · doi:10.1109/access.2025.3545032 · Cited by 25
TransVCL: Attention-Enhanced Video Copy Localization Network with Flexible Supervision
He, S., He, Y., Lu, M., Jiang, C., Yang, X., Qian, F., Zhang, X., Yang, L., & Zhang, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25158 · Cited by 34
Relative Positional Encoding for Transformers with Linear ComplexityPreprint
Antoine Liutkus, A., Cífka, O., Wu, S.-L., Şimşekli, U., Yang, Y., & Gaël Richard, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08399 · Cited by 28
Design in the DARK: Learning Deep Generative Models for De Novo Protein Design
Moffat, L., Kandathil, S. M., & Jones, D. T. (2022) · doi:10.1101/2022.01.27.478087 · Cited by 25
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24
“For example, the main difference between Transformers [1] and linear Transformers [7] is the memory structure as well as the memory updating step, in which linear Transformers compress the historical data into a fixed-size matrix-valued memory while Transformers keep all historical data (within the…”
A Novel SE-TCN-BiGRU Hybrid Network for Automatic Seizure Detection
Zhu, P., Zhou, W., Cao, C., Liu, G., Liu, Z., & Shang, W. (2024) · IEEE Access · doi:10.1109/access.2024.3406909 · Cited by 24
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Gao, J., Xu, H., Han, S., Ren, X., Yu, P. L. H., Liang, X., Jiang, X., & Li, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21311 · Cited by 32
Structure-Enhanced Pop Music Generation via Harmony-Aware Learning
Zhang, X., Zhang, J., Qiu, Y., Wang, L., & Zhou, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548084 · Cited by 24
Variational Regression for Multi-Target Energy Disaggregation
Gkalinikis, N. V., Nalmpantis, C., & Vrakas, D. (2023) · Italian National Conference on Sensors · doi:10.3390/s23042051 · Cited by 24
Hu, V. T., Zhang, D. W., Asano, Y. M., Burghouts, G. J., & Snoek, C. G. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01766 · Cited by 28
Ghosh, S., & Mitra, P. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107955 · Cited by 24
Understanding the architecture of vision transformer and its variants: A review
Neha, F., & Bansal, A. K. (2024) · 2024 1st International Conference on Innovative Engineering Sciences and Technological Research (ICIESTR) · doi:10.1109/iciestr60916.2024.10798341 · Cited by 29
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.346 · Cited by 27
Xiong, J., Wang, G., Zhang, P., Huang, W., Zha, Y., & Zhai, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00623 · Cited by 24
High-Resolution Traffic Demand Prediction, Based on Deep Learning and Hierarchical ReconciliationPreprint
Khalesian, M., Furno, A., & Leclercq, L. (2022) · SSRN Electronic Journal · doi:10.1016/j.trc.2023.104410 · Cited by 28
PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registration
Yuan, M., Fu, K., Li, Z., Meng, Y., & Wang, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01622 · Cited by 23
Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints
Cao, C., & Fu, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01114 · Cited by 24
Karmakar, P., Teng, S. W., & Lu, G. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200406 · Cited by 36
Zhang, X., Weng, Z., Zhu, P., Han, X., Zhu, J., & Jiao, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3351437 · Cited by 26
Qian, X., Wang, B., Chen, J., Fan, Y., Mo, R., Xu, C., Liu, W., Liu, J., & Zhong, P.-A. (2025) · Journal of Hydrology · doi:10.1016/j.jhydrol.2025.133968 · Cited by 28
Buchholz, T.-O., & Jug, F. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00201 · Cited by 24
Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders
Kong, X., Renduchintala, A., Cross, J., Tang, Y., Gu, J., & Li, X. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.138 · Cited by 24
Scaling Language Model Size in Cross-Device Federated Learning
Ro, J., Breiner, T., McConnaughey, L., Chen, M., Suresh, A., Kumar, S., & Mathews, R. (2022) · doi:10.18653/v1/2022.fl4nlp-1.2 · Cited by 35
Learning 3D Scene Priors with 2D Supervision
Nie, Y., Dai, A., Han, X., & NieBner, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00083 · Cited by 22
Scalable Visual Transformers with Hierarchical PoolingPreprint
Pan, Z., Zhuang, B., Liu, J., He, H., & Cai, J. (2021) · arXiv (Cornell University) · Cited by 22
When Meta-Learning Meets Online and Continual Learning: A Survey
Son, J., Lee, S., & Kim, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3463709 · Cited by 34
Gershman, S. J., Fiete, I., & Irie, K. (2025) · Neuron · doi:10.1016/j.neuron.2025.02.029 · Cited by 51
Shao, K., Li, D., Tang, H., Zhang, Y., Xu, B., & Bhatti, U. A. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110329 · Cited by 30
Maiti, S., Erdoğan, H., Wilson, K., Wisdom, S., Watanabe, S., & Hershey, J. R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414841 · Cited by 21
Semi-Dense Feature Matching With Transformers and its Applications in Multiple-View Geometry
Shen, Z., Sun, J., Wang, Y., He, X. H., Bao, H., & Zhou, X.-W. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3223530 · Cited by 24
Long-Term Visual Localization with Mobile Sensors
Yan, S., Liu, Y., Wang, L., Shen, Z., Peng, Z., Liu, H.-M., Zhang, M.-J., Zhang, G., & Zhou, X.-W. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01654 · Cited by 25
PoseMatcher: One-shot 6D Object Pose Estimation by Deep Feature Matching
Castro, P., & Kim, T. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00229 · Cited by 21
Modeling Spatio-Temporal Mobility Across Data Silos via Personalized Federated Learning
Zhang, Y., Wang, X., Wang, P., Wang, B., Zhou, Z., & Wang, Y. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3453657 · Cited by 22
Centroid Transformers: Learning to Abstract with AttentionPreprint
Wu, L., Liu, X., & Liu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08606 · Cited by 28
Transformers: "The End of History" for NLP?Preprint
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · ECML/PKDD · doi:10.48550/arxiv.2105.00813 · Cited by 36
Neuro-Symbolic AI: An Emerging Class of AI Workloads and their CharacterizationPreprint
Susskind, Z., Arden, B., John, L. K., Stockton, P., & John, E. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06133 · Cited by 21
Multi-View Stereo with TransformerPreprint
Zhu, J., Peng, B., Li, W., Shen, H., Zhang, Z., & Lei, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00336 · Cited by 28
Latent Semantic and Disentangled Attention
Chien, J., & Huang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3432631 · Cited by 20
Khambhawala, A., Lee, C. H., & Kwon, J. S. (2025) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.5c01387 · Cited by 21
Pérez, J., Barceló, P., & Marinković, J. (2021) · Journal of Machine Learning Research · Cited by 20
EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation
Shi, Y., Cai, H., Ansari, A., & Porikli, F. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00017 · Cited by 20
HECPG: Hyperbolic Embedding and Confident Patch-Guided Network for Point Cloud Matching
Xie, Y., Zhu, J., Li, S., Hu, N., & Shi, P. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3370591 · Cited by 20
Zhang, Y., Zou, Y., & Liu, P. X. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3457228 · Cited by 20
Phyloformer: towards fast and accurate phylogeny estimation with self-attention networksPreprint
Nesterenko, L., Boussau, B., & Jacob, L. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.06.24.496975 · Cited by 20
How generative Artificial Intelligence can transform drug discovery?
Jusoh, A. S., Remli, M. A., Mohamad, M. S., Cazenave, T., & Fong, C. (2025) · European Journal of Medicinal Chemistry · doi:10.1016/j.ejmech.2025.117825 · Cited by 22
Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences
Ibing, M., Kobsik, G., & Kobbelt, L. P. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00270 · Cited by 30
UCONV-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognition
Andrusenko, A., Nasretdinov, R., & Romanenko, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095430 · Cited by 19
Deep Learning Algorithms for Pancreas Segmentation from Radiology Scans
Zhang, Z., Yao, L., Keleş, E., Velichko, Y., & Bağcı, U. (2023) · Advances in Clinical Radiology · doi:10.1016/j.yacr.2023.05.001 · Cited by 22
Physics-informed few-shot deep learning for elastoplastic constitutive relationships
Wang, C., He, Y.-Q., Lu, H., Nie, J., & Fan, J. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106907 · Cited by 19
Dai, K., Xie, T., Wang, K., Jiang, Z., Li, R., & Zhao, L. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110094 · Cited by 33
Weld TOFD defect classification method based on multi-scale CNN and cascaded focused attention
Tang, D., Zhang, J., Wang, P., & He, Y. (2025) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2025.02.019 · Cited by 19
Group Equivariant Stand-Alone Self-Attention For VisionPreprint
Romero, D. W., & Cordonnier, J.-B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00977 · Cited by 44
Diffusion Model-Based Path Follower for a Salamander-Like Robot
Liu, Z., Liu, Y., & Fang, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3549307 · Cited by 19
FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction
Yang, Z., Ren, Z., Bautista, M. Á., Zhang, Z., Shan, Q., & Qi-xing Huang, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00253 · Cited by 18
A Survey on Sparsity Exploration in Transformer-Based Accelerators
Fuad, K. A. A., & Chen, L. (2023) · Electronics · doi:10.3390/electronics12102299 · Cited by 21
SAM-Net: Self-Attention based Feature Matching with Spatial Transformers and Knowledge Distillation
Kelényi, B., Domșa, V., & Tamás, L. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122804 · Cited by 18
A Lightweight Pine Wilt Disease Detection Method Based on Vision Transformer-Enhanced YOLO
Yuan, Q., Suhua, Z., Wang, H., Luo, W., Zheng, X., Liu, L., & Meng, Z. (2024) · Forests · doi:10.3390/f15061050 · Cited by 18
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.331 · Cited by 21
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency EncodingPreprint
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06097 · Cited by 27
STAR: Sparse Transformer-based Action RecognitionPreprint
Shi, F., Lee, C., Qiu, L., Zhao, Y., Shen, T., Muralidhar, S., Han, T., Zhu, S. C., & Narayanan, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07089 · Cited by 32
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v1 · Cited by 24
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
“…LLaMA (Touvron et al., 2023), and Chinchilla (Hoffmann et al., 2022) Table 1: Complexity comparison with different Trans-formers: Reformer (Kitaev et al., 2020), Linear Trans-former (Katharopoulos et al., 2020), Performer (Choro-manski et al., 2020), AFT (Zhai et al., 2021), MEGA (Ma et al., 2023).”
BackgroundInfluentialJia, H., Hou, P., Zhou, Y. Z., Zhu, H., & Chai, H. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103930 · Cited by 17
CCDFormer: A dual-backbone complex crack detection network with transformer
Hu, X., Li, H., Feng, Y., Qian, S., Li, J., & Li, S. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111251 · Cited by 17
Unified Spatial-Frequency Modeling and Alignment for Multi-Scale Small Object Detection
Liu, J., Wang, Y., Cao, Y., Guo, C., Shi, P., & Li, P. (2025) · Symmetry · doi:10.3390/sym17020242 · Cited by 19
Controlling Perceived Emotion in Symbolic Music Generation with Monte Carlo Tree Search
Ferreira, L. N., Mou, L., Whitehead, J., & Lelis, L. H. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v18i1.21960 · Cited by 17
An accelerated end-to-end method for solving routing problems
Zhu, T., Shi, X., Xu, X., & Cao, J. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.05.003 · Cited by 17
Zhang, Y., Chen, D., Kundu, S., Li, C., & Beerel, P. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00472 · Cited by 20
Sun, J., Chen, B., Lu, R., Cheng, Z., Qu, C., & Yuan, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3400809 · Cited by 18
Khanduja, N., Kumar, N., & Chauhan, A. (2024) · Systems and Soft Computing · doi:10.1016/j.sasc.2024.200112 · Cited by 22
SEQ2SEQ++: A Multitasking-Based Seq2seq Model to Generate Meaningful and Relevant Answers
Palasundram, K., Sharef, N. M., Kasmiran, K. A., & Azman, A. (2021) · IEEE Access · doi:10.1109/access.2021.3133495 · Cited by 18
Chess as a Testbed for Language Model State Tracking
Toshniwal, S., Wiseman, S., Livescu, K., & Gimpel, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21390 · Cited by 42
Fernandez-Zelaia, P., Dryepondt, S. N., Ziabari, A., & Kirka, M. M. (2023) · Computational Materials Science · doi:10.1016/j.commatsci.2023.112603 · Cited by 18
Deformable medical image registration based on wavelet transform and linear attention
Li, W., gan, K., Yang, L., & Zhang, Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106413 · Cited by 16
PERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Tang, Z., Cho, J., Lei, J., & Bansal, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00439 · Cited by 17
Guo, Z., Kang, J., & Herremans, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25635 · Cited by 24
Patch-guided point matching for point cloud registration with low overlap
Zhao, T., Li, L., Tian, T., Ma, J., & Tian, J. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109876 · Cited by 16
Structured Epipolar Matcher for Local Feature Matching
Chang, J., Yu, J., & Zhang, T. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00657 · Cited by 16
Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers
Dutson, M., Li, Y., & Gupta, M. (2023) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51070.2023.01551 · Cited by 28
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
