The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 701–800 of 855
Kerui, 王. W., Chuncheng, 周. Z., Yi, 马. M., Lincong, 彭. P., Hao, 周. Z., & Pengfei, 余. Y. (2025) · Laser & Optoelectronics Progress · doi:10.3788/lop241686 · Cited by 1
Drop-in efficient self-attention approximation method
François, D., Saillot, M., Klein, J., Bissyandé, T. F., & Skupin, A. (2025) · Machine Learning · doi:10.1007/s10994-025-06768-3 · Cited by 2
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning
Hao, Z., Guo, J., Shen, L., Luo, Y., Hu, H., & Wen, Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02440-4 · Cited by 5
Pei, J., Wang, Y., Ge, M., Li, J., Li, Y., Wang, W., & Zhou, X. (2025) · Bioengineering · doi:10.3390/bioengineering12050550 · Cited by 1
He, Q., & Sun, T. (2025) · IET Image Processing · doi:10.1049/ipr2.70055 · Cited by 1
Lightweight Structure-Aware Attention for Visual Understanding
Kwon, H., Castro, F. M., Marín-Jiménez, M. J., Guil, N., & Alahari, K. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02475-7 · Cited by 1
Progressive Two-Stage Decoder Segment Anything Model for Polyp Segmentation
Xu, J., & Zhao, Y. (2025) · International Conference on Computer, Artificial Intelligence and Control Engineering (CAICE) · doi:10.1145/3727648.3727777 · Cited by 1
Graph Anomaly Detection via Multi-Level Information Alignment and Decoupling
Wu, W., & Gu, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114045 · Cited by 1
Unpaired to paired data synthesis and generative ensemble network for smudged image restoration
Feng, C.-B., Liu, K., Sun, J., Lai, Q., Jin, J., Su, H., Wang, G., & Vong, C. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113648 · Cited by 2
Reunamo, A., Moen, H., Salanterä, S., & Lähteenmäki, P. M. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1585309 · Cited by 1
Static-dynamic class-level perception consistency in video semantic segmentation
Cen, Z., Guo, N., Xu, W., Feng, Z., & Huang, D. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107953 · Cited by 1
Pan, X., Wang, S., Li, G., Zhu, J., Xu, Y., Zhang, J., & Zhou, W. Y. (2025) · Optics Express · doi:10.1364/oe.580821 · Cited by 1
Fast streaming translation using machine learning with transformer
Qiu, J., Moh, M., & Moh, T.-S. (2021) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/3409334.3452059 · Cited by 1
Acronyms and Opportunities for Improving Deep Nets
Church, K., & Liu, B. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.732381 · Cited by 1
A Cooperative Lightweight Translation Algorithm Combined with Sparse-ReLU
Xu, X., Liu, Y., Chen, G., Ye, J., Li, Z., & Lu, H. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/4398839 · Cited by 1
LDAM: line descriptors augmented by attention mechanism
Cao, X., Huang, Y., Huang, Y., Li, Y., & Cai, S. (2022) · International Conference on Digital Image Processing · doi:10.1117/12.2644245 · Cited by 1
Selective Memory-Augmented Document Translation with Diverse Global Context
Zhang, X., Li, J., & Liu, M. (2022) · International Conference on Big Data Computing and Communications (BigCom) · doi:10.1109/bigcom57025.2022.00030 · Cited by 1
The reusability prior: comparing deep learning models without training
Polat, A. G., & Alpaslan, F. N. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acc713 · Cited by 1
A Continuous Encoder-Decoder Method for Spatial-Temporal Forecasting
Liu, L., Shen, Y., & Qi, H. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads56603.2022.00113 · Cited by 1
DGN: Descriptor Generation Network for Feature Matching in Monocular Endoscopy 3D Reconstruction
Zhang, K., Fan, W., Chen, Y., & Luó, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096371 · Cited by 1
Efficient Non-Sampling Graph Neural Networks
Ji, J., Li, Z., Xu, S., Ge, Y., Tan, J., & Zhang, Y. (2023) · Information · doi:10.3390/info14080424 · Cited by 1
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights
Mao, H. H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.697 · Cited by 16
Improving Autoregressive NLP Tasks via Modular Linearized Attention
Agostinelli, V., & Chen, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43421-1_6 · Cited by 1
Accelerating Toeplitz Neural Network with Constant-time Inference Complexity
Qin, Z., & Zhong, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.750 · Cited by 3
Cascade Cost Volume Multi-View Stereo Network with Transformer and Pseudo 3D
Qu, J., Zhao, S., Liang, H., Zhang, Q., Li, T., & Liu, B. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10393903 · Cited by 1
Arabic text sentiment analysis and emotion classification using transformers
Mossad, N., Mohamed, Y., Fares, A. H., & Zaky, A. B. (2023) · International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC) · doi:10.1109/jac-ecc61002.2023.10479609 · Cited by 3
Zhang, X., Dong, T., Yan, L., Yang, Z., & Zhang, J. (2024) · doi:10.1145/3653804.3656278 · Cited by 1
Shen, Z., Lu, X., Wang, Y., & Nie, J. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3433889 · Cited by 1
MAS-Net: Multi-Attention Hybrid Network for Superpixel Segmentation
Yan, G., Wei, C., Jia, X., Li, Y., & Chang, W. (2024) · Symmetry · doi:10.3390/sym16081000 · Cited by 1
GENPIA: A Genre-Conditioned Piano Music Generation System
Nguyen, Q.-V., Lai, H.-W., Nguyen, K.-D., Sun, M.-T., Hwang, W., Sakai, K., & Ku, W. (2024) · IEEE International Symposium on the Internet of Sounds (IS2) · doi:10.1109/is262782.2024.10704094 · Cited by 1
ACT-FRCNN: Progress Towards Transformer-Based Object Detection
Zulfqar, S., Elgamal, Z. M., Zia, M. A., Razzaq, A., Ullah, S., & Dawood, H. (2024) · Algorithms · doi:10.3390/a17110475 · Cited by 1
Park, T., & Kim, H. (2024) · Mathematics · doi:10.3390/math12213390 · Cited by 2
Zhao, M., Li, Y., Zhang, C., Du, Q., & Peng, S. (2024) · Electronics · doi:10.3390/electronics13234756 · Cited by 1
Enriching visual feature representations for vision–language tasks using spectral transforms
Ondeng, O., Absaloms, H. O., & Akuon, P. O. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105390 · Cited by 1
Optimal RoPE extension via Bayesian Optimization for training-free length generalization
Zhang, X., Hu, S., Zhao, W., Wang, H., Han, X., He, C., Zeng, G., Liu, Z., & Sun, M. (2025) · AI Open · doi:10.1016/j.aiopen.2025.01.002 · Cited by 1
EfficientViT for Video Action Recognition
Jin, D., & Zeng, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825506 · Cited by 1
Sun, Y. T., Fu, X., Zhang, Z., Zhang, N., Zhang, H., & Cao, Y. (2024) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat62293.2024.00129 · Cited by 1
Spatio-Temporal Weighted Graph Reason Learning for Multivariate Time-Series Anomaly Detection
Zhang, H., Pang, H., Zhao, Y., Gao, G., Song, Y., Wei, S., & Zhao, Y. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3569316 · Cited by 3
Linear Transformers beat YOLO for Embedded Object Detection
Ancilotto, A., Castagnini, E., & Farella, E. (2025) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops65533.2025.00071 · Cited by 1
Dual‐Domain Fusion Attention Network for Remote Sensing Image Dehazing
Qiu, S., Dun, Y., Yao, B., Zhang, D., Ma, M., & Li, Q. (2025) · The Photogrammetric Record · doi:10.1111/phor.70018 · Cited by 1
Multi-Stage Feature Interaction Model with Abundant Semantic Information for Image Captioning
Li, X., An, G., & Ruan, Q. (2020) · International Conference on Signal Processing · doi:10.1109/icsp48669.2020.9320971 · Cited by 1
Adaptive Multi-Resolution Attention with Linear Complexity
Zhang, Y., Ma, Y., Seidl, T., & Tresp, V. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191567 · Cited by 1
Deep Place Recognition for 3D LiDAR Point Cloud with Transformer
Li, C., Zhang, X., Yan, F., & Zhuang, Y. (2021) · China Automation Congress (CAC) · doi:10.1109/cac53003.2021.9728443 · Cited by 1
Chakrabarti, K., & Chopra, N. (2022) · Indian Control Conference · doi:10.1109/icc56513.2022.10093397 · Cited by 2
S2H-GNN: Learning Soft to Hard Feature Matching with Sparsified Graph Neural Network
Xie, T., Li, R., Jiang, Z., Dai, K., Wang, K., Zhao, L., & Li, S. (2023) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar58764.2023.10249019 · Cited by 1
Transformer With Linear-Window Attention for Feature Matching
Shen, Z., Kong, B., & Dong, X. (2023) · IEEE Access · doi:10.1109/access.2023.3328855 · Cited by 1
Translational Equivariance in Kernelizable AttentionPreprint
Horn, M., Shridhar, K., Groenewald, E., & Baumann, P. F. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07680 · Cited by 3
Beyond Nyströmformer -- Approximation of self-attention by Spectral ShiftingPreprint
Verma, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05638 · Cited by 1
Learning Chess Blindfolded: Evaluating Language Models on State Tracking.Preprint
Toshniwal, S., Wiseman, S., Livescu, K., & Gimpel, K. (2021) · arXiv (Cornell University) · Cited by 1
Ripple Attention for Visual Perception with Sub-quadratic ComplexityPreprint
Lin, Z., Pan, H., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02453 · Cited by 2
Transformer with a Mixture of Gaussian Keys.Preprint
Nguyen, T., Nguyen, T. M., Le, D. D., Nguyen, K. C. D., Tran, A. N., Baraniuk, R. G., Ho, N., & Osher, S. (2021) · arXiv (Cornell University) · Cited by 1
StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential DataPreprint
Pellegrain, V., Tami, M., Batteux, M., & Hudelot, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08021 · Cited by 2
Singing Beat Tracking With Self-supervised Front-end and Linear TransformersPreprint
Heydari, M., & Duan, Z. (2022) · doi:10.36227/techrxiv.20743012 · Cited by 1
Pano-Matching: Panoramic Local Feature Matching Using Interconnected Dynamic TransformerPreprint
Li, T., Xue, Y., Xia, R., & Cui, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4457478 · Cited by 1
ASFNet: Audio Spectrogram Fourier Network for Efficient Medical Sound Event DetectionPreprint
Hassan, K., & Haque, M. A. (2023) · doi:10.36227/techrxiv.23732205 · Cited by 1
Vinuesa, R., Sanchís-Agudo, M., Wang, Y., Guastoni, L., & Duraisamy, K. (2023) · Research Square · doi:10.21203/rs.3.rs-3545247/v1 · Cited by 1
Employing Transformers and emoji to perform sentiment classification of social media texts
de Barros, T. M. (2021) · doi:10.47749/t/unicamp.2021.1165392 · Cited by 1
From Transformers to the Future: An In-Depth Exploration of Modern Language Model ArchitecturesPreprint
Xu, H., Bi, Z., Tseng, H.-M., Song, X., & Feng, P. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/n8r5j · Cited by 1
Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed TransformerPreprint
Yang, Y. Y., Lian, J., & Li, L. (2025) · Qeios · doi:10.32388/9qh8rx · Cited by 1
Zhang, J., Nolte, N., Sadhukhan, R., Chen, B., & Bottou, L. (2024) · arXiv · doi:10.48550/arxiv.2405.06394 · Cited by 10
“Several recent papers (e.g., Katharopoulos et al., 2020; Peng et al., 2023; Sun et al., 2023; Gu & Dao, 2023) propose transformer alternatives that use efficient recurrences to cut the quadratic computational cost of transformers.”
xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Beck, M., Pöppel, K., Lippe, P., Kurle, R., Blies, P. M., Klambauer, G., Böck, S., & Hochreiter, S. (2025) · arXiv · doi:10.48550/arxiv.2503.13427 · Cited by 9
“In contrast, Eagle-7B (Peng et al., 2024) builds on the RWKV architecture and uses a matrix-valued hidden state similar to linear attention and gated linear attention (Katharopoulos et al., 2020; Yang et al., 2024b).”
End-to-End Test-Time Training for Long Context
Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., et al. (2025) · arXiv
“In particular, [79] (Schmidhuber, 1992) has been a major source of inspiration for modern RNN layers, such as linear attention [55, 77], DeltaNet [76, 106], and Gated DeltaNet [105], one of our baselines.”
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“It replaces the explicit attention matrix with a fixed-size recurrent state, turning sequence mixing into a linear-time recurrence whose memory does not grow with context length [1].”
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
Pandey, V., & Singh, G. (2026) · arXiv
“Linear attention [6] removes the quadratic bottleneck but introduces a different failure mode: the internal memory state grows without bound, producing a Frobenius norm that scales as O ( T ) and degrades associative retrieval accuracy at long range.”
InfluentialBehrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2504.13173 · Cited by 65
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs [54].”
InfluentialContinuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
Li, H., He, R., Mang, Q., Zhang, Q., Mao, H., Chen, X., Zhou, H., Zhang, H., Cheung, A., Gonzalez, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.02230
ATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
“Consequently, there’s growing interest in enhancing RNN memory management through more effective: (i) Learning rules, from additive learning (Katharopoulos et al. 2020) to DeltaNet’s Delta rule (Schlag et al. 2021); (ii) Forget (Retention) Gates, from RetNet’s input-independent gating (Sun, Dong,…”
Learning Hard Retrieval Cross Attention for Transformer.Preprint
Xu, H., & Liu, Q. (2020) · arXiv (Cornell University)
Higher Order Linear TransformerPreprint
Mercat, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14816
CoRe: An Efficient Coarse-refined Training Framework for BERTPreprint
Yang, C. H., Wang, S., Li, Y., Yang, C., Yan, M., Zhang, J., & Lin, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13633 · Cited by 1
Decision Machines: Congruent Decision TreesPreprint
Zhang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11347
Benchmarking a transformer-FREE model for ad-hoc retrieval
de Almeida, T. D., & Matos, S. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.293
Transfer training from smaller language modelPreprint
Zhang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11390
T-EMDE: Sketching-based global similarity for cross-modal retrievalPreprint
Rychalska, B., Wieczorek, M., & Dabrowski, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04242
Dube, S. (2021) · Springer eBooks · doi:10.1007/978-3-030-68624-6_10
Self-Attention Network for Text Representation Learning
Shen, T. (2021) · UTS ePRESS (University of Technology Sydney)
Weight Squeezing: Reparameterization for Knowledge Transfer and Model CompressionPreprint
Chumachenko, A., Gavrilov, D., Balagansky, N., & Kalaidin, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06993 · Cited by 1
Learning Hard Retrieval Decoder Attention for Transformers
Xu, H., Liu, Q., van Genabith, J., & Xiong, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.67
DCT: Dynamic Compressive Transformer for Modeling Unbounded SequencePreprint
Chang, K.-P., & Ma, W.-Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04821
PermuteFormer: Efficient Relative Position Encoding for Long SequencesPreprint
Chen, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02377
Sparse Attention with Linear Units
Zhang, B., Titov, I., & Sennrich, R. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-208886
Frame Attention Mechanism and Transfer Learning Network for Classification on Motion Video
Liu, K., Zhou, G., & Cheng, X. F. (2022) · Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC) · doi:10.1109/ipec54454.2022.9777334
Contextualizer: Connecting the Dots of Context with Second-Order Attention
Maupomé, D., & Meurs, M. (2022) · Information · doi:10.3390/info13060290
Efficient Transformers for End-to-End Neural Speaker Diarization
del Alamo, S. I., Labrador, B., Lozano-Díez, A., & Toledano, D. T. (2022) · IberSPEECH · doi:10.21437/iberspeech.2022-32
Optimized parallel causal product for linear transformer
Liu, S. (2023) · International Conference on Artificial Intelligence, Automation and High Performance Computing (AIAHPC) · doi:10.1117/12.2684707
ASFNet: Audio Spectrogram Fourier Network for Efficient Medical Sound Event DetectionPreprint
HASSAN, K. M. N., & Haque, M. A. (2023) · doi:10.36227/techrxiv.23732205.v1
Fixed Input Parameterization for Efficient Prompting
Choi, E., Jo, Y., Jang, J., Jang, J., & Seo, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.533 · Cited by 1
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference
Li, H., Keung, P., Cheng, D., Kasai, J., & Smith, N. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.146 · Cited by 1
Pre-trained Models for Representation Learning
Lin, Y., Ding, N., Liu, Z., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_5 · Cited by 2
An Underwater Features Matching System Integrated with Image Enhancement
Zhang, S., & Yu, Z. (2023) · IEEE International Conference on Electronic Information and Communication Technology (ICEICT) · doi:10.1109/iceict57916.2023.10245163
Seq2Space: A Lightweight Solution for Sequence Modelling
Rixen, J., & Renz, M. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230488
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding
Sengupta, A., Akhtar, M., & Chakraborty, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.228
Xue, H., & Αλέτρας, Ν. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.695
The MixGauss AttentionPreprint
Bloch, A., & Bloch, D. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4685416
Cap-CasMVSNet:Capsul-based Cascade Cost Volume for Multi-View Stereo Network
Zhou, H., & Li, Y. (2023) · Youth Academic Annual Conference of Chinese Association of Automation (YAC) · doi:10.1109/yac59482.2023.10401477
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
