The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 601–700 of 855
Glassoformer: A Query-Sparse Transformer for Post-Fault Power Grid Voltage Prediction
Zheng, Y., Hu, C., Lin, G., Yue, M., Wang, B., & Xin, J. X. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747394 · Cited by 3
Liu, Q., Dou, F., Yang, M., Amdework, E., Wang, G., & Bi, J. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/676 · Cited by 3
Spatial constraint for efficient semi-supervised video object segmentation
Chen, Y., Ji, C., Yang, Z.-X., & Wu, E. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103843 · Cited by 3
Segmenting Urban Scene Imagery in Real Time Using an Efficient UNet-like Transformer
Xu, H., Yu, M., Zhou, F., & Yin, H. (2024) · Applied Sciences · doi:10.3390/app14051986 · Cited by 3
TD-GPT: Target Protein-Specific Drug Molecule Generation GPT
He, Z., Chen, L., Xu, J., Lv, H., Zhou, R.-N., Hu, J., Chen, Y., & Gao, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447303 · Cited by 3
CSFormer: Cross-Scale Transformer for Feature Matching
Zhou, Y., Cheng, X., Zhai, X., Lei, X., & Ду, С. (2023) · International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD) · doi:10.1109/icsmd60522.2023.10490714 · Cited by 3
Joint features-guided linear transformer and CNN for efficient image super-resolution
Wang, B., Zhang, Y., Long, W., & Cui, Z. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02277-2 · Cited by 3
Feature matching based on local windows aggregation
Guo, Y. C., Li, W., Zhai, P., & Wu, L. (2024) · iScience · doi:10.1016/j.isci.2024.110825 · Cited by 3
GeoMamba: Toward Efficient Geography-Aware Sequential POI Recommendation
Chen, J., Wang, H., & Shang, J. (2024) · IEEE Access · doi:10.1109/access.2024.3482116 · Cited by 3
LGSNet: Local-Global Semantics Learning Object Detection
Li, Y., Jiao, L., Liu, X., Liu, F., Li, L., & Chen, P. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521850 · Cited by 3
Pu, X., Luo, C., Yin, Y., & Luo, Y. (2024) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc62609.2024.10941971 · Cited by 3
Bandit-based Attention Mechanism in Vision Transformers
Chowdhury, A. R., Diddigi, R. B., Prabuchandran, K. J., & Tripathi, A. M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00929 · Cited by 3
ReBotNet: Fast Real-Time Video Enhancement
Valanarasu, J. M. J., Garg, R. D., Toor, A., Tong, X., Xi, W., Lugmayr, A., Patel, V. M., & Menini, A. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00146 · Cited by 3
Retentive neural quantum states: efficient ansätze for ab initio quantum chemistry
Knitter, O., Zhao, D., STOKES, J. M., Ganahl, M., Leichenauer, S., & Veerapaneni, S. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adcb88 · Cited by 3
IBSS: Intelligent Behavior Sensing System Based on Comparative Language-CSI Learning
Zhang, M., Li, J., Liang, J., Liao, S., Tang, C., Liu, J., & Soekmadji, H. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3572889 · Cited by 3
Improving Fastspeech TTS with Efficient Self-Attention and Compact Feed-Forward Network
Xiao, Y., Wang, X., He, L., & Soong, F. K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746408 · Cited by 3
ABC: Attention with Bounded-memory Control
Peng, H., Kasai, J., Pappas, N., Yogatama, D., Wu, Z., Kong, L., Schwartz, R., & Smith, N. A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.515 · Cited by 11
Self-citationAAN+: Generalized Average Attention Network for Accelerating Neural Transformer
Zhang, B., Xiong, D., Ge, Y., Yao, J., Yue, H., & Su, J. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13896 · Cited by 3
FGCNet: Fast Graph Convolution for Matching Features
Liu, L., Pan, L., Luo, W., Xu, Q., Wen, Y., & Li, J. (2022) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct57072.2022.00096 · Cited by 3
The Role of Global and Local Context in Named Entity Recognition
Amalvy, A., Labatut, V., & Dufour, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.62 · Cited by 4
TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language ModelingPreprint
Zhang, S., Zhang, P., Ma, X., Wei, V. J., Wang, N., & Liu, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01547 · Cited by 3
On the Regularity of AttentionPreprint
Vuckovic, J., Baratin, A., & Combes, R. T. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.05628 · Cited by 15
Combining Frame-Synchronous and Label-Synchronous Systems for Speech RecognitionPreprint
Li, Q., Zhang, C., & Woodland, P. C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4194372 · Cited by 3
Fastformer: Additive Attention is All You NeedPreprint
Wu, C., Wu, F., Qi, T., & Huang, Y. (2021) · arXiv (Cornell University) · Cited by 3
Singing Beat Tracking With Self-supervised Front-end and Linear TransformersPreprint
Heydari, M., & Duan, Z. (2022) · doi:10.36227/techrxiv.20743012.v1 · Cited by 3
Ni, J. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/168430 · Cited by 3
Rough Transformers: Lightweight Continuous-Time Sequence Modelling with Path SignaturesPreprint
Moreno-Pino, F., Arroyo, Á., Waldon, H., Dong, X., & Cartea, Á. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4850217 · Cited by 3
SC-MAMBA2: Leveraging State-Space Models for Efficient Single-Cell Ultra-Long Transcriptome ModelingPreprint
Zhao, Y., Zhao, B., Zhang, F., He, C., Wu, W., & Lai, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.30.615775 · Cited by 3
Zhang, C., & Duc, K. D. (2025) · Qeios · doi:10.32388/adif5a · Cited by 3
Li, J. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100163 · Cited by 3
S$^\text{3}$Attention: Improving Long Sequence Attention With Smoothed Skeleton Sketching
Wang, X., Zhou, T., Zhu, J., Liu, J., Yuan, K., Yao, T., Yin, W., Jin, R., & Cai, H. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3446173 · Cited by 2
Zhang, K., Zhang, Y., Wu, J., Wang, T., Jiang, W., Zeng, M., & Yang, Z. (2024) · Chemosensors · doi:10.3390/chemosensors12090172 · Cited by 2
ICViT: Integrated Framework for Complexity Reduction in Vision Transformer
Dang, T., Nguyen, K.-D., & Nguyen, K. (2024) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr63514.2024.10660940 · Cited by 2
SlowFormer: Adversarial Attack on Compute and Energy Consumption of Efficient Vision Transformers
Navaneet, K. L., Koohpayegani, S. A., Sleiman, E., & Pirsiavash, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02341 · Cited by 3
MVP: One-Shot Object Pose Estimation by Matching With Visible Points
Cheng, W., & Luo, M. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3472492 · Cited by 2
Intelligent Fish Detection System with Similarity-Aware Transformer
Li, S., Zuo, H., Fu, C., Wang, Z., & Xu, Z. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802394 · Cited by 2
Domínguez-Vidal, J. E., & Sanfeliu, A. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801981 · Cited by 2
An Analysis of Linear Complexity Attention Substitutes With Best-RQ
Whetten, R., Parcollet, T., Moumen, A., Dinarelli, M., & Estève, Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832323 · Cited by 3
DarkMatcher: an efficient self-supervised network for image matching in extremely dark environments
Zhang, J., Wang, P., Li, Y., Li, L., Song, Q., He, R., & Fu, L. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/adb206 · Cited by 2
Latency Adjustable Transformer Encoder for Language Understanding
Kachuee, S., & Sharifkhani, M. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3539727 · Cited by 4
Multi-View Stereo Using Perspective-Aware Features and Metadata to Improve Cost Volume
Zuo, Z., Li, Y., Zhou, Y., & Mo, F. (2025) · Sensors · doi:10.3390/s25072233 · Cited by 2
Mamba-Hybrid Language Models: Advancing Detection of AI-Generated Text in Academic Contexts
Prajapati, M., Baliarsingh, S. K., Das, S., Dash, A. K., Sarkar, A., & Pradhan, R. (2025) · International Conference on Emerging Systems and Intelligent Computing (ESIC) · doi:10.1109/esic64052.2025.10962699 · Cited by 3
Tingxu, 魏. W., Ying, 陈. C., Chenghao, 李. L., & Wenhao, 马. M. (2025) · Laser & Optoelectronics Progress · doi:10.3788/lop242348 · Cited by 2
Leveraging negative correlation for Full-Range Self-Attention in Vision Transformers
Long, W., Chen, Z., Li, W., Zhang, Y., Yao, H., Peng, J., & Cui, Z. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111899 · Cited by 2
Beyond KV caching: Shared attention for efficient LLMs
Liao, B., & Vargas, D. V. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130587 · Cited by 2
Blending 3D geometry and machine learning for multi-view stereopsis
Vats, V., Reza, M. A., Crandall, D., & Jung, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131250 · Cited by 2
Graph-Agnostic Linear Transformers
Guo, Z., Liu, Y., Ao, X., Tang, Y., Chen, X., Zheng, X., & He, Q. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108595 · Cited by 2
A Fast and Lightweight Speech Synthesis Model based on FastSpeech2
Nguyen, H.-K., Jeong, K., & Kang, H.-G. (2021) · International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC) · doi:10.1109/itc-cscc52171.2021.9501479 · Cited by 2
KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE Using Mel-Spectrograms
Liao, C.-F., Liu, J.-Y., & Yang, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747441 · Cited by 2
Skill Matching for Migrant Guidance Based on AI Tools
Uribe, S., Hernández, G., Belmonte-Hernández, A., & MARTIN, D. J. (2022) · Security informatics and law enforcement · doi:10.1007/978-3-030-93266-4_12 · Cited by 2
Relation is an option for processing context information
Yamada, K. D., Baladram, M. S., & Lin, F. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.924688 · Cited by 4
Development of a Weather Prediction Device Using Transformer Models and IoT Techniques
Olarewaju, I. K., & Kim, K. K. (2023) · Journal of Sensor Science and Technology · doi:10.46670/jsst.2023.32.3.164 · Cited by 2
Semantic‐guided fusion for multiple object tracking and RGB‐T tracking
Liu, X., Luo, Y., Zhang, Y., & Lei, Z. (2023) · IET Image Processing · doi:10.1049/ipr2.12861 · Cited by 2
CAHIR: Co-Attentive Hierarchical Image Representations for Visual Place Recognition
Peng, G., Li, H., Huang, Y., Zhang, J., Wen, M., Rahul, S., & Wang, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160512 · Cited by 2
Hybrid and Collaborative Passage Reranking
Zhang, Z., Zhou, W., Shi, J., & Li, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.880 · Cited by 3
KDLGT: A Linear Graph Transformer Framework via Kernel Decomposition Approach
Wu, Y. J., Xu, Y., Zhu, W., Song, G., Lin, Z., Wang, L., & Liu, S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/263 · Cited by 2
Rainfall-runoff prediction based on multi-modal data fusion
Wu, S., & Li, X. (2023) · International Conference on Algorithms, High Performance Computing and Artificial Intelligence (AHPCAI) · doi:10.1117/12.3011460 · Cited by 2
Sparse Transformer-Based Algorithm for Long-Short Temporal Association Action Recognition
Lu, Y., & Yang, Y. (2023) · International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/cost60524.2023.00027 · Cited by 2
Learning Counterfactual Explanation of Graph Neural Networks via Generative Flow Network
He, K., Liu, L., Zhang, Y., Wang, Y., Liu, Q., & Wang, G. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3387406 · Cited by 2
Benchmarking Inference of Transformer-Based Transcription Models With Clustering on Embedded GPUs
Schubert, M. E., Langerman, D., & George, A. D. (2024) · IEEE Access · doi:10.1109/access.2024.3426471 · Cited by 3
COLAFormer: Communicating local–global features with linear computational complexity
Miao, Z., Luo, H., Li, M., & Zhang, J. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110870 · Cited by 2
Fused Transformers: Fused Information of Arabic long Article for Summarization
Ezzat, Z., Khalfallah, A., & Khoriba, G. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.10.182 · Cited by 2
AE-AMT: Attribute-Enhanced Affective Music Generation With Compound Word Representation
Yao, W., Chen, C. L. P., Zhang, Z., & Zhang, T. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3486536 · Cited by 2
Xu, X., Sun, C., Qi, Z., & Chen, X. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774381 · Cited by 2
Trimformer: A Novel Sequence Compression Mechanism with Local Attention
Dou, R., Ru, L., & Príncipe, J. C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890757 · Cited by 2
CacheFormer: High-Attention-Based Segment Caching
Singh, S. K., & Mahmood, A. (2025) · AI · doi:10.3390/ai6040085 · Cited by 2
MHNet: A Masked Hybrid Network for Robust Water Body Segmentation From Aerial Images
Wang, S., Wei, B., Shi, B., Wang, N., Zhang, Y., & Zhu, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3580479 · Cited by 3
Li, Z., Jiang, B., Xu, L., Lu, Z., Wang, Z.-T., Liu, B., Jia, S.-Y., Liu, H.-D., & Li, B. (2025) · Algorithms · doi:10.3390/a18080454 · Cited by 2
Hestrio, Y. F., Mantau, A. J., & Jatmiko, W. (2025) · IEEE Access · doi:10.1109/access.2025.3593389 · Cited by 2
Wang, F., Ji, J., Wang, Y., Xu, Z., & Miao, Q. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3599170 · Cited by 2
Benchmarking Transformer-Based Transcription on Embedded GPUs for Space Applications
Schubert, M. E., & George, A. D. (2021) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct52877.2021.9622586 · Cited by 2
Progressive Token Reduction and Compensation for Hyperspectral Image Representation
Ma, C., Jiang, J., Li, H., Cui, W., & Li, G. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3241048 · Cited by 2
GPT-2 Metadata Pretraining Towards Instruction Finetuning for Ukrainian
Kyrylov, V., & Chaplynskyi, D. (2023) · Ukrainian Natural Language Processing Workshop (UNLP) · doi:10.18653/v1/2023.unlp-1.4 · Cited by 2
FocusLearn: Fully-Interpretable, High-Performance Modular Neural Networks for Time Series
Su, Q., Christos Kloukinas, Z., & Garcez, A. D. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651481 · Cited by 3
Multi-split Reversible Transformers Can Enhance Neural Machine Translation
Zhao, Y., Zhou, S., & Zhang, Z. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.19 · Cited by 2
Fast Spatial-Temporal Transformer Network
Escher, R. M., de, R., & Drews, P. (2021) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi54419.2021.00018 · Cited by 2
Deep Learning: Our Miraculous Year 1990-1991Preprint
Schmidhuber, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05744 · Cited by 9
Sparse Attention with Linear UnitsPreprint
Zhang, B., Titov, I., & Sennrich, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07012 · Cited by 9
Transformed CNNs: recasting pre-trained convolutional layers with self-attentionPreprint
d’Ascoli, S., Sagun, L., Biroli, G., & Morcos, A. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05795 · Cited by 5
Poly-NL: Linear Complexity Non-local Layers with PolynomialsPreprint
Babiloni, F., Marras, I., Kokkinos, F., Deng, J., Chrysos, G. G., & Zafeiriou, S. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02859 · Cited by 2
Armour: Generalizable Compact Self-Attention for Vision TransformersPreprint
Meng, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01778 · Cited by 3
An Optimized Dataflow for Mitigating Attention Performance BottlenecksPreprint
Kao, S.-C., Subramanian, S., Agrawal, G., & Krishna, T. (2021) · arXiv (Cornell University) · Cited by 2
Smart Bird: Learnable Sparse Attention for Efficient and Effective TransformerPreprint
Wu, C., Wu, F., Qi, T., Jiao, B., Jiang, D., Huang, Y., & Xie, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09193 · Cited by 2
\infty-former: Infinite Memory TransformerPreprint
Martins, P. H., Marinho, Z., & Martins, A. F. T. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.00301 · Cited by 11
Sparse Factorization of Large Square MatricesPreprint
Khalitov, R., Yu, T., Cheng, L., & Yang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08184 · Cited by 2
Ordinary Least Squares as an Attention MechanismPreprint
Coulombe, P. G. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5200864 · Cited by 2
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
“…keys and values are input tokens (e.g., tokenized text), the choice of objective and the optimization process for solving Equation 1 can result in distinct sequence modeling architectures (see [59] and [58]) such as global/local softmax attention [27], or other modern recurrent models [28, 60, 61].”
InfluentialMamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“10 Relative Total State Size”
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences
Chen, Y., Zeng, Q., Hakkani-Tür, D. Z., Jin, D., Ji, H., & Yang, Y. L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.381 · Cited by 1
Blin: A Multi-Task Sequence Recommendation Based on Bidirectional KL-Divergence and Linear Attention
Bai, Y., Wang, H., & He, J. (2024) · Mathematics · doi:10.3390/math12152391 · Cited by 1
Rock lithology classification algorithm based on improved self-attention mechanism VIT
Liu, H., Shi, S., & Ma, T. (2024) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2816/1/012049 · Cited by 1
Guo, Y., Zheng, J., Ding, Y., Hu, Y., Cao, Y., & Wang, Z. (2024) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar61438.2024.10670877 · Cited by 1
3D Single Object Tracking Network Based on Point Cloud Pre-segmentation
Li, J., Wu, Q., Yi, C., Chen, K., & Xuan, S. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10661198 · Cited by 1
Basheer, R., Daraie, A. H., & Mishra, D. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734832 · Cited by 1
Li, C., Zheng, X., Chen, C., Deng, Z., & Shu, Y. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3494540 · Cited by 1
Agent Attention Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification
Chang, D., Wu, J., & Li, X. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849185 · Cited by 1
F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation
Agarwal, M., Wang, C., & Gaël Richard, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888904 · Cited by 1
Zhou, L., Yin, L., & Qian, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889355 · Cited by 1
Dual-Ascent-Inspired Transformer for Compressed Sensing
Lin, R., Shen, Y., & Chen, Y. (2025) · Sensors · doi:10.3390/s25072157 · Cited by 1
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
