The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 401–500 of 855
Asymptotic theory of in-context learning by linear attention
Lu, Y. M., Letey, M. I., Zavatone-Veth, J. A., Maiti, A., & Pehlevan, C. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2502599122 · Cited by 50
Two-View Geometry Scoring Without Correspondences
Barroso-Laguna, A., Brachmann, E., Prisacariu, V. A., Brostow, G., & Turmukhambetov, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00867 · Cited by 10
Multi-Correlation Siamese Transformer Network With Dense Connection for 3D Single Object Tracking
Feng, S., Liang, P., Gao, J. G., & Cheng, E. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3325715 · Cited by 10
Jiang, W. Q., Pan, H., Wang, Y., Li, Y., Lin, Y., & Bi, F. (2024) · Remote Sensing · doi:10.3390/rs16162880 · Cited by 10
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field AttentionPreprint
Nguyen, T. M., Suliafu, V., Osher, S., Chen, L., & Bao, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02347 · Cited by 12
Transformer-based severity detection of Parkinson's symptoms from gait
Sun, H.-J., & Zhang, Z. (2022) · International Congress on Image and Signal Processing (CISP) · doi:10.1109/cisp-bmei56279.2022.9980289 · Cited by 9
TAG-fusion: Two-stage attention guided multi-modal fusion network for semantic segmentation
Zhang, Z., Wang, W., Zhu, L., & Tang, Z. (2024) · Digital Signal Processing · doi:10.1016/j.dsp.2024.104807 · Cited by 9
La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection
Zou, H., Du, C., Zhang, H., Zhang, Y., Liu, A., Wan, J., & Lei, Z. (2024) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744523 · Cited by 10
DSAFuse: Infrared and visible image fusion via dual-branch spatial adaptive feature extraction
Shen, S., Feng, Y., Liu, N., Liu, M., & Li, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128957 · Cited by 9
Hyena architecture enables fast and efficient protein language modeling
Zhang, Y., Bian, B., & Okumura, M. (2024) · iMetaOmics. · doi:10.1002/imo2.45 · Cited by 10
STAT-LSTM: A multivariate spatiotemporal feature aggregation model for SPEI-based drought prediction
Chen, Y., Wu, H., Xie, N., Liang, X., Jiang, L., Qiu, M., & Li, Y. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-01813-0 · Cited by 9
Structured State Space Decoder for Speech Recognition and Synthesis
Miyazaki, K., Murata, M., & Koriyama, T. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096135 · Cited by 9
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 9
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences
Li, J., Aitken, W., Bhambhoria, R. V., & Zhu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.120 · Cited by 15
Deep Learning-Based Action Classification Using One-Shot Object Detection
Yoo, H., Lee, S.-E., & Chung, K. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.039263 · Cited by 9
A Comprehensive Survey On Efficient Transformers
Elouargui, Y., Zyate, M., Sassioui, A., Chergui, M., Kamili, M. E., & Ouzzif, M. (2023) · International Conference on Wireless Networks and Mobile Communications (WINCOM) · doi:10.1109/wincom59760.2023.10322921 · Cited by 9
SeTformer Is What You Need for Vision and Language
Shamsolmoali, P., Zareapoor, M., Granger, É., & Felsberg, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28272 · Cited by 9
Towards Asynchronous Multimodal Signal Interaction and Fusion via Tailored Transformers
Yang, D., Kuang, H., Yang, K., Li, M., & Zhang, L. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3409211 · Cited by 9
Zhao, Y., Jing, S., Wu, H., Li, H., & Masahiro Todoh, A. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3472778 · Cited by 9
Ravenscroft, W., Goetze, S., & Hain, T. (2022) · Frontiers in Signal Processing · doi:10.3389/frsip.2022.856968 · Cited by 9
Park, J., Artin, M. G., Lee, E., May, B. L., Park, M. K., Hur, C., & Tatonetti, N. P. (2022) · Patterns · doi:10.1016/j.patter.2022.100636 · Cited by 10
An overlap estimation guided feature metric approach for real point cloud registration
Zhang, F., Zhang, L., He, T., Sun, Y., Zhao, S., Zhang, Y., Zhao, X., & Zhao, W. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.01.010 · Cited by 9
scFormer: A Universal Representation Learning Approach for Single-Cell Data Using TransformersPreprint
Cui, H., Wang, C., Maan, H., Duan, N., & Wang, B. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.20.517285 · Cited by 10
Wang, Y., Wang, Z., Sadeh, G., Zancato, L., Achille, A., Karypis, G., & Rangwala, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.29.620988 · Cited by 12
Zhang, Z., Wang, Y., Huang, X., Fang, T., Zhang, H., Deng, C., Li, S., & Yu, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.485 · Cited by 9
Time Series Forecasting in Marine Environment Monitoring Sensor Networks Based on Edge Computing
Teng, J., Qin, R., & Xu, C. (2025) · IEEE Access · doi:10.1109/access.2025.3554178 · Cited by 8
ET-Mamba: A Mamba Model for Encrypted Traffic Classification
Xu, J. F., Chen, L., Xu, W., Dai, L., Wang, C., & Hu, L. (2025) · Information · doi:10.3390/info16040314 · Cited by 8
Speech and multilingual natural language framework for speaker change detection and diarization
Anidjar, O. H., Estève, Y., Hajaj, C., Dvir, A. Z., & Lapidot, I. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119238 · Cited by 8
MMusic: a hierarchical multi-information fusion method for deep music recommendation
Xu, J., Gan, M., & Zhang, X. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-023-00786-0 · Cited by 8
InstaFormer++: Multi-Domain Instance-Aware Image-to-Image Translation with Transformer
Kim, S.-H., Park, J., Park, J., Ha, E., Jung, H., Lee, T., & Kim, S. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01866-y · Cited by 10
Zhang, Z., & Yin, J. (2024) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2024.2305318 · Cited by 8
MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion
Ji, S., & Yang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29178 · Cited by 8
Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning
Li, R., Fischer, T., Segù, M., Pollefeys, M., Gool, L. V., & Tombari, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00940 · Cited by 10
Optimizing Vision Transformers: Unveiling ‘Focus and Forget’ for Enhanced Computational Efficiency
Latibari, B. S., Homayoun, H., & Sasan, A. (2025) · IEEE Access · doi:10.1109/access.2025.3540399 · Cited by 9
Gu, Y., Qin, X., Peng, Y., & Li, L. (2022) · IET Image Processing · doi:10.1049/ipr2.12575 · Cited by 9
Ferdowsi, S., Borissov, N., Knafou, J., Amini, P., & Teodoro, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.48 · Cited by 8
Simple Local Attentions Remain Competitive for Long-Context Tasks
Xiong, W., Oğuz, B., Gupta, A., Chen, X., Liskovich, D., Levy, O., Yih, S., & Mehdad, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.144 · Cited by 8
Patch-based stochastic attention for image editing
Cherel, N., Almansa, A., Gousseau, Y., & Newson, A. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103866 · Cited by 11
Long-range Sequence Modeling with Predictable Sparse Attention
Zhuang, Y., Zhang, J., & Tu, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.19 · Cited by 10
WLiT: Windows and Linear Transformer for Video Action Recognition
Sun, R., Zhang, T., Wan, Y., Zhang, F., & Wei, J. (2023) · Sensors · doi:10.3390/s23031616 · Cited by 9
AdaVITS: Tiny VITS for Low Computing Resource Speaker Adaptation
Song, K., Xue, H., Wang, X., Cong, J., Zhang, Y., Xie, L., Yang, B., Zhang, X., & Su, D. (2022) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp57327.2022.10037585 · Cited by 8
Diffuser: Efficient Transformers with Multi-Hop Attention Diffusion for Long Sequences
Feng, A., Li, I., Jiang, Y., & Ying, R. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26502 · Cited by 12
ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading
Xiao, Y., Zhang, S., Wang, X., Tan, X., He, L., Zhao, S., Soong, F. K., & Lee, T. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-122 · Cited by 9
Generating Data for Symbolic Language with Large Language Models
Ye, J., Li, C., Kong, L., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.523 · Cited by 18
A Mathematical Theory of AttentionPreprint
Vuckovic, J., Baratin, A., & Combes, R. T. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02876 · Cited by 68
Urban Sound Classification : striving towards a fair comparisonPreprint
Arnault, A., Hanssens, B., & Riche, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11805 · Cited by 15
DeepConsensus: Gap-Aware Sequence Transformers for Sequence CorrectionPreprint
Baid, G., Cook, D. E., Shafin, K., Yun, T., Llinares-López, F., Berthet, Q., Wenger, A., Rowell, W., Nattestad, M., Yang, H., et al. (2021) · bioRxiv · doi:10.1101/2021.08.31.458403 · Cited by 13
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v2 · Cited by 11
A Rewiring Contrastive Patch PerformerMixer Framework for Graph Representation Learning
Sun, Z., Harit, A., Cristea, A. I., Wang, J., & Lió, P. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386951 · Cited by 8
ContextMatcher: Detector-Free Feature Matching With Cross-Modality Context
Li, D., & Ду, С. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3383334 · Cited by 7
Adaptive Patching for High-resolution Image Segmentation with Transformers
Zhang, E., Lyngaas, I., Chen, P., Wang, X., Igarashi, J., Huo, Y., Munetomo, M., & Wahib, M. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00082 · Cited by 7
Zong, X., Luo, W., Ning, B., He, K., Lian, L., & Sun, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3399820 · Cited by 9
MR-Matcher: A Multirouting Transformer-Based Network for Accurate Local Feature Matching
Jiang, Z., Wang, K., Kong, Q., Dai, K., Xie, T., Qin, Z., Li, R., Perner, P., & Zhao, L. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3412217 · Cited by 7
Remote Sensing Semantic Change Detection Model for Improving Objects Completeness
Yang, W., Cheng, Y., & Xu, W. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3493945 · Cited by 7
Artificial intelligence-driven plant bio-genomics research: a new era
Yang, L., Wang, H., Zou, M., Chai, H., & Xia, Z. (2025) · Tropical Plants · doi:10.48130/tp-0025-0008 · Cited by 9
Gong, H., Li, M., Wang, L., Huang, S., & Li, M. (2025) · Applied Sciences · doi:10.3390/app15105518 · Cited by 7
Wu, W., & Gu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130507 · Cited by 9
Huang, D., Zhang, G., Li, Z., Liu, K., & Luo, W. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104428 · Cited by 7
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Csordás, R., Irie, K., & Schmidhuber, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.49 · Cited by 19
Tan, H., He, L., Du, W., Liu, H., Chen, H., Zhang, Y., & Yang, H. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.104008 · Cited by 7
An Efficient CNN + Sparse Transformer-Based Intrusion Detection Method for IoT
Zhang, Y., Fan, Y., Guo, Z., Ma, H., Guan, Q., Jiang, W., & Li, W. J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5609-4_38 · Cited by 7
LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching
Zhong, W., & Jiang, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126393 · Cited by 7
Song, K., Jung, Y., Kim, D., & Moon, I.-C. (2020) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17168 · Cited by 22
Double Linear Transformer for Background Music Generation from Videos
Yang, X., Yu, Y., & Wu, X. (2022) · Applied Sciences · doi:10.3390/app12105050 · Cited by 7
Rad, A.-C., Lemnaru, C., & Munteanu, A. (2022) · Sensors · doi:10.3390/s22197457 · Cited by 7
Context-Aware Sequence Alignment using 4D Skeletal Augmentation
Kwon, T., Tekin, B., Tang, S., & Pollefeys, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00800 · Cited by 11
Lightweight Monocular Depth Estimation with an Edge Guided Network
Dong, X., Garratt, M., Anavatti, S. G., Abbass, H. A., & Dong, J. (2022) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv57592.2022.10004313 · Cited by 7
Learning General Descriptors for Image Matching With Regression Feedback
Rao, Y., Ju, Y., Li, C., Rigall, E., Yang, J., Fan, H., & Dong, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3267279 · Cited by 9
Learning Neural Eigenfunctions for Unsupervised Semantic Segmentation
Deng, Z., & Luo, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00057 · Cited by 7
Montalbo, F. J. P., Hernandez, L. R. T., Palad, L. P., Castillo, R. C., Alon, A. S., & Ocampo, A. L. P. D. (2023) · International Conference on Biomedical Imaging, Signal Processing (ICBSP) · doi:10.1145/3634875.3634878 · Cited by 7
Gated Recurrent Context: Softmax-Free Attention for Online Encoder-Decoder Speech Recognition
Lee, H., Kang, W. H., Cheon, S. J., Kim, H., & Kim, N. S. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3049344 · Cited by 7
High Performance Natural Language Processing
Ilharco, G., Magalhaes, C. I., Turc, I., Dettmers, T., Ferreira, F., & Lee, K. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-tutorials.4 · Cited by 7
PoNet: Pooling Network for Efficient Token Mixing in Long SequencesPreprint
Tan, C.-H., Chen, Q., Wang, W., Zhang, Q., Zheng, S., & Ling, Z.-H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02442 · Cited by 9
On Learning the Transformer KernelPreprint
Chowdhury, S. P., Solomou, A., Dubey, A., & Sachan, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08323 · Cited by 13
SSAformer: Spatial–Spectral Aggregation Transformer for Hyperspectral Image Super-Resolution
Wang, H., Zhang, Q., Peng, T., Xu, Z., Cheng, X., Xing, Z., & Li, T. (2024) · Remote Sensing · doi:10.3390/rs16101766 · Cited by 8
SelfCP: Compressing over-limit prompt via the frozen large language model itself
Gao, J., Cao, Z., & Li, W. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103873 · Cited by 7
Cross-city traffic state prediction based on knowledge transfer framework
Xu, D., Tang, Y., Ju, J., Yu, Z., Zheng, J., Gu, T., & Guo, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126747 · Cited by 8
LRetUNet: A U-Net-based retentive network for single-channel speech enhancement
Zhang, Y., Zhang, Z., Guo, W., Chen, W., Liu, Z., & Liu, H. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101798 · Cited by 6
Wang, S., Shi, B., Wang, N., Zhang, Y., & Zhu, Y. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3569410 · Cited by 6
DIFNet: Dual-Information Fusion Network for depth completion
Wu, K., Lin, J., Miao, J., Li, Z., Zhang, X., Xing, G., Fan, Y., Luo, J., Zhao, H., Liu, Y., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103424 · Cited by 6
WTxGRN: Wavelet Transform-Based Extended Gated Recurrent Network for Palm Vein Recognition
Qin, H., Fu, Y., Jing, C., Song, Q., Li, Y., El‐Yacoubi, M. A., & Zhong, D. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3592561 · Cited by 6
Transformer spectral optimization: From gradient frequency analysis to adaptive spectral integration
Huang, Z., Chen, M., & Zheng, S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113637 · Cited by 6
CGMamba: Intelligent Identification of Counterfeit Goods Based on State Space Models
Li, Y., Wang, T., Luo, N., Zhou, L., & Chen, Q. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/9939880 · Cited by 6
EndoDepthL: Lightweight Endoscopic Monocular Depth Estimation with CNN-Transformer
Li, Y. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10386008 · Cited by 6
DeepLIR: Attention-Based Approach for Mask-Based Lensless Image Reconstruction
Poudel, A., & Nakarmi, U. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00052 · Cited by 7
Traffic signal current prediction algorithm based on CNN and LSTM
Feng, W., Qi, S., Guo, J., Zuo, X., Chen, Y., & Zhu, Y. (2024) · Measurement Science and Technology · doi:10.1088/1361-6501/ad8e7a · Cited by 6
Wang, Q., Zhang, D., Cai, X., Zhang, T., & Xu, B. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1516868 · Cited by 6
Toward Unifying Saliency Transformer for Video Saliency Prediction and Detection
Xiong, J., Li, C., Liu, T., Zhang, P., Huo, Y., Huang, W., & Zha, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3544031 · Cited by 8
LEST: Large-Scale LiDAR Semantic Segmentation With Deployment-Friendly Transformer Architecture
Luo, C., Cheng, N., Ma, S., Li, H., Li, X., Lei, S., & Li, P. (2025) · IEEE Access · doi:10.1109/access.2025.3545957 · Cited by 6
Bond-Taylor, S., Hessey, P., Sasaki, H., Breckon, T. P., & Willcocks, C. G. (2021) · Durham Research Online (Durham University) · doi:10.48550/arxiv.2111.12701 · Cited by 6
Sparse factorization of square matrices with application to neural attention modeling
Khalitov, R., Yu, T., Cheng, L., & Yang, Z. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.04.014 · Cited by 11
ChapterBreak: A Challenge Dataset for Long-Range Language Models
Sun, S., Thai, K., & Iyyer, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.271 · Cited by 13
OBTAIN: Observational Therapy-Assistance Neural Network for Training State Recognition
Wang, M., & Yang, N. (2023) · IEEE Access · doi:10.1109/access.2023.3263117 · Cited by 6
MSFORMER: Multi-Scale Transformer with Neighborhood Consensus for Feature Matching
Li, D., Yan, Y., Liang, D., & Ду, С. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095240 · Cited by 6
OTA-NN: Observational Therapy-Assistance Neural Network for Enhancing Autism Intervention Quality
Wang, M., & Yang, N. (2022) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc49033.2022.9700714 · Cited by 6
The Beauty of Repetition in Machine Composition Scenarios
Hu, Z., Ma, X., Liu, Y., Chen, G., & Liu, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548130 · Cited by 6
On Data Sampling Strategies for Training Neural Network Speech Separation Models
Ravenscroft, W., Goetze, S., & Hain, T. (2023) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco58844.2023.10289800 · Cited by 6
Context Compression for Auto-regressive Transformers with Sentinel Tokens
Ren, S., Jia, Q., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.794 · Cited by 11
MuseMorphose: Full-Song and Fine-Grained Music Style Transfer with Just One Transformer VAE.Preprint
Wu, S.-L., & Yang, Y. (2021) · arXiv (Cornell University) · Cited by 6
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
