The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 801–855 of 855
Wei-dong, S., He, H., Dai, J., Jia, G., & Fan, J. (2024) · Preprints.org · doi:10.20944/preprints202403.0833.v1
Enhancing Swipe Typing with Gated Linear Transformer
Jia, N., Zhang, Y., & Wu, Q. Y. (2024) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci59607.2024.10497245
MAGI-NET: Masked Global Information Network for Symbolic Music Generation
Wang, D., Li, P., & Wu, J. (2023) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics60724.2023.00138
Forecasting of Bitcoin Prices Using Hashrate Features: Wavelet and Deep Stacking approachPreprint
Heidary, G., Afrookhteh, M., Khaloo, H., & Mousa, R. (2024) · Research Square · doi:10.21203/rs.3.rs-4406210/v1
Multi-source Fusion in Audio Scene Classification with Refined Transformers
Chen, S., Liu, W., Liu, X., Tang, P., Lv, J., & Guo, C. (2024) · International Conference on Algorithms, Software Engineering and Network Security (ASENS) · doi:10.1145/3677182.3677242
Reviewing the past enables us to learn
Zheng, W., & Wang, F. (2024) · Elsevier eBooks · doi:10.1016/b978-0-44-321619-0.00008-x
Remote Sensing Domain Adaptive Alignment via Student-Teacher Learning
Liu, Q., Yan, W., Wang, B., Veeravalli, B., & Tan, R. T. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10641122
CPF6D: 6DoF Pose Estimation Net based on Context Part Fusion for Robotic Grasping
Yuan, S., Ouyang, D., Yang, L., & Li, T. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650939
DfHM: A Hierarchical Approach for Matching Pairs of Images Using Graph Attention Neural Networks
Ouali, M. A., Bouguessa, M., & Ksantini, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650367
From Large Language Models to Generative AI Systems
Tomczak, J. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-64087-2_11
A Detector-Free Image Matching Method based on Enhanced Feature Fusion
Chen, Z., & Xu, C. Y. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10662731
FusionNetV2: Explicit Enhancement of Edge Features for 6D Object Pose Estimation
Ye, Y., & Park, H. (2024) · Electronics · doi:10.3390/electronics13183736
URL-AFLTnet: A Deep Neural Network Based on Self-Attention Feature Fusion
Chen, Z., Yuan, H., Hu, P., Wang, L., & Qi, F. (2024) · International Conference on Electronic Communication and Artificial Intelligence (ICECAI) · doi:10.1109/icecai62591.2024.10674776
LVS: A Learned Video Storage for Fast and Efficient Video Understanding
Lee, Y., & Park, J. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00808
Linear Retention Convolutional Network for Few-Shot Network Threat Detection
Ding, Y., Li, Z., Yang, C., Liang, H., Jia, S., & Li, Z. (2024) · Asia Joint Conference on Information Security (AsiaJCIS) · doi:10.1109/asiajcis64263.2024.00022
Research on Short Video Copyright Protection by Video Similarity Learning
Chen, R., Li, D., & Jin, X. (2024) · International Conference on Interactive Intelligent Systems and Techniques (IIST) · doi:10.1109/iist62526.2024.00072
Lightweight Neural Data Sequence Modeling by Scale Causal Blocks
Akutsu, H., & Arai, K. (2024) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7074
BeatNet+: Real‑Time Rhythm Analysis for Diverse Music Audio
Heydari, M., & Duan, Z. (2024) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.198
SLUMBR: SLeep statUs estiMation from aBdominal Respiratory effort
Romero, H. E., Ma, N., Brown, G. J., & Johnson, S. R. (2024) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc53108.2024.10782490
Few-shot Object Detection Algorithm Based on YOLOv8-Mamba-Like Linear Attention
Wang, H., Wang, K., Ji, B., & Deng, C. (2024) · IEEE International Conference on Communication Software and Networks · doi:10.1109/iccsn63464.2024.10793390
Region Aware Framework for Constrained Image Splicing Detection and Localization
Cao, H., Li, Y., Yang, X., & Li, X. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849068
Tenagyei, E. K., Gao, Y., Lewis, A., Nikzad, M., & Zhou, J. (2024) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta63115.2024.00024
MoPoSD: monocular 3D object detector with potential for specialized exploration in depth
Lin, J.-Y., Liu, S., & Wang, Y. (2025) · International Conference on Graphics and Image Processing (ICGIP) · doi:10.1117/12.3057826
Distortion Correction for Wide-Angle Videos of Power Inspection Robots
Ren, Z., & Su, Y. (2024) · doi:10.1109/epee63731.2024.10875060
SpliceSelectNet: a hierarchical Transformer-based deep learning model for splice site prediction
Miyachi, Y., & Nakai, K. (2026) · Nucleic Acids Research · doi:10.1093/nar/gkag625
scFTAT: a novel cell annotation method integrating FFT and transformer
Tang, B., & Chen, Y. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06061-z
MambaNext: An Enhanced Backbone Network with Focus Linear Attention
Zhang, D., & Liu, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887988
Li, B., He, Q., Cheng, J., Guan, T., He, Y., & Zhou, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889091
Enhancing Generalized EEG Classification with Decomposed Statistics-diverse Feature Augmentation
He, Y., Chen, C. L. P., Chen, B., & Zhang, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887610
Efficient Fusion of Computationally Diverse Modalities Using Chunking and Cross-Attention
Flores, C., Goncalves, L., & Busso, C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890415
Mining global relativity consistency without neighborhood modeling for correspondence pruning
Zhong, W., & Jiang, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127239
Transformers: A Graph Processing Perspective
Routhu, M. S. S. S., Yanduru, S. D., Tomczak, N., & Kuppannagari, S. R. (2024) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec62836.2024.10938487
A 0-Shot Self-Attention Mechanism for Accelerated Diagonal Attention
Mario, V., Shvai, N., Llanza, A., & Nakib, A. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00710
Dual-Representation Interaction Driven Image Quality Assessment with Restoration Assistance
Yue, J., Lin, X., Yang, Z., & Ren, C. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00310
Defierithos: The Lonely Warrior Rises from ResonancePreprint
Wang, L. (2025) · doi:10.36227/techrxiv.174181746.67591790/v3
Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models
Agrawal, S., Vemuri, D. S., P, S., & Balasubramanian, V. N. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i15.33681
Generating Synthetic Spectral Data using Conditional DDPM
Kubiczek, F., Patzke, S., & Thiem, J. (2025) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2025.es2025-189
Specific Emitter Identification of ADS-B Signal Based on Visual State Space Duality
Liu, Y., & Wu, X. (2025) · International Conference on Sensors and Information Technology (ICSI) · doi:10.1109/icsi64877.2025.11009602
Defierithos: The Lonely Warrior Rises from ResonancePreprint
Wang, L. (2025) · doi:10.36227/techrxiv.174181746.67591790/v4
Thota, L. N., Goru, H. K., Kaki, C., & Thota, K. S. (2025) · International Conference on Pervasive Computing and Social Networking (ICPCSN) · doi:10.1109/icpcsn65854.2025.11035827
EML-SAM: Incorporating Multi-Scale Fusion in SAM for Cine-CMR Segmentation
Hao, H., Wang, W., Ding, Q., Cheng, H., Hao, F., Zhang, H., & Bai, J. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033294
Algorithm-Hardware Co-Design for Ultra-Low-Power Large Language Models
Abreu, S., & Eshraghian, J. K. (2025) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas56072.2025.11043353
Geometry Meets Attention: Interpretable Transformers via SVD Inspiration
Arampatzakis, V., Pavlidis, G. P., Mitianoudis, N., & Papamarkos, N. (2025) · IEEE Access · doi:10.1109/access.2025.3586739
Common neighbor-aware link weight prediction with simplified graph transformer
Liu, L. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113614
GeoAT: Geometry-Aware Attention Feature Matching Network
Li, Y., Wu, Y., Yang, M., Zhang, Y., & Cheng, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3590252
Alonso, M. N. I. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5368783
Li, Y., Zhang, R., Li, K., Wang, Y., He, M., & Qian, J. (2025) · Journal of Biophotonics · doi:10.1002/jbio.202500407
3-D Single Object Tracking With Temporal-Aware and Attention Mechanism
Reta, B. Y., & Xia, W. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3548061
LPAT-MT: Lightweight Partial Convolution and Attention-Based Single Path Makeup Transfer
Liu, J., Guo, B., & He, Q. (2024) · doi:10.1109/isdh64927.2024.00023
Improving training efficiency for length-imbalanced translation data using optimal packing point
Seo, J. W., & Jung, H.-Y. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.114991
A review of deep learning transformer variants used in solar PV forecasting
Sharma, D., Kumar, C., Kumar, M., & -, B. B. S. (2026) · Sustainable Energy Research · doi:10.1186/s40807-026-00270-7
Kummara, H. (2026) · Cambridge Open Engage (preprint) · doi:10.33774/coe-2026-xpkvb
GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2023) · doi:10.1101/2023.06.12.544594 · Cited by 137
“Conversely, linear attention methods approximate full token-to-token interactions through softmax linearization [23, 24].”
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
“The attention weights follow a softmax kernel ϕ ( q , k ) = exp (cid:0) q ⊤ RMSNorm( k ) (cid:1) [56, 148], where the RMSNorm prevents layers with large-magnitude outputs from dominating the weights: Since network depth is modest ( L < 100 ), the O ( L 2 d ) arithmetic of this full form is…”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Dong, Y., Cordonnier, J.-B., & Loukas, A. (2021) · arXiv · doi:10.48550/arxiv.2103.03404 · Cited by 195
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
