The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Open accessAbstract
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Despite efficiency and the ability to scale to longer context, linear Transformers do not show competitive performance compared to Transformers as the kernel trick makes the model a linear recurrent network, in which the data is compressed into a matrix-valued states (Katharopoulos et al. 2020).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
From FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While both linear Transformers and linear RNNs (including state space models) compress the information in memory update step, the critical difference lies in the structure of the memory, where linear RNNs (vs. linear Transformers) use a vector-valued memory (vs. matrix-valued memory).”
From Titans: Learning to Memorize at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current large language models (LLMs) exhibit remarkable single-turn capabilities but suffer from severe limitations in maintaining coherent, personalized interactions over extended temporal horizons a phenomenon we term the 'goldfish problem.'”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Architectural constraints : Self-attention complexity scales as O ( n 2 ) with sequence length n , making unlimited context computationally infeasible even with recent linear attention variants [5].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [5]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As an example of this type of knowledge transfer, in linear Transformer (or FWP) (Katharopoulos et al. 2020; Schlag et al. 2021), where the initial memory state is zero, the stored knowledge of a lower level (fast weight) directly affect the output of the model in another level.”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given different settings for 𝛼 𝑡 (i.e., either is 1, learnable, channel-wise, and/or input-dependent) and also 𝜙 (·) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule (Katharopoulos et al. 2020; Sun et al. 2023; Arora et al. 2024; Beck et al. 2024; Kacham et al. 2024; Peng et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One of the commonly used techniques to enhance the capacity of a memory is to use higher-order feature maps on the keys (Katharopoulos et al. 2020; Kacham et al. 2024).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“E [ ∥ S T ∥ F ] = O ( T ) for random inputs”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results show that, in contrast to other methods (Katharopoulos et al., 2020;Sun et al., 2023;Qin et al., 2023;Katsch, 2023;Yang et al., 2023;Qin et al., 2024;Peng et al., 2024), having the gates both learnable and input dependent gives the best results.”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Katharopoulos et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The Learned Attention (i.e., Linear Attention (Katharopoulos et al., 2020), Attention with Gating (Xue et al., 2020), etc.) is designed for performance , not interpretability. And they obfuscate the explicit interaction structure between token representations that we modeled in RN.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants (Vaswani et al. 2017), modern recurrent neural networks (Katharopoulos et al. 2020; Schlag et al. 2021; Behrouz et al. 2025c; Peng et al. 2025b), canon layers (Allen-Zhu 2025), global convolutions (Hasani et al. 202”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“The first generation of modern recurrent architectures (e.g., Linear attention (Katharopoulos et al. 2020), RetNet (Sun et al. 2023), RWKV (Peng et al. 2023), lightening attention (Li et al. 2025)) are based on Hebbian-like learning rules (Hebb 2005).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“which recovers the original linear attention recurrence (Katharopoulos et al. 2020).”
From Nested Learning: The Illusion of Deep Learning Architectures (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention [6] removes the quadratic bottleneck”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“This update is unconditional: every new key-value pair is accumulated with equal weight regardless of what S already stores.”
From Variational Linear Attention: Stable Associative Memory for Long-Context Transformers (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state Φ ∈ R n v × n a according to the linear dynamics”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020).”
From MesaNet: Sequence Modeling by Locally Optimal Test-Time Training (2025) · cites this paper as Katharopoulos et al., 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.”
From Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“In the online case ( 𝑐 = 1) and 𝜙 (·) = (·) , this recurrence is the same as linear attention (Katharopoulos et al. 2020).”
From ATLAS: Learning to Optimally Memorize the Context at Test Time (2025) · cites this paper as Katharopoulos et al. 2020
The cited source says what the claim attributes to it.
In the citing paper
“Linear-attention Transformers can evaluate their next-token function in O(n) time in their input length, as observed by Katharopoulos et al. (2020).”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Katharopoulos et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 33
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proceedings of the 5th International Conference on Learning Representations , San Diego, CA, USA, 2015.
- Blanc, G. and Rendle, S. Adaptive sampled softmax with kernel based sampling. arXiv preprint arXiv:1712.00527 , 2017. · arXiv
- Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. · arXiv
- Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.
- Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv preprint arXiv:1511.07289 , 2015. · arXiv
- Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations , 2020.
- Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
- Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, Ł. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018. · arXiv
- Goodman, J. Classes for fast maximum entropy training. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221) , volume 1, pp. 561-564. IEEE, 2001.
- Graves, A., Fern´ andez, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pp. 369-376, 2006.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Kitaev, N., Kaiser, Ł., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. · arXiv
- Lample, G., Sablayrolles, A., Ranzato, M. A., Denoyer, L., and Jegou, H. Large memory layers with product keys. In Wallach, H., Larochelle, H., Beygelzimer, A., d ´ Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 85468557. Curran Associates, Inc., 2019.
- Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations , 2020.
- Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019. · arXiv
- Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 14014-14024. Curran Associates, Inc., 2019.
- Mnih, A. and Hinton, G. E. A scalable hierarchical distributed language model. In Advances in neural information processing systems , pp. 1081-1088, 2009.
- Morin, F. and Bengio, Y. Hierarchical probabilistic neural network language model. In Aistats , volume 5, pp. 246252. Citeseer, 2005.
- Parmar, N., Ramachandran, P., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in vision models. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32 , pp. 68-80. Curran Associates, Inc., 2019.
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pp. 8024-8035, 2019.
- Paul, D. B. and Baker, J. M. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language , pp. 357-362. Association for Computational Linguistics, 1992.
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
- Rawat, A. S., Chen, J., Yu, F. X. X., Suresh, A. T., and Kumar, S. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems , pp. 13834-13844, 2019.
- Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017. · arXiv
- Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. arXiv preprint arXiv:1812.01243 , 2020. · arXiv
- Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MASS: Masked sequence to sequence pre-training for language generation. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 5926-5936, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
- Sperber, M., Niehues, J., Neubig, G., Stker, S., and Waibel, A. Self-attentional acoustic models. In 19th Annual Conference of the International Speech Communication Association (InterSpeech 2018) , Hyderabad, India, September 2018.
- Sukhbaatar, S., Grave, E., Bojanowski, P., and Joulin, A. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 331-335, Florence, Italy, July 2019. Association for Computational Linguistics.
- Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27 , pp. 31043112. Curran Associates, Inc., 2014.
- Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4343-4352, Hong Kong, China, November 2019. Association for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS , 2017.
- Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR , abs/1906.08237, 2019. · arXiv
- Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: quantized 8bit BERT. CoRR , abs/1910.06188, 2019. · arXiv
Cited by
Sorted by citations
Items 101–200 of 855
Diffusion Kernel Attention Network for Brain Disorder Classification
Zhang, J., Zhou, L., Wang, L., Liu, M., & Shen, D. G. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3170701 · Cited by 86
Eye Know You Too: Toward Viable End-to-End Eye Movement Biometrics for User Authentication
Lohr, D. J., & Komogortsev, O. V. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3201369 · Cited by 97
Exploring Relational Context for Multi-Task Dense Prediction
Brüggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Gool, L. V. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01557 · Cited by 90
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Bondarenko, Y., Nagel, M., & Blankevoort, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.627 · Cited by 172
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine TranslationPreprint
Kasai, J., Pappas, N., Peng, H., Cross, J., & Smith, N. A. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2006.10369 · Cited by 161
Self-citationTowards Accurate Post-Training Quantization for Vision Transformer
Ding, Y., Qin, H., Yan, Q., Chai, Z., Liu, J., Wei, X., & Liu, X. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547826 · Cited by 79
DeepMatcher: A deep transformer-based network for robust and accurate local feature matching
Xie, T., Dai, K., Wang, K., Li, R., & Zhao, L. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121361 · Cited by 86
FsaNet: Frequency Self-Attention for Semantic Segmentation
Zhang, F., Panahi, A., & Gao, G. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3305090 · Cited by 78
Depression Detection in Speech Using Transformer and Parallel Convolutional Neural Networks
Yin, F., Du, J. K., Xu, X., & Li, Z. (2023) · Electronics · doi:10.3390/electronics12020328 · Cited by 79
XoFTR: Cross-modal Feature Matching Transformer
Tuzcuoğlu, Ö., Köksal, A., Sofu, B., Kalkan, S., & Alatan, A. A. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00431 · Cited by 76
You Only Segment Once: Towards Real-Time Panoptic Segmentation
Hu, J., Huang, L., Ren, T., Zhang, S., Ji, R., & Cao, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01709 · Cited by 81
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks
Kao, S.-C., Subramanian, S., Agrawal, G., Yazdanbakhsh, A., & Krishna, T. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575747 · Cited by 98
Energformer: A New Transformer Model for Energy Disaggregation
Angelis, G.-F., Timplalexis, C., Salamanis, A. I., Krinidis, S., Ioannidis, D., Kehagias, D. D., & Tzovaras, D. K. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3237862 · Cited by 74
Dass, J., Wu, S., Shi, H., Li, C., Ye, Z., Wang, Z., & Lin, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071081 · Cited by 79
A review of AI edge devices and lightweight CNN and LLM deployment
Sun, K., Wang, X., Miao, X., & Zhao, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128791 · Cited by 87
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction
Jie, Z., Li, J., & Lu, W. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.410 · Cited by 83
A Transformer based approach to electricity load forecasting
Chan, J. W., & Yeo, C. K. (2024) · The Electricity Journal · doi:10.1016/j.tej.2024.107370 · Cited by 80
Learning All Dynamics: Traffic Forecasting via Locality-Aware Spatio-Temporal Joint Transformer
Fang, Y., Zhao, F., Qin, Y., Luo, H., & Wang, C. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3197640 · Cited by 71
Efficient Movie Scene Detection using State-Space Transformers
Islam, M. M., Hasan, M., Athrey, K. S., Braskich, T., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01798 · Cited by 76
Graph Neural Networks for Natural Language Processing: A SurveyPreprint
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06090 · Cited by 70
Self-attention-based adaptive remaining useful life prediction for IGBT with Monte Carlo dropout
Xiao, D., Qin, C., Ge, J., Xia, P., Huang, Y., & Liu, C. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107902 · Cited by 68
Pfeffer, M. A., Ling, S. H., & Wong, J. K. W. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108705 · Cited by 75
Patro, B. N., & Agneeswaran, V. S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111279 · Cited by 81
Deep Learning in Diverse Intelligent Sensor Based Systems
Zhu, Y., Wang, M., Yin, X., Zhang, J., Meijering, E. H. W., & Hu, J. (2022) · Sensors · doi:10.3390/s23010062 · Cited by 73
Transformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 115
End-to-End Structural analysis in civil engineering based on deep learning
Wang, C., Song, L.-H., & Fan, J. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104255 · Cited by 63
Long-Short Transformer: Efficient Transformers for Language and Vision
Zhu, C., Ping, W., Xiao, C., Shoeybi, M., Goldstein, T., Anandkumar, A., & Catanzaro, B. (2021) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2107.02192 · Cited by 74
Detector-Free Structure from Motion
He, X., Sun, J., Wang, Y., Peng, S., Qi-xing Huang, A., Bao, H., & Zhou, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02040 · Cited by 66
Spatio-temporal multi-level attention crop mapping method using time-series SAR imagery
Han, Z., Zhang, C., Gao, L., Zeng, Z., Zhang, B., & Atkinson, P. M. (2023) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2023.11.016 · Cited by 62
MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer With One Transformer VAE
Wu, S.-L., & Yang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3270726 · Cited by 97
Training language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60
T-former: An Efficient Transformer for Image Inpainting
Deng, Y., Hui, S., Zhou, S., Meng, D., & Wang, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548446 · Cited by 64
A Length-Extrapolatable Transformer
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.816 · Cited by 59
You, H., Xiong, Y., Dai, X., Wu, B., Zhang, P., Fan, H., Vajda, P., & Lin, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01387 · Cited by 67
Coordination among neural modules through a shared global workspace
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., & Bengio, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.01197 · Cited by 59
InstaFormer: Instance-Aware Image-to-Image Translation with Transformer
Kim, S.-H., Baek, J., Park, J., Kim, G., & Kim, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01778 · Cited by 62
Zhang, X., Zhang, X., Chen, W., Li, C., & Yu, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-60278-1 · Cited by 68
Building transformers from neurons and astrocytes
(2023) · doi:10.1073/pnas.2219150120 · Cited by 57
Adaptive Assignment for Geometry Aware Local Feature Matching
Huang, D., Chen, Y., Liu, Y., Liu, J., Xu, S., Wu, W., Ding, Y., Tang, F., & Wang, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00525 · Cited by 57
Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., & Susskind, J. (2021) · arXiv · doi:10.48550/arxiv.2105.14103 · Cited by 57
When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo
Liu, T., Ye, X., Zhao, W., Pan, Z., Shi, M., & Cao, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01658 · Cited by 56
Swin-fusion: An adaptive multi-source information fusion framework for enhanced tool wear monitoring
Hou, K., Li, R., Liu, X., Yue, C., Wang, Y., Liu, X., & Xia, W. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.02.003 · Cited by 59
Token Pooling in Vision Transformers for Image Classification
Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., & Tuzel, O. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00010 · Cited by 67
Sun, W., Qin, Z., Deng, H., Wang, J., Zhang, Y., Zhang, K., Barnes, N. M., Birchfield, S. T., Kong, L., & Zhong, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3285569 · Cited by 66
The Explainability of Transformers: Current Status and Directions
Fantozzi, P., & Naldi, M. (2024) · Computers · doi:10.3390/computers13040092 · Cited by 67
Review of synthetic aperture radar with deep learning in agricultural applications
Hashemi, M. G. Z., Jalilvand, E., Alemohammad, H., Tan, P., & Das, N. N. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.08.018 · Cited by 54
Wu, S.-L., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01307 · Cited by 74
Rethinking Few-shot 3D Point Cloud Semantic Segmentation
An, Z., Sun, G., Liu, Y., Liu, F., Wu, Z., Wang, D., Gool, L. V., & Belongie, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00383 · Cited by 54
Quantum Machine Learning in Drug Discovery: Applications in Academia and Pharmaceutical Industries
Smaldone, A. M., Shee, Y., Kyro, G. W., Xu, C., Vu, N. P., Dutta, R., Farag, M. H., Galda, A., Kumar, S., Kyoseva, E. S., et al. (2025) · Chemical Reviews · doi:10.1021/acs.chemrev.4c00678 · Cited by 74
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 87
Deep dive into predictive excellence: Transformer's impact on groundwater level prediction
Sun, W., Chang, L., & Chang, F. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131250 · Cited by 50
FeatureBooster: Boosting Feature Descriptors with a Lightweight Neural Network
Wang, X., Liu, Z., Hu, Y., Xi, W., Yu, W., & Zou, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00737 · Cited by 49
Jain, E., Kaushik, P., Kukreja, V., Ati, M., Hariharan, S., Ahuja, V., Bhattacherjee, A., & Kaushal, R. K. (2025) · IEEE Access · doi:10.1109/access.2025.3563256 · Cited by 49
Rethinking Optical Flow from Geometric Matching Consistent Perspective
Dong, Q., Cao, C., & Fu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00135 · Cited by 48
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine TranslationPreprint
Kasai, J., Pappas, N., Peng, H., Cross, J., & Smith, N. A. (2020) · arXiv (Cornell University) · Cited by 44
Self-citationClassification of long sequential data using circular dilated convolutional neural networks
Cheng, L., Khalitov, R., Yu, T., Zhang, J., & Yang, Z. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.054 · Cited by 51
Ren, Y., Wang, F., Zhang, T., Pollefeys, M., & Susstrunk, S. E. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01601 · Cited by 47
A Deformable Attention Network for High-Resolution Remote Sensing Images Semantic Segmentation
Zuo, R., Zhang, G., Zhang, R., & Jia, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3119537 · Cited by 48
Container: Context Aggregation NetworkPreprint
Gao, P., Lu, J., Li, H., Mottaghi, R., & Kembhavi, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01401 · Cited by 80
EstraNet: An Efficient Shift-Invariant Transformer Network for Side-Channel Analysis
Hajra, S., Chowdhury, S., & Mukhopadhyay, D. (2023) · IACR Transactions on Cryptographic Hardware and Embedded Systems · doi:10.46586/tches.v2024.i1.336-374 · Cited by 49
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
Zhang, J., Li, X., Wang, Y., Wang, C., Yang, Y., Liu, Y., & Tao, D. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02034-6 · Cited by 57
Dynamic Sparse Attention for Scalable Transformer Acceleration
Liu, L., Qu, Z., Chen, Z., Tu, F., Ding, Y., & Xie, Y. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3208206 · Cited by 84
Batch Prompting: Efficient Inference with Large Language Model APIs
Cheng, Z., Kasai, J., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.74 · Cited by 95
Map-Relative Pose Regression for Visual Re-Localization
Chen, S., Cavallari, T., Prisacariu, V. A., & Brachmann, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01953 · Cited by 43
An Efficient Transformer Based on Global and Local Self-Attention for Face Photo-Sketch Synthesis
Yu, W., Zhu, M., Wang, N., Wang, X., & Gao, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3229614 · Cited by 48
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43
GC-MVSNet: Multi-View, Multi-Scale, Geometrically-Consistent Multi-View Stereo
Vats, V., Joshi, S., Crandall, D., Reza, M. A., & Jung, S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00321 · Cited by 42
Koizumi, Y., Karita, S., Wisdom, S., Erdogan, H., Hershey, J., Jones, L., & Bacchiani, M. (2021) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics · doi:10.1109/waspaa52581.2021.9632794 · Cited by 50
Edge-featured multi-hop attention graph neural network for intrusion detection system
Deng, P., & Huang, Y. H. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104132 · Cited by 41
PMatch: Paired Masked Image Modeling for Dense Geometric Matching
Zhu, S., & Liu, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02098 · Cited by 45
ICDAR 2021 Competition on Scientific Literature Parsing
Yepes, A. J., Xu, Z., & Burdick, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86337-1_40 · Cited by 40
Melons: Generating Melody With Long-Term Structure Using Transformers And Structure Graph
Zou, Y. F., Zou, P., Zhao, Y., Zhang, K., Zhang, R., & Wang, X. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747802 · Cited by 42
Sparse Cross-Scale Attention Network for Efficient LiDAR Panoptic Segmentation
Xu, S., Wan, R., Ye, M., Zou, X., & Cao, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20197 · Cited by 40
Bhuiyan, M. S. M., Rafi, M. A., Rodrigues, G. N., Mir, M. N. H., Ishraq, A., Mridha, M. F., & Shin, J. (2025) · Array · doi:10.1016/j.array.2025.100390 · Cited by 65
Dong, Y., Jiang, H., Wang, X., & Mu, M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103340 · Cited by 38
Mokey: enabling narrow fixed-point inference for out-of-the-box floating-point transformer models
Zadeh, A. H., Mahmoud, M., Abdelhadi, A., & Moshovos, A. (2022) · Proceedings of the 49th Annual International Symposium on Computer Architecture · doi:10.1145/3470496.3527438 · Cited by 37
Adaptive Spot-Guided Transformer for Consistent Local Feature Matching
Yu, J., Chang, J., He, J., Zhang, T., Yu, J., & Wu, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02097 · Cited by 37
Transformer-Based Single-Cell Language Model: A Survey
Lan, W., He, G., Liu, M., Chen, Q., Cao, J., & Peng, W. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020034 · Cited by 40
Video Background Music Generation: Dataset, Method and Evaluation
Zhuo, L., Wang, Z., Wang, B., Liao, Y., Bao, C., Peng, S., Han, S., Zhang, A., Fang, F., & Liu, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01433 · Cited by 36
Hadamard Product in Deep Learning: Introduction, Advances and Challenges
Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 43
Combiner: Full Attention Transformer with Sparse Computation CostPreprint
Ren, H., Dai, H., Dai, Z., Yang, M., Leskovec, J., Schuurmans, D., & Dai, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05768 · Cited by 36
Efficient Vision Transformer via Token Merger
Feng, Z., & Zhang, S. S. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3293763 · Cited by 37
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
Lei, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.602 · Cited by 46
CATs++: Boosting Cost Aggregation With Convolutions and Transformers
Cho, S., Hong, S., & Kim, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3218727 · Cited by 42
Revisiting Token Pruning for Object Detection and Instance Segmentation
Liu, Y., Gehrig, M., Messikommer, N., Cannici, M., & Scaramuzza, D. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00264 · Cited by 54
Machine Translation Systems Based on Classical-Statistical-Deep-Learning Approaches
Sharma, S., Diwakar, M., Singh, P., Singh, V. P., Kadry, S., & Kim, J. (2023) · Electronics · doi:10.3390/electronics12071716 · Cited by 46
Token Pooling in Vision Transformers
Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., & Tuzel, O. (2021) · arXiv · doi:10.48550/arxiv.2110.03860 · Cited by 34
Yao, Y. F., Gui, G., Yang, S., & Zhang, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108240 · Cited by 37
vid-TLDR: Training Free Token merging for Light-Weight Video Transformer
Choi, J., Lee, S., Chu, J., Choi, M., & Kim, H. J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01776 · Cited by 35
Hierarchical Transformers Are More Efficient Language Models
Nawrot, P., Tworkowski, S., Tyrolski, M., Kaiser, Ł., Wu, Y., Szegedy, C., & Michalewski, H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.117 · Cited by 98
LSG Attention: Extrapolation of Pretrained Transformers to Long Sequences
Condevaux, C., & Harispe, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33374-3_35 · Cited by 40
CCVS: Context-aware Controllable Video SynthesisPreprint
Moing, G. L., Ponce, J., & Schmid, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08037 · Cited by 34
Geometrized Transformer for Self-Supervised Homography Estimation
Liu, J., & Li, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00876 · Cited by 32
Wang, S., Laskar, Z., Melekhov, I., Li, X., Zhao, Y., Tolias, G., & Kannala, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01982-9 · Cited by 33
MonoPGC: Monocular 3D Object Detection with Pixel Geometry Contexts
Wu, Z., Gan, Y., Wang, L., Chen, G., & Pu, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161442 · Cited by 32
The Devil in Linear Transformer
Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N. M., & Zhong, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.473 · Cited by 36
MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing
Marsocci, V., Scardapane, S., & Nikos Komodakis, A. (2021) · Remote Sensing · doi:10.3390/rs13163275 · Cited by 31
Zhang, Z., Huang, X., & Li, J. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3241366 · Cited by 32
A Survey of Deep Learning Approaches for OCR and Document UnderstandingPreprint
Subramani, N., Matton, A., Greaves, M. W., & Lam, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13534 · Cited by 112
Automated identification of cell populations in flow cytometry data with transformers
Wödlinger, M., Reiter, M., Weijler, L., Maurer‐Granofszky, M., Schumich, A., Sajaroff, E. O., Groeneveld‐Krentz, S., Rossi, J. G., Karawajew, L., Ratei, R., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105314 · Cited by 32
Metadata
- Authors
- Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- University of Washington (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Volume
- 1
- Pages
- 5156-5165
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16236
- OpenAlex
- W3034573343
- Cited by
- 827
- References
- 0
- Citations by year
- 2026: 5 · 2025: 149 · 2024: 202 · 2023: 188 · 2022: 125 · 2021: 135 · 2020: 23
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- linear transformer, linear attention, self-attention, autoregressive transformers, recurrent neural networks, quadratic complexity, iterative implementation, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
