Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 401–500 of 1,010
Resource-Efficient Separation Transformer
Libera, L. D., Subakan, C., Ravanelli, M., Cornell, S., Lepoutre, F., & Grondin, F. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446670 · Cited by 40
Real-time prediction of TBM penetration rates using a transformer-based ensemble deep learning model
Zhang, M., Ji, A., Zhou, C., Ding, Y., & Wang, L. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105793 · Cited by 40
Biomedical Data and Deep Learning Computational Models for Predicting Compound-Protein Relations
Zhao, Q., Yang, M., Cheng, Z. A., Li, Y., & Wang, J. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3069040 · Cited by 40
Context‐aware trajectory prediction for autonomous driving in heterogeneous environments
Li, Z., Chen, Z., Li, Y., & Xu, C. (2023) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.12989 · Cited by 40
pNLP-Mixer: an Efficient all-MLP Architecture for Language
Fusco, F., Pascual, D., Staar, P., & Antognini, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.6 · Cited by 40
Haugsdal, E., Aune, E., & Ruocco, M. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04927-4 · Cited by 40
Scatterbrain: Unifying Sparse and Low-rank Attention ApproximationPreprint
Chen, B., Dao, T., Winsor, E., Song, Z., Rudra, A., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15343 · Cited by 40
How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System?
Papi, S., Polák, P., Macháček, D., & Bojar, O. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00740 · Cited by 39
State-space modeling in long sequence processing: a survey on recurrence in the transformer era
Tiezzi, M., Casoni, M., Betti, A., Gori, M., & Melacci, S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108039 · Cited by 39
Building Blocks for a Complex-Valued Transformer Architecture
Eilers, F., & Jiang, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095349 · Cited by 38
Hacking, C., Verbeek, H., Hamers, J. P. H., & Aarts, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292578 · Cited by 38
Challenges in Context-Aware Neural Machine Translation
Jin, L., He, J., May, J., & Ma, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.943 · Cited by 38
Li, W., Hu, A., Xu, N., & He, G. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3350661 · Cited by 38
Lin, C., Chen, X., Guo, W., Jiang, N., Farina, D., & Su, J. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3216528 · Cited by 38
Transformer-based deep learning model for forced oscillation localization
Matar, M., Estevez, P. G., Marchi, P., Messina, F. J., Elmoudi, R., & Wshah, S. (2022) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2022.108805 · Cited by 38
A Voyage on Neural Machine Translation for Indic Languages
Sheshadri, S. K., Gupta, D., & Costa‐jussà, M. R. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.242 · Cited by 38
Hierarchical Wi-Fi Trajectory Embedding for Indoor User Mobility Pattern Analysis
Zhang, Q., Zhu, H., Wang, P., Chen, E., & Xiong, H. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3596237 · Cited by 38
Clinical Relation Extraction Using Transformer-based ModelsPreprint
Yang, X., Yu, Z., Guo, Y., Bian, J., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08957 · Cited by 38
L2MAC: Large Language Model Automatic Computer for Extensive Code Generation
Holt, S., Luyten, M. R., & van der Schaar, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.02003 · Cited by 37
PReFormer: A memory-efficient transformer for point cloud semantic segmentation
Akwensi, P. H., Wang, R., & Guo, B. (2024) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2024.103730 · Cited by 37
Unified Spatial–Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction
Li, R., Qiao, T., Katsigiannis, S., Zhu, Z., & Shum, H. P. H. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3539522 · Cited by 37
CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
Li, R., Zhang, Z., Bo, X., Tian, Z., Chen, X., Dai, Q., Dong, Z., & Tang, R. (2025) · neural information processing systems · doi:10.52202/085713-3781 · Cited by 37
Ngo, T. T. A., Vo, T. T. A., & Phan, M. T. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.105789 · Cited by 37
Machine learning global atomic representations with Euclidean fast attention
Frank, J. T., Chmiela, S., Müller, K.-R., & Unke, O. T. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01195-y · Cited by 37
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.346 · Cited by 37
Fiok, K., Karwowski, W., Gutiérrez-Franco, E., Davahli, M. R., Wilamowski, M., Ahram, T., Aljuaid, A. M., & Zurada, J. M. (2021) · IEEE Access · doi:10.1109/access.2021.3099758 · Cited by 37
Extended context-based semantic communication system for text transmission
Liu, Y., Jiang, S., Zhang, Y., Cao, K., Zhou, L., Seet, B., Zhao, H., & Wei, J. (2022) · Digital Communications and Networks · doi:10.1016/j.dcan.2022.09.023 · Cited by 37
Zhu, Q., Duan, J., Chen, C., Liu, S., Feng, G., Lv, X., Chuanfu, X., Lin, D., & Chao, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.15486 · Cited by 37
Zhang, J., Fan, Y., Cai, K., Huang, Z., Sun, X., Wang, J., Tang, C., & Wang, K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02785 · Cited by 37
Dong, S., Meng, Y., Yin, S., & Liu, X. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112473 · Cited by 36
Foundation models and Transformers for anomaly detection: A survey
Ammar, M. B., Mendoza, A., Belkhir, N., Manzanera, A., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103517 · Cited by 36
Knowledge Aggregation Transformer Network for Multivariate Time Series Classification
Xiao, Z., Xing, H., Qu, R., Li, H., Tong, H., Luo, S., Song, J., Feng, L., & Wan, Q. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3594294 · Cited by 36
Mixture of Nested Experts: Adaptive Processing of Visual Tokens
Jain, G., Hegde, N., Kusupati, A., Nagrani, A., Buch, S., Jain, P., Arnab, A., & Paul, S. (2024) · neural information processing systems · doi:10.52202/079017-1863 · Cited by 36
Attention Meets Perturbations: Robust and Interpretable Attention With Adversarial Training
Kitada, S., & Iyatomi, H. (2021) · IEEE Access · doi:10.1109/access.2021.3093456 · Cited by 36
Learning to Retrieve User Behaviors for Click-through Rate Estimation
Qin, J., Zhang, W., Su, R., Liu, Z., Liu, W., Zhao, G., Li, H., Tang, R., He, X., & Yu, Y. H. (2023) · ACM Transactions on Information Systems · doi:10.1145/3579354 · Cited by 36
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
He, Z., Yang, M., Feng, M., Yin, J., Wang, X., Leng, J., & Lin, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.570 · Cited by 36
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency EncodingPreprint
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06097 · Cited by 36
Code Execution with Pre-trained Language ModelsPreprint
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05383 · Cited by 36
ReaderLM-v2: Small Language Model for HTML to Markdown and JSONPreprint
Wang, F., Shi, Z., Wang, B., Wang, N., & Xiao, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.01151 · Cited by 36
DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Ben-Kish, A., Zimerman, I., Abu-Hussein, S., Cohen, N., Globerson, A., Wolf, L., & Giryes, R. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2406.14528 · Cited by 37
MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
Chou, Y., Yao, M., Wang, K., Pan, Y., Zhu, R., Zhong, Y., Qiao, Y., Wu, J., Xu, B., & Li, G. (2024) · neural information processing systems · doi:10.52202/079017-2270 · Cited by 35
Block-Skim: Efficient Question Answering for Transformer
Guan, Y., Li, Z., Lin, Z., Zhu, Y., Leng, J., & Guo, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21316 · Cited by 35
Data-driven modeling of power generation for a coal power plant under cycling
Sharma, H., Marinovici, L. D., Adetola, V., & Schaef, H. T. (2022) · Energy and AI · doi:10.1016/j.egyai.2022.100214 · Cited by 35
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34
Self-citationToken Pooling in Vision Transformers
Marin, D., Chang, J.-H. R., Ranjan, A., Prabhu, A., Rastegari, M., & Tuzel, O. (2021) · arXiv · doi:10.48550/arxiv.2110.03860 · Cited by 34
“While most prior work in Natural Language Processing (NLP) improve the time-complexity of attention (Tay et al., 2020b; Ilharco et al., 2020), in vision transformers the main computation bottleneck is the fully-connected layers, as we show in §3.1.”
BackgroundInfluentialH-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
Zhu, Z., & Soricut, R. (2021) · doi:10.18653/v1/2021.acl-long.294 · Cited by 34
A Unified Framework for Real Time Motion Completion
Duan, Y., Lin, Y., Zou, Z., Yuan, Y. G., Qian, Z., & Zhang, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20368 · Cited by 34
Jiang, J., Zhang, R., Ma, J., Liu, Y., Yang, E., Du, S., Zhao, Z., & Yuan, Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04280-y · Cited by 34
Challenges in Domain-Specific Abstractive Summarization and How to Overcome Them
Afzal, A., Vladika, J., Braun, D., & Matthes, F. (2023) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0011744500003393 · Cited by 34
CCBERT: Self-Supervised Code Change Representation Learning
Zhou, X., Xu, B., Han, D., Yang, Z., He, J., & Lo, D. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00028 · Cited by 34
GatorTron: A Large Language Model for Clinical Natural Language Processing
Yang, X., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2022) · doi:10.1101/2022.02.27.22271257 · Cited by 33
Personalizing simulation-based learning in higher education
Bauer, E., Heitzmann, N., Bannert, M., Chernikova, O., Fischer, M. R., Frenzel, A. C., Gartmeier, M., Hofer, S. I., Holzberger, D., Kasneci, E., et al. (2025) · Learning and Individual Differences · doi:10.1016/j.lindif.2025.102746 · Cited by 33
Lin, Z., Zang, S., Wang, R., Sun, Z., Senthilnath, J., Xu, C. Y., & Kwoh, C. K. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3208633 · Cited by 33
Exploring Dimensionality Reduction Techniques in Multilingual Transformers
Huertas-García, Á., Martín, A. G., Huertas‐Tato, J., & Camacho, D. (2022) · Cognitive Computation · doi:10.1007/s12559-022-10066-8 · Cited by 33
M-GaitFormer: Mobile biometric gait verification using Transformers
Delgado-Santos, P., Tolosana, R., Guest, R., Vera-Rodríguez, R., & Fiérrez, J. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106682 · Cited by 33
Adapting Pretrained Text-to-Text Models for Long Text Sequences
Xiong, W., Gupta, A., Toshniwal, S., Mehdad, Y., & Yih, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.370 · Cited by 33
Representation learning for neural population activity with Neural Data TransformersPreprint
Ye, J., & Pandarinath, C. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.16.426955 · Cited by 33
OmniNet: Omnidirectional Representations from TransformersPreprint
Tay, Y., Dehghani, M., Aribandi, V., Gupta, J. P., Pham, P., Qin, Z., Bahri, D., Juan, D.-C., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.01075 · Cited by 33
Self-citationNonparametric Modern Hopfield ModelsPreprint
Hu, J. Y.-C., Chen, B.-Y., Wu, D., Ruan, F., & Liu, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.03900 · Cited by 33
Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation
Rondanini, C., Carminati, B., Ferrari, E., Kundu, A., & Gaudiano, A. (2025) · ACM Transactions on Internet Technology · doi:10.1145/3769681 · Cited by 32
Learning accurate template matching with differentiable coarse-to-fine correspondence refinement
Gao, Z., Yi, R., Qin, Z., Ye, Y., Zhu, C., & Xu, K. (2024) · Computational Visual Media · doi:10.1007/s41095-023-0333-9 · Cited by 32
MATE: Multi-view Attention for Table Transformer EfficiencyPreprint
Eisenschlos, J. M., Gor, M., Müller, T., & Cohen, W. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04312 · Cited by 32
Exploring Extreme Parameter Compression for Pre-trained Language ModelsPreprint
Ren, Y., Wang, B., Shang, L., Jiang, X., & Liu, Q. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10036 · Cited by 32
Efficient Long Sequential User Data Modeling for Click-Through Rate PredictionPreprint
Chen, Q., Xu, Y., Pei, C., Lv, S., Zhuang, T., & Ge, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.12212 · Cited by 32
Phenotype prediction from single-cell RNA-seq data using attention-based neural networks
Mao, Y., Lin, Y.-Y., Wong, N. K. Y., Volik, S., Sar, F., Collins, C., & Ester, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae067 · Cited by 31
“Considering its poor scalability (Tay et al. 2020), a cropping strategy is applied to the input sample before passing it to the attention layer.”
MethodOn the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
Ji, T., Jain, S., Ferdman, M., Milder, P. A., Schwartz, H. A., & Balasubramanian, N. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.363 · Cited by 31
Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models
Lupo, L., Dinarelli, M., & Besacier, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.312 · Cited by 31
Zhou, Q., Wang, H., Yu, X., Li, C., Bai, Y., Yan, F., & Xu, Y. (2023) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071077 · Cited by 31
RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models
Li, Y., Li, Z., Yang, W. P., & Liu, C. (2023) · IEEE Real-Time Systems Symposium (RTSS) · doi:10.1109/rtss59052.2023.00023 · Cited by 31
Transformers for Multi-Object Tracking on Point Clouds
Ruppel, F., Faion, F., Gläser, C., & Dietmayer, K. (2022) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv51971.2022.9827344 · Cited by 31
In Defense of RAG in the Era of Long-Context Language Models
Yu, T., Xu, A., & Akkiraju, R. (2024) · arXiv · doi:10.48550/arxiv.2409.01666 · Cited by 30
“…models, many efforts have been devoted to improving the computing efficiency of self-attention (Choromanski et al., 2020; Zaheer et al., 2020; Tay et al., 2020; Dao et al., 2022; Dao, 2024) and boosting extensibility of positional encoding (Press et al., 2021; Sun et al., 2022; Chen et al.,…”
BackgroundLearning to Compress Prompts with Gist Tokens
Mu, J., Li, X. (., & Goodman, N. D. (2023) · neural information processing systems · doi:10.52202/075280-0848 · Cited by 30
Protecting Intellectual Property of Large Language Model-Based Code Generation APIs via Watermarks
Li, Z., Wang, C., Wang, S., & Gao, C. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623120 · Cited by 30
Ortloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 30
Even Sparser Graph Transformers
Shirzad, H., Lin, H., Venkatachalam, B., Velingker, A., Woodruff, D. P., & Sutherland, D. J. (2024) · neural information processing systems · doi:10.52202/079017-2277 · Cited by 30
Collaborative Feature Learning for Gait Recognition Under Cloth Changes
Yao, L., Kusakunniran, W., Wu, Q., Xu, J., & Zhang, J. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3112564 · Cited by 30
Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
Zhao, R., Yu, J., Zhao, H., & Ngai, E. C. ‐. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610873 · Cited by 30
Softmax-Free Linear Transformers
Lu, J., Zhang, J., Zhu, X., Feng, J., Xiang, T., & Zhang, L. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02035-5 · Cited by 30
NLQuAD: A Non-Factoid Long Question Answering Data Set
Soleimani, A., Monz, C., & Worring, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.106 · Cited by 30
Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories
Wilmot, D., & Keller, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.65 · Cited by 30
Deploying Foundation Model Powered Agent Services: A Survey
Xu, W., Chen, J., Zheng, P., Yi, X., Tian, T., Zhu, W., Wan, Q., Wang, H., Fan, Y., Su, Q., et al. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3580745 · Cited by 29
Xu, H., Zlatanova, S., Liang, R., & Canbulat, I. (2025) · Fire · doi:10.3390/fire8080293 · Cited by 30
A Survey on Spatio-Temporal Prediction: From Transformers to Foundation Models
Mao, Y., Zhou, H., Chen, L., Qi, R., Sun, Z., Rong, Y., He, X., Chen, M., Mumtaz, S., Frascolla, V., et al. (2025) · ACM Computing Surveys · doi:10.1145/3766546 · Cited by 29
Multimodal foundation transformer models for multiscale genomics
Khan, S. A., Martinez‐de‐Morentin, X., Alsabbagh, A. R., Maillo, A., Lagani, V., Gómez-Cabrero, D., Lehmann, R., & Tegner, J. N. (2025) · Nature Methods · doi:10.1038/s41592-025-02918-6 · Cited by 29
Wang, Q., Deng, H., Wu, X. M., Yang, Z., Liu, Y., Wang, Y., & Hao, G. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.010 · Cited by 29
UCONV-Conformer: High Reduction of Input Sequence Length for End-to-End Speech Recognition
Andrusenko, A., Nasretdinov, R., & Romanenko, A. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095430 · Cited by 29
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences
Li, J., Aitken, W., Bhambhoria, R. V., & Zhu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.120 · Cited by 29
SMARTAVE: Structured Multimodal Transformer for Product Attribute Value Extraction
Wang, Q., Li, Y., Wang, J., Krishnan, J., Dai, B., Wang, S., Xu, Z., Khabsa, M., & Ma, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.20 · Cited by 29
Ding, F., Yuan, Y., Lv, L., Zhang, R., & Zhou, W. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3442997 · Cited by 29
$\infty$-former: Infinite Memory TransformerPreprint
Martins, P. H., Marinho, Z., & Martins, A. F. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00301 · Cited by 29
FIT: Far-reaching Interleaved TransformersPreprint
chen, T., & Li, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.12689 · Cited by 29
Using Temporal Deep Learning Models to Estimate Daily Snow Water Equivalent Over the Rocky Mountains
Duan, S., Ullrich, P. A., Risser, M. D., & Rhoades, A. M. (2024) · Water Resources Research · doi:10.1029/2023wr035009 · Cited by 28
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers
Nauen, T. C., Palacio, S., Raue, F., & Dengel, A. R. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00676 · Cited by 28
Towards large language models with human-like episodic memory
Dong, C., Lu, Q., Norman, K. A., & Michelmann, S. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.06.016 · Cited by 28
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Chen, Y., Zhang, L., Shang, J., Zhang, Z., Liu, T., Wang, S., & Sun, Y. (2024) · neural information processing systems · doi:10.52202/079017-1459 · Cited by 28
Gu, F., Jiang, Z., García‐Fernández, Á. F., Stefanidis, A., Su, J., & Li, H. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261416593 · Cited by 28
Cross-layer Attention Sharing for Pre-trained Large Language Models
Mu, Y.-Y., Wu, Y., Fan, Y.-C., Wang, C., Li, H.-Y., Zeng, J.-L., He, Q.-Z., Yang, M.-R., Meng, F.-D., Zhou, J., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.616 · Cited by 10
Triplet Attention: Rethinking the Similarity in Transformers
Zhou, H., Li, J., Peng, J., Zhang, S., & Zhang, S. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467241 · Cited by 28
Poly-NL: Linear Complexity Non-local Layers With 3rd Order Polynomials
Babiloni, F., Marras, I., Kokkinos, F., Deng, J., Chrysos, G. G., & Zafeiriou, S. P. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01035 · Cited by 28
ABC: Attention with Bounded-memory Control
Peng, H., Kasai, J., Pappas, N., Yogatama, D., Wu, Z., Kong, L., Schwartz, R., & Smith, N. A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.515 · Cited by 28
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.