Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 301–400 of 1,010
Are Pretrained Convolutions Better than Pretrained Transformers?
Tay, Y., Dehghani, M., Gupta, J. P., Aribandi, V., Bahri, D., Qin, Z., & Metzler, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.335 · Cited by 56
Self-citationCharacterization of MPC-based Private Inference for Transformer-based Models
Wang, Y., Suh, G. E., Xiong, W., Lefaudeux, B., Knott, B., Annavaram, M., & Lee, H.-H. S. (2022) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass55109.2022.00025 · Cited by 56
A hybrid partitioned deep learning methodology for moving interface and fluid–structure interaction
Gupta, R., & Jaiman, R. K. (2021) · Computers & Fluids · doi:10.1016/j.compfluid.2021.105239 · Cited by 56
Shortformer: Better language modeling using shorter inputs
Press, O., Smith, N. A., & Lewis, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.427 · Cited by 55
Efficient Streaming Language Models with Attention Sinks
Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023) · arXiv · doi:10.48550/arxiv.2309.17453 · Cited by 55
LooGLE: Can Long-Context Language Models Understand Long Contexts?
Li, J., Wang, M., Zheng, Z., & Zhang, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.859 · Cited by 55
Peng, H., Jiang, B., Mao, Z., & Liu, S. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3291787 · Cited by 55
Transformer fusion for indoor RGB-D semantic segmentation
Wu, Z., Zhou, Z., Allibert, G., Stolz, C., Demonceaux, C., & Ma, C. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104174 · Cited by 55
Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers
Jiang, Z., Gu, J., Zhu, H., & Pan, D. Z. (2023) · neural information processing systems · doi:10.52202/075280-1983 · Cited by 55
Recurrent Glimpse-based Decoder for Detection with Transformer
Chen, Z., Zhang, J., & Tao, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00519 · Cited by 55
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) InferencePreprint
Huang, H., Ardalani, N., Sun, A., Liu, K., Lee, H.-H. S., Sridhar, A., Bhosale, S., Wu, C.-J., & Lee, B. C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.06182 · Cited by 55
BERT-based Ensemble Approaches for Hate Speech Detection
Mnassri, K., Rajapaksha, P., Farahbakhsh, R., & Crespi, N. (2022) · Globecom · doi:10.1109/globecom48099.2022.10001325 · Cited by 54
STGlow: A Flow-Based Generative Framework With Dual-Graphormer for Pedestrian Trajectory Prediction
Liang, R., Li, Y., Zhou, J., & Li, X. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3294998 · Cited by 54
He, B., Armaghani, D. J., Lai, S. H., Samui, P., & Mohamad, E. T. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121616 · Cited by 54
Transformers for Trajectory Optimization with Application to Spacecraft Rendezvous
Guffanti, T., Gammelli, D., D’Amico, S., & Pavone, M. (2024) · IEEE Aerospace Conference (AERO) · doi:10.1109/aero58975.2024.10521334 · Cited by 54
Machine learning interatomic potentials at the centennial crossroads of quantum mechanics
Kalita, B., Gökcan, H., & Isayev, O. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00930-6 · Cited by 54
SHAPE: Shifted Absolute Position Embedding for Transformers
Kiyono, S., Kobayashi, S., Suzuki, J., & Inui, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.266 · Cited by 54
My Body is a Cage: the Role of Morphology in Graph-Based Incompatible ControlPreprint
Kurin, V., Igl, M., Rocktäschel, T., Böhmer, W., & Whiteson, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01856 · Cited by 54
Dynamic N:M Fine-Grained Structured Sparse Attention Mechanism
Chen, Z., Qu, Z., Quan, Y., Liu, L., Ding, Y., & Xie, Y. (2023) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3572848.3577500 · Cited by 53
PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers
Grainger, R., Paniagua, T., Song, X., Cuntoor, N. P., Lee, M. W., & Wu, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01781 · Cited by 53
Enhancing hydrological modeling with transformers: a case study for 24-h streamflow prediction
Demiray, B. Z., Sit, M. A., Mermer, O., & Demir, İ. (2024) · Water Science & Technology · doi:10.2166/wst.2024.110 · Cited by 53
Transformer encoder based self-supervised learning for HVAC fault detection with unlabeled data
Abdollah, M. A. F., Scoccia, R., & Aprile, M. (2024) · Building and Environment · doi:10.1016/j.buildenv.2024.111568 · Cited by 53
Transkimmer: Transformer Learns to Layer-wise Skim
Guan, Y., Li, Z., Leng, J., Lin, Z., & Guo, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.502 · Cited by 53
Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender SystemsPreprint
Gui, H., Wang, R., Yin, K., Jin, L., Kula, M., Xu, T., Hong, L., & Ed, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.05884 · Cited by 53
Quantum Transformer: Accelerating model inference via quantum linear algebraPreprint
Guo, N., Yu, Z., Choi, M., Han, Y., Agrawal, A., Nakaji, K., Aspuru‐Guzik, A., & Rebentrost, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16714 · Cited by 53
Foundation Models in Medical Imaging: A Review and OutlookPreprint
van Veldhuizen, V., Botha, V., Lu, C., Cesur, M. E., Lipman, K. B. W. G., de Jong, E. D., Horlings, H. M., Sánchez, C., Snoek, C. G. M., Wessels, L. F. A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.09095 · Cited by 54
A Deformable Attention Network for High-Resolution Remote Sensing Images Semantic Segmentation
Zuo, R., Zhang, G., Zhang, R., & Jia, X. (2021) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3119537 · Cited by 52
Deformable Mesh Transformer for 3D Human Mesh Recovery
Yoshiyasu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01631 · Cited by 53
Fast geometric learning with symbolic matrices
Feydy, J., Glaunès, J., Charlier, B., & Bronstein, M. M. (2020) · HAL · Cited by 52
Allam, T., & McEwen, J. D. (2023) · RAS Techniques and Instruments · doi:10.1093/rasti/rzad046 · Cited by 52
Expediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning
Liang, W., Yuan, Y., Ding, H., Luo, X., Lin, W., Jia, D., Zhang, Z., Zhang, C., & Hu, H. (2022) · neural information processing systems · doi:10.52202/068431-2570 · Cited by 52
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591687 · Cited by 51
Code Execution with Pre-trained Language Models
Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.308 · Cited by 51
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
Fu, D. Y., Kumbong, H., Nguyen, E., & Ré, C. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.05908 · Cited by 51
Machine learning for full lifecycle management of lithium-ion batteries
Zhai, Q., Jiang, H., Long, N., Kang, Q., Meng, X., Zhou, M., Yan, L., & Ma, T. (2024) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2024.114647 · Cited by 51
DBMLLA: Double-Branch Mamba-Like Linear Attention Network for Hyperspectral Image Classification
Liang, L., Xie, P., Zhang, Y., Li, J., Zhang, Z., Li, J., & Plaza, A. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3605798 · Cited by 51
Kandathil, S. M., Greener, J. G., Lau, A. M., & Jones, D. T. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2113348119 · Cited by 38
Jiang, F., Xu, M., Li, Y., Cui, H., & Wang, R. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106358 · Cited by 50
Giabbanelli, P. J. (2023) · Winter Simulation Conference · doi:10.1109/wsc60868.2023.10408017 · Cited by 50
Graph Neural Network contextual embedding for Deep Learning on tabular data
Villaizán-Vallelado, M., Salvatori, M., Carro, B., & Sanchez-Esguevillas, A. J. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106180 · Cited by 50
DeepGate3: Towards Scalable Circuit Representation Learning
Shi, Z., Zheng, Z., Khan, S., Zhong, J., Li, M., & Xu, Q. (2024) · International Conference on Computer Aided Design · doi:10.1145/3676536.3676791 · Cited by 50
Long-Span Summarization via Local Attention and Content Selection
Manakul, P., & Gales, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.470 · Cited by 50
ICFormer: A Deep Learning model for informed lithium-ion battery diagnosis and early knee detection
Costa, N., Anseán, D., Dubarry, M., & Sánchez, L. (2023) · Journal of Power Sources · doi:10.1016/j.jpowsour.2023.233910 · Cited by 49
When Meta-Learning Meets Online and Continual Learning: A Survey
Son, J., Lee, S., & Kim, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3463709 · Cited by 49
Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences
Ibing, M., Kobsik, G., & Kobbelt, L. P. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00270 · Cited by 49
BOAT: Bilateral Local Attention Vision TransformerPreprint
Yu, T., Zhao, G., Li, P., & Yu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.13027 · Cited by 49
Transformers Meet Visual Learning Understanding: A Comprehensive ReviewPreprint
Yang, Y., Jiao, L., Liu, X., Liu, F., Yang, S., Feng, Z., & Tang, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.12944 · Cited by 49
Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image AnalysisPreprint
Heidari, M., Kolahi, S. G., Karimijafarbigloo, S., Azad, B., Bozorgpour, A., Hatami, S., Azad, R., Diba, A., Bağcı, U., Merhof, D., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.03430 · Cited by 49
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48
Self-citationWu, Y., Zhao, S., Qi, S., Feng, J., Pang, H., Chang, R., Bai, L., Li, M., Xia, S., Qian, W., et al. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102637 · Cited by 48
López, E. O., Kaplan, D., & Linhoss, A. C. (2024) · Water Resources Research · doi:10.1029/2023wr036337 · Cited by 48
Back to recurrent processing at the crossroad of transformers and state-space models
Tiezzi, M., Casoni, M., Betti, A., Guidi, T., Gori, M., & Melacci, S. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01034-6 · Cited by 48
Xiong, T., He, J., Wang, H., Tang, X., Shi, Z., & Zeng, Q. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3128522 · Cited by 48
BayesFormer: Transformer with Uncertainty EstimationPreprint
Sankararaman, K. A., Wang, S., & Fang, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.00826 · Cited by 48
Prompt Injection: Parameterization of Fixed InputsPreprint
Choi, E., Jo, Y., Jang, J., & Seo, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11349 · Cited by 48
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory ConstraintsPreprint
Ao, R., Luo, G., Simchi‐Levi, D., & Wang, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5195463 · Cited by 48
Somvanshi, S., Islam, M. M., Mimi, M. S., Polock, S. B. B., Chhetri, G., Dutta, A., Rafe, A., & Das, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18970 · Cited by 48
Xu, X., & Sankar, R. T. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8100127 · Cited by 47
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström MethodPreprint
Chen, Y., Zeng, Q., Ji, H., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00035 · Cited by 47
MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal ModelsPreprint
Zhang, J., Cai, K., Guo, X., Liu, S., Lv, Q., Chen, R., Yang, J., Fan, Y., Sun, X., Wang, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.08228 · Cited by 47
Liu, J., Zhao, Y., Feng, Y., Hu, Y., & Ma, X. (2023) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2023.103690 · Cited by 46
Efficient attention mechanisms for large language models
Sun, Y., Li, Z., Zhang, Y., Pan, T., Dong, B., Guo, Y., & Wang, J. (2026) · Patterns · doi:10.1016/j.patter.2026.101594 · Cited by 46
LW-CMDANet: A Novel Attention Network for SAR Automatic Target Recognition
Lang, P., Fu, X., Cheng, F., Dong, J., Qin, R., & Martorella, M. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3195074 · Cited by 46
Leveraging redundancy in attention with Reuse TransformersPreprint
Bhojanapalli, S., Chakrabarti, A., Veit, A., Łukasik, M., Jain, H., Liu, F., Chang, Y.-W., & Kumar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06821 · Cited by 46
NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time
Chen, Y., Wang, G., Shang, J., Cui, S., Zhang, Z., Liu, T., Wang, S., Sun, Y., Yu, D., & Wu, H. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.428 · Cited by 45
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models
Ignat, O., Jin, Z., Abzaliev, A., Biester, L., Castro, S., Deng, N., Gao, X., Gunal, A. E., He, J., Kazemi, A., et al. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2bw7xfcwyeq3 · Cited by 45
Kumar, Y., Shah, J., Shah, R. R., & Chen, C. (2021) · Preprints.org · doi:10.20944/preprints202101.0081.v1 · Cited by 46
STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining
Guo, L., Choe, W., & Lin, F. X. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3575693.3575698 · Cited by 44
C2FTFNet: Coarse-to-fine transformer network for joint optic disc and cup segmentation
Yi, Y., Jiang, Y., Zhou, B., Zhang, N., Dai, J., Huang, X., Zeng, Q., & Zhou, W. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107215 · Cited by 44
Kroon, A. C., Welbers, K., Trilling, D., & van Atteveldt, W. (2023) · Communication Methods and Measures · doi:10.1080/19312458.2023.2261372 · Cited by 44
Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
Liu, S., Deng, N., Sabour, S., Jia, Y., Huang, M., & Mihalcea, R. F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.944 · Cited by 44
Ensemble of temporal Transformers for financial time series
Olorunnimbe, K., & Viktor, H. L. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00851-2 · Cited by 44
Scaling Language Model Size in Cross-Device Federated Learning
Ro, J., Breiner, T., McConnaughey, L., Chen, M., Suresh, A., Kumar, S., & Mathews, R. (2022) · doi:10.18653/v1/2022.fl4nlp-1.2 · Cited by 44
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language ModelsPreprint
Kaddour, J., Key, O., Nawrot, P., Minervini, P., & Kusner, M. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.06440 · Cited by 44
Speed Always Wins: A Survey on Efficient Architectures for Large Language ModelsPreprint
Sun, W., Hu, J., Zhou, Y., Du, J., Lan, D., Wang, K., Zhu, T., Qu, X., Zhang, Y., Mo, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.09834 · Cited by 44
MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers
Havtorn, J. D., Royer, A., Blankevoort, T., & Bejnordi, B. E. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00091 · Cited by 43
Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges
Becker, J., Wahle, J. P., Gipp, B., & Ruas, T. (2026) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.22258 · Cited by 43
Multimodal market information fusion for stock price trend prediction in the pharmaceutical sector
Wang, H., Xie, Z., Chiu, D. K. W., & Ho, K. K. W. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05894-0 · Cited by 43
Improving the Efficiency of Transformers for Resource-Constrained Devices
Tabani, H., Balasubramaniam, A., Marzban, S., Arani, E., & Zonooz, B. (2021) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd53832.2021.00074 · Cited by 43
Designing Large Foundation Models for Efficient Training and Inference: A SurveyPreprint
Liu, D., Yu, Y., Wang, Y., Wu, J., Wan, Z., Alinejad, S., Lengerich, B. J., & Wu, Y. N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.01990 · Cited by 43
Efficient long-text understanding with short-text models
Ivgi, M., Shaham, U., & Berant, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00547 · Cited by 42
“For a survey on the families of efficient transformers, see Tay et al. (2020).”
BackgroundInfluentialRoy, R. S., & Anand, A. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s0113ed1v01y202109icr076 · Cited by 42
Transformer-VQ: Linear-Time Transformers via Vector Quantization
Lingle, L. D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.16354 · Cited by 42
TypeFormer: transformers for mobile keystroke biometrics
Stragapede, G., Delgado-Santos, P., Tolosana, R., Vera-Rodríguez, R., Guest, R., & Morales, A. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10140-2 · Cited by 44
csKT: Addressing cold-start problem in knowledge tracing via kernel bias and cone attention
Bai, Y., Li, X., Liu, Z., Huang, Y., Guo, T., Hou, M., Xia, F., & Luo, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125988 · Cited by 42
On the Fundamental Limits of LLMs at Scale
Mohsin, M. A., Umer, M., Bilal, A., Memon, Z., Qadir, M. I., Bhattacharya, S., Rizwan, H., Gorle, A. R., Kazmi, M. Z., Amir, N., et al. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2511.12869 · Cited by 42
Wang, G., Ai, J., Mo, L., Yi, X., Wu, P., Wu, X., & Kong, L. (2023) · Drones · doi:10.3390/drones7050326 · Cited by 42
Efficient Transformers in Reinforcement Learning using Actor-Learner DistillationPreprint
Parisotto, E., & Salakhutdinov, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01655 · Cited by 42
Hybrid Token Compression for Vision-Language ModelsPreprint
Zhang, J., Guo, X., Mo, T., Lv, Q., Chai, W., Wang, J., Wang, K., & Lin, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.08240 · Cited by 42
Vision Transformers for Road Accident Detection from Dashboard Cameras
hajri, F., & Fradi, H. (2022) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss56176.2022.9959545 · Cited by 41
Adapting transformer-based language models for heart disease detection and risk factors extraction
Houssein, E. H., Mohamed, R. E., Hu, G., & Ali, A. A. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00903-y · Cited by 41
Large Language Models for Conducting Advanced Text Analytics Information Systems Research
Ampel, B., Yang, C.-H., Hu, J. L., & Chen, H. (2024) · ACM Transactions on Management Information Systems · doi:10.1145/3682069 · Cited by 41
Solvent Screening for Separation Processes Using Machine Learning and High-Throughput Technologies
Edaugal, J. P., Zhang, D., Liu, D., Glezakou, V., & Sun, N. (2025) · Chem & Bio Engineering · doi:10.1021/cbe.4c00170 · Cited by 41
Video Crowd Localization With Multifocus Gaussian Neighborhood Attention and a Large-Scale Benchmark
Li, H., Liu, L., Yang, K., Liu, S., Gao, J., Zhao, B., Zhang, R., & Hou, J. Y. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3205210 · Cited by 41
Mobile Keystroke Biometrics Using Transformers
Stragapede, G., Delgado-Santos, P., Tolosana, R., Vera-Rodríguez, R., Guest, R., & Morales, A. (2023) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg57933.2023.10042710 · Cited by 41
Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering
Caciularu, A., Peters, M. E., Goldberger, J., Dagan, I., & Cohan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.110 · Cited by 41
Faster Causal Attention Over Large Sequences Through Sparse Flash AttentionPreprint
Pagliardini, M., Paliotta, D., Jaggi, M., & Fleuret, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.01160 · Cited by 41
SCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40
Global and Local Spatio-Temporal Encoder for 3D Human Pose Estimation
Wang, Y., Kang, H., Wu, D., Yang, W., & Zhang, L. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321438 · Cited by 40
GRAformer: A gated residual attention transformer for multivariate time series forecasting
Yang, C., Wang, Y., Yang, B., & Chen, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127466 · Cited by 40
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.