Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 101–200 of 1,010
Deep reinforcement and transfer learning for abstractive text summarization: A review
Alomari, A., Idris, N. B., Sabri, A. Q. M., & Alsmadi, I. M. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101276 · Cited by 170
PointMixer: MLP-Mixer for Point Cloud Understanding
Choe, J., Park, C., Rameau, F., Park, J., & Kweon, I. S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_36 · Cited by 170
Anomaly Detection in Dynamic Graphs via Transformer
Liu, Y., Pan, S., Wang, Y. G., Xiong, F., Wang, L., Chen, Q., & Lee, V. C. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3124061 · Cited by 169
“For further details about transformers on NLP and CV please see related surveys [27], [28].”
BackgroundPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 169
Unlocking the potential of deep learning for marine ecology: overview, applications, and outlook
Goodwin, M., Halvorsen, K. T., Jiao, L., Knausgård, K. M., Martin, A. H., Moyano, M., Oomen, R. A., Rasmussen, J. H., Sørdalen, T. K., & Thorbjørnsen, S. H. (2021) · ICES Journal of Marine Science · doi:10.1093/icesjms/fsab255 · Cited by 169
ForecastPFN: Synthetically-Trained Zero-Shot Forecasting
Dooley, S. W., Khurana, G. S., Mohapatra, C., Naidu, S., & White, C. (2023) · neural information processing systems · doi:10.52202/075280-0112 · Cited by 169
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer
Baade, A., Peng, P., & Harwath, D. F. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10961 · Cited by 168
SpecTr: Fast Speculative Decoding via Optimal Transport
Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., & Yu, F. (2023) · neural information processing systems · doi:10.52202/075280-1314 · Cited by 168
SELFormer: molecular representation learning via SELFIES language models
Yüksel, A., Ulusoy, E., Ünlü, A., & Doğan, T. (2023) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/acdb30 · Cited by 81
RealFormer: Transformer Likes Residual Attention
He, R., Ravula, A., Kanagal, B., & Ainslie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.81 · Cited by 163
Shehzad, A. H., Xia, F., Abid, S., Peng, C., Yu, S., Zhang, D., & Verspoor, K. M. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3646122 · Cited by 161
Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D. (2024) · Proceedings of the 17th ACM International Conference on Web Search and Data Mining · doi:10.1145/3616855.3635752 · Cited by 155
A Comprehensive Survey of Transformers for Computer Vision
Jamil, S., Piran, M. J., & Kwon, O. (2023) · Drones · doi:10.3390/drones7050287 · Cited by 155
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
Are we ready for a new paradigm shift? A survey on visual deep MLP
Liu, R., Li, Y., Tao, L., Liang, D., & Zheng, H.-T. (2022) · Patterns · doi:10.1016/j.patter.2022.100520 · Cited by 153
Deep learning for hate speech detection: a comparative study
Malik, J. S., Qiao, H., Pang, G., & van den Hengel, A. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00650-6 · Cited by 151
Deep learning for computational cytology: A survey
Jiang, H., Zhou, Y., Lin, Y., Chan, R., Liu, J., & Chen, H. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102691 · Cited by 149
Hierarchical Graph Transformer with Adaptive Node Sampling
Zhang, Z., Liu, Q., Hu, Q., & Lee, C. (2022) · neural information processing systems · doi:10.52202/068431-1539 · Cited by 147
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep LearningPreprint
Kossen, J., Band, N., Lyle, C., Gomez, A. N., Rainforth, T., & Gal, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02584 · Cited by 147
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146
An Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 146
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145
Multi-Frame Self-Supervised Depth with Transformers
Guizilini, V., Ambruş, R., Chen, D., Zakharov, S., & Gaidon, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00026 · Cited by 144
Jafari, M., Shoeibi, A., Khodatars, M., Ghassemi, N., Moridian, P., Alizadehsani, R., Khosravi, A., Ling, S. H., Delfan, N., Zhang, Y., et al. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106998 · Cited by 144
Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach
Wang, Z., Han, H., Wang, L., Li, X., & Zhou, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3171661 · Cited by 108
LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
Liu, L., Cai, L., Zhang, C., Zhao, X., Gao, J., Wang, W., Lv, Y., Fan, W., Wang, Y., He, M. X., et al. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591717 · Cited by 143
Prompt Compression for Large Language Models: A Survey
Li, Z., Liu, Y., Su, Y., & Collier, N. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.368 · Cited by 143
Antonesi, G., Cioara, T., Anghel, I. M., Michalakopoulos, V., Sarmas, E., & Toderean, L. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126670 · Cited by 141
Zhou, X., Kim, K., Xu, B., Han, D., & Lo, D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639222 · Cited by 140
Peng, Y., Dalmia, S., Lane, I. R., & Watanabe, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.02971 · Cited by 139
An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
Cao, Y. P., Iqbal, T., Kong, Q., An, F., Wang, W., & Plumbley, M. D. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413473 · Cited by 134
Qu, H., Ning, L., An, R., Fan, W., Derr, T., Liu, H., Xin, X., & Li, Q. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3819582 · Cited by 131
LambdaNetworks: Modeling Long-Range Interactions Without Attention
Bello, I. (2021) · arXiv · doi:10.48550/arxiv.2102.08602 · Cited by 129
“…Parmar et al., 2018; Ramachandran et al., 2019) • axial attention patterns (Ho et al., 2019; Wang et al., 2020a; Shen et al., 2020) • static sparse attention patterns (Child et al.; Beltagy et al., 2020) • dynamic sparse attention patterns (Kitaev et al., 2020) See Tay et al. (2020) for a review.”
BackgroundThe Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 129
Self-citationDynamic Sparse Attention for Scalable Transformer Acceleration
Liu, L., Qu, Z., Chen, Z., Tu, F., Ding, Y., & Xie, Y. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3208206 · Cited by 127
DTCM: Deep Transformer Capsule Mutual Distillation for Multivariate Time Series Classification
Xiao, Z., Xu, X., Xing, H., Zhao, B., Wang, X., Song, F., Qu, R., & Li, F. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3370219 · Cited by 126
Transformer in reinforcement learning for decision-making: a survey
Yuan, W., Chen, J., Chen, S., Lu, L., Hu, Z., Li, P., Feng, D., Liu, F., & Chen, J. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300548 · Cited by 127
Transformers in speech processing: Overcoming challenges and paving the future
Latif, S., Zaidi, S. A. M., Cuayáhuitl, H., Shamshad, F., Shoukat, M., Usama, M., & Qadir, J. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100768 · Cited by 126
Fang, L., Yu, X., Cai, J., Chen, Y., Wu, S., Liu, Z., Yang, Z., Lu, H., Gong, X., Liu, Y., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11423-3 · Cited by 124
Revisiting Transformer-based Models for Long Document Classification
Dai, X., Chalkidis, I., Darkner, S., & Elliott, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.534 · Cited by 123
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 122
Transformer-Based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey
Dubey, S. R., & Singh, S. K. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3404910 · Cited by 122
Sparse is Enough in Scaling TransformersPreprint
Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, Ł., Gajewski, W., Michalewski, H., & Kanerva, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12763 · Cited by 122
CCTEST: Testing and Repairing Code Completion Systems
Li, Z., Wang, C., Liu, Z., Wang, H., Chen, D., Wang, S., & Gao, C. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00110 · Cited by 52
Ding, Y., & Jia, M. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3181933 · Cited by 120
Tabatabaei, S., Rezaee, K., & Zhu, M. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105119 · Cited by 119
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118
A Review on the Use of Large Language Models as Virtual Tutors
García-Méndez, S., de Arriba-Pérez, F., & Somoza-López, M. D. C. (2024) · Science & Education · doi:10.1007/s11191-024-00530-2 · Cited by 118
On the Challenges and Opportunities in Generative AI
Manduchi, L., Meister, C., Pandey, K., Bamler, R., Cotterell, R., Däubener, S., Fellenz, S., Fischer, A., Gärtner, T., Kirchler, M., et al. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2403.00025 · Cited by 117
Wu, C., Wu, F., Qi, T., & Huang, Y. (2021) · doi:10.18653/v1/2021.acl-short.107 · Cited by 116
Corpus for Automatic Structuring of Legal Documents
Kalamkar, P., Tiwari, A. K., Agarwal, A., Karn, S., Gupta, S., Raghavan, V., & Modi, A. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/543m4q3q6ts6 · Cited by 116
Li, W., Fu, H., Han, Z., Zhang, X., & Jin, H. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2022.102368 · Cited by 116
Finetuning Pretrained Transformers into RNNs
Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.830 · Cited by 115
Ren, Q., Yang, L., & Liu, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120203 · Cited by 115
SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114
Vision Transformer Hashing for Image Retrieval
Dubey, S. R., Singh, S. K., & Chu, W.-T. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859900 · Cited by 114
(2023) · doi:10.52202/075280-1506 · Cited by 113
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
Self-citationChasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111
High-Resolution Optical Flow from 1D Attention and Correlation
Xu, H., Yang, J., Cai, J., Zhang, J., & Tong, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01033 · Cited by 111
A Primer on Pretrained Multilingual Language Models
Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A., & Kumar, P. (2025) · ACM Computing Surveys · doi:10.1145/3727339 · Cited by 108
Shrivastava, A. D., Swainston, N., Samanta, S., Roberts, I., Wright Muelas, M., & Kell, D. B. (2021) · Biomolecules · doi:10.3390/biom11121793 · Cited by 87
Scene Adaptive Sparse Transformer for Event-based Object Detection
Peng, Y., Li, H., Zhang, Y., Sun, X., & Wu, F. H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01589 · Cited by 106
(2024) · doi:10.52202/079017-2193 · Cited by 103
Research on bearing fault diagnosis method based on transformer neural network
Yang, Z., Cen, J., Liu, X., Xiong, J., & Chen, H. (2022) · Measurement Science and Technology · doi:10.1088/1361-6501/ac66c4 · Cited by 103
Foundation Models for Natural Language Processing: Pre-trained Language Models Integrating Media
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2 · Cited by 103
A Survey on Long Text Modeling with TransformersPreprint
Dong, Z., Tang, T., Li, J., & Zhao, W. X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14502 · Cited by 103
A Survey on Transformers in Reinforcement Learning
Li, W., Luo, H., Lin, Z.-C., Zhang, C.-J., Lu, Z.-Q., & Ye, D.-H. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2301.03044 · Cited by 95
Zualkernan, I. A., Abuhani, D. A., Hussain, M. H., Khan, J., & ElMohandes, M. (2023) · Drones · doi:10.3390/drones7060382 · Cited by 100
Representation learning for neural population activity with Neural Data Transformers
Ye, J., & Pandarinath, C. (2021) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.27358 · Cited by 98
Efficient Classification of Long Documents Using Transformers
Park, H., Vyas, Y., & Shah, K. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.79 · Cited by 98
Lorenzo, G., Ahmed, S. R., Hormuth, D. A., Vaughn, B., Kalpathy–Cramer, J., Solorio, L., Yankeelov, T. E., & Gómez, H. (2024) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-081623-025834 · Cited by 98
SparQ Attention: Bandwidth-Efficient LLM InferencePreprint
Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., & Orr, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04985 · Cited by 98
Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting
Lin, C., Ouyang, Z., Zhuang, J., Chen, J., Li, H., & Wu, R. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00026 · Cited by 97
A Survey of Vibe Coding with Large Language ModelsPreprint
Ge, Y., Mei, L., Duan, Z., Li, T., Zheng, Y., Wang, Y., Wang, L., Yao, J., Liu, T., Cai, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.12399 · Cited by 98
Transformers for Tabular Data Representation: A Survey of Models and Applications
Badaro, G., Saeed, M., & Papotti, P. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00544 · Cited by 96
“While there exist methods to make transformers more efficient for long context (Tay et al., 2020), all optimizations consider an unstructured textual input.”
BackgroundMethodInfluentialAn Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
Fang, C., Zhou, A., & Wang, Z. (2022) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2022.3197282 · Cited by 94
“TRANSFORMER-BASED networks are a formidable force in deep learning [1].”
BackgroundHow Far Are We From AGI: Are LLMs All We Need?
Feng, T., Jin, C., Liu, J., Zhu, K., Tu, H., Cheng, Z., Lin, G., & You, J. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.10313 · Cited by 94
Single-Shot Motion Completion with TransformerPreprint
Duan, Y., Shi, T., Zou, Z., Lin, Y., Qian, Z., Zhang, B., & Yuan, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00776 · Cited by 94
Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks
Dong, H., Cheng, Z., He, X., Zhou, M., Zhou, A., Zhou, F., Liu, A., Han, S., & Zhang, D. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/761 · Cited by 93
Vision transformers for dense prediction: A survey
Zuo, S., Xiao, Y., Chang, X., & Wang, X. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109552 · Cited by 93
Álvarez‐Carmona, M. Á., Aranda, R., Rodríguez‐González, A. Y., Fajardo‐Delgado, D., Sánchez, M. G., Pérez-Espinosa, H., Martínez-Miranda, J., Rodríguez, R. G., Bustio-Martínez, L., & Pacheco, A. D. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.10.010 · Cited by 93
FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification
Cheng, M., Liu, Q., Liu, Z., Li, Z., Luo, Y., & Chen, E. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583205 · Cited by 93
KDEformer: Accelerating Transformers via Kernel Density EstimationPreprint
Zandieh, A., Han, I., Daliri, M., & Karbasi, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.02451 · Cited by 93
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2106.12672 · Cited by 92
“Many ef-ficient self-attention models have been proposed (Choromanski et al., 2020; Wang et al., 2020; Zaheer et al., 2020) to tackle this problem; see (Tay et al., 2020b;a) for a comprehensive overview.”
BackgroundSelf-citationGuo, J., Jia, N., Zhang, M., Zhang, R., & Li, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-19831-z · Cited by 92
X-Former: In-Memory Acceleration of Transformers
Sridharan, S., Stevens, J. R., Roy, K., & Raghunathan, A. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3282046 · Cited by 68
How Much Temporal Long-Term Context is Needed for Action Segmentation?
Bahrami, E., Francesca, G., & Gall, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00950 · Cited by 92
Confident adaptive language modeling
Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., & Metzler, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1269 · Cited by 91
Self-citationNatural Language Processing in the Legal Domain
Katz, D. M., Hartung, D., Gerlach, L., Jana, A., & Bommarito, M. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4336224 · Cited by 91
Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention
Zhou, Z., Liu, J., Gu, Z., & Sun, G. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3170848 · Cited by 91
User-LLM: Efficient LLM Contextualization with User EmbeddingsPreprint
Ning, L., Liu, L., Wu, J., Wu, N., Berlowitz, D., Prakash, S., Green, B., O’Banion, S., & Xie, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13598 · Cited by 91
Wu, Y., Rabe, M. N., Hutchins, D., & Szegedy, C. (2022) · arXiv · doi:10.48550/arxiv.2203.08913 · Cited by 90
“A great deal of work has been done on efficient long-range attention mechanisms; see Tay et al. (2020; 2021) recent surveys.”
BackgroundMalBERT: Malware Detection using Bidirectional Encoder Representations from Transformers
Rahali, A., & Akhloufi, M. A. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9659287 · Cited by 90
The knowledge and innovation challenges of ChatGPT: A scoping review
Ali, O., Murray, P., Momin, M., & Al–Anzi, F. S. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102402 · Cited by 90
Pre-trained Language Model based Ranking in Baidu Search
Zou, L., Zhang, S., Cai, H., Ma, D., Cheng, S., Wang, S., Shi, D., Cheng, Z., & Yin, D. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467147 · Cited by 89
A Historical Survey of Advances in Transformer Architectures
Sajun, A. R., Zualkernan, I. A., & Sankalpa, D. (2024) · Applied Sciences · doi:10.3390/app14104316 · Cited by 89
Training-Free and Adaptive Sparse Attention for Efficient Long Video Generation
Xia, Y., Ling, S., Fu, F., Wang, Y., Li, H., Xiao, X., & Cui, B. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01483 · Cited by 89
Ding, W., Abdel‐Basset, M., Alrashdi, I., & Hawash, H. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120338 · Cited by 88
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.