Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 201–300 of 1,010
Liu, Z., Kang, X., & Ren, F. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3282092 · Cited by 87
Neural natural language processing for long texts: A survey on classification and summarization
Tsirmpas, D., Gkionis, I., Papadopoulos, G. T., & Mademlis, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108231 · Cited by 87
Efficient High-Resolution Deep Learning: A Survey
Bakhtiarnia, A., Zhang, Q., & Iosifidis, A. (2024) · ACM Computing Surveys · doi:10.1145/3645107 · Cited by 86
Yu, H., Zhang, H., Zhang, Z., & Yang, S. (2025) · eTransportation · doi:10.1016/j.etran.2025.100420 · Cited by 86
HeBERT and HebEMO: A Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
Chriqui, A., & Yahav, I. (2022) · INFORMS Journal on Data Science · doi:10.1287/ijds.2022.0016 · Cited by 85
DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor Cores
Fan, R., Wang, W., & Chu, X. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620666.3651378 · Cited by 85
Frisoni, G., Moro, G., & Carbonaro, A. (2021) · IEEE Access · doi:10.1109/access.2021.3130956 · Cited by 84
IF-ConvTransformer: A Framework for Human Activity Recognition Using IMU Fusion and ConvTransformer
Zhang, Y., Wang, L., Chen, H., Tian, A., Zhou, S., & Guo, Y. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3534584 · Cited by 46
Cui, L., Jing, X., Wang, Y., Huan, Y., Xu, Y., & Zhang, Q. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3225150 · Cited by 84
Investigating Efficiently Extending Transformers for Long Input Summarization
Phang, J., Zhao, Y., & Liu, P. X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.240 · Cited by 83
Yu, B. X. B., Chang, J., Wang, H., Liu, L., Wang, S., Wang, Z., Lin, J., Xie, L., Li, H., Lin, Z., et al. (2024) · ACM Computing Surveys · doi:10.1145/3657632 · Cited by 83
Resource-Efficient Neural Networks for Embedded SystemsPreprint
Roth, W., Schindler, G., Klein, B., Peharz, R., Tschiatschek, S., Fröning, H., Pernkopf, F., & Ghahramani, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03048 · Cited by 83
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Bondarenko, Y., Nagel, M., & Blankevoort, T. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.627 · Cited by 81
Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization
Laskar, M. T. R., Hoque, E., & Huang, J. X. (2022) · Computational Linguistics · doi:10.1162/coli_a_00434 · Cited by 81
Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry
Cong, S., Wang, H., Zhou, Y., Wang, Z., Yao, X., & Yang, C. (2024) · Brain‐X · doi:10.1002/brx2.57 · Cited by 81
Transformers in biosignal analysis: A review
Anwar, A., Khalifa, Y., Coyle, J. L., & Sejdić, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102697 · Cited by 81
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
Fu, D. Y., Arora, S., Grogan, J., Johnson, I., Eyuboglu, E. S., Thomas, A. W., Spector, B., Poli, M., Rudra, A., & Ré, C. (2023) · neural information processing systems · doi:10.52202/075280-3390 · Cited by 80
Sun, W., Qin, Z., Deng, H., Wang, J., Zhang, Y., Zhang, K., Barnes, N. M., Birchfield, S. T., Kong, L., & Zhong, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3285569 · Cited by 79
MalBERT: Using Transformers for Cybersecurity and Malicious Software DetectionPreprint
Rahali, A., & Akhloufi, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03806 · Cited by 79
Systems engineering issues for industry applications of large language model
Chen, W., Yan-yi, L., Tie-zheng, G., Li, D., Tao, H., Li, Z., Yang, Q., Hui-han, W., & Ying-you, W. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.111165 · Cited by 78
Machine Learning for Human Activity Recognition: State-of-the-Art Techniques and Emerging Trends
Hossen, M. A., & Abas, P. E. (2025) · Journal of Imaging · doi:10.3390/jimaging11030091 · Cited by 79
Machine Translation for Long Sequences with Enhanced Attention Mechanisms
Agrawal, A. M., Gajula, S., Shinde, R. P., Shah, H., & Ghosh, H. (2025) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet63943.2025.11471912 · Cited by 78
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse TransformersPreprint
Yun, C., Chang, Y.-W., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., & Kumar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04862 · Cited by 78
Empirical study of transformers for source code
Chirkova, N., & Troshin, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468611 · Cited by 77
Transformer Meets Remote Sensing Video Detection and Tracking: A Comprehensive Survey
Jiao, L., Zhang, X., Liu, X., Liu, F., Yang, S., Ma, W., Li, L., Chen, P., Feng, Z., Guo, Y., et al. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3289293 · Cited by 77
Privformer: Privacy-preserving Transformer with MPC
Akimoto, Y., Fukuchi, K., Akimoto, Y., & Sakuma, J. (2023) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp57164.2023.00031 · Cited by 77
Deo, B. S., Pal, M., Panigrahi, P. K., & Pradhan, A. (2024) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.23043 · Cited by 77
Dao, T., & Gu, A. (2024) · arXiv · doi:10.48550/arxiv.2405.21060 · Cited by 76
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 76
TransformGraph: A novel short-term electricity net load forecasting model
Zhang, Q., Chen, J., Xiao, G., He, S., & Deng, K. (2023) · Energy Reports · doi:10.1016/j.egyr.2023.01.050 · Cited by 76
Galke, L. P. A., & Scherp, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.279 · Cited by 75
STRec: Sparse Transformer for Sequential Recommendations
Li, C., Wang, Y., Liu, Q., Zhao, X., Wang, W., Wang, Y., Zou, L., Fan, W., & Li, Q. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608779 · Cited by 75
On Sparse Modern Hopfield Model
Hu, J. Y.-C., Yang, D., Wu, D., Xu, C., Chen, B.-Y., & Liu, H. (2023) · neural information processing systems · doi:10.52202/075280-1200 · Cited by 75
Compute-Efficient Deep Learning: Algorithmic Trends and OpportunitiesPreprint
Bartoldson, B. R., Kailkhura, B., & Blalock, D. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06640 · Cited by 75
Rethinking attention mechanism in time series classification
Zhao, B., Xing, H., Wang, X., Song, F., & Xiao, Z. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.093 · Cited by 74
AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference With Transformers
Tuli, S., & Jha, N. K. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3273992 · Cited by 74
Exploring transformers for behavioural biometrics: A case study in gait recognition
Delgado-Santos, P., Tolosana, R., Guest, R., Deravi, F., & Vera-Rodríguez, R. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109798 · Cited by 74
Efficient tensor core-based GPU kernels for structured sparsity under reduced precision
Chen, Z., Qu, Z., Liu, L., Ding, Y., & Xie, Y. (2021) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3458817.3476182 · Cited by 73
Ke, W., Zheng, Y., Li, Y., Xu, H., Nie, D., Wang, P., & He, Y. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768156 · Cited by 73
Point Cloud Completion: A Survey
Tesema, K. W., Hill, L., Jones, M. W., Ahmad, M., & Tam, G. K. L. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3344935 · Cited by 72
Positional encoding in transformer-based time series models: a survey
Irani, H., & Metsis, V. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-026-03151-z · Cited by 72
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D ParallelismPreprint
Chen, Y., Pan, X., Li, Y., Ding, B., & Zhou, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04916 · Cited by 72
CIForm as a Transformer-based model for cell-type annotation of large-scale single-cell RNA-seq data
Xu, J., Zhang, A., Liu, F., Chen, L., & Zhang, X. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad195 · Cited by 71
TBFormer: Two-Branch Transformer for Image Forgery Localization
Liu, Y., Lv, B., Jin, X., Chen, X., & Zhang, X. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3279018 · Cited by 71
Sparsity in transformers: A systematic literature review
Farina, M., Ahmad, U., Taha, A. T. M., Younes, H., Mesbah, Y., Yu, X., & Pedrycz, W. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127468 · Cited by 71
Crafting clarity: Leveraging large language models to decode consumer reviews
Praveen, S. V., Gajjar, P., Ray, R. K., & Dutt, A. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.103975 · Cited by 71
Survey: Transformer based video-language pre-training
Ruan, L., & Jin, Q. (2022) · AI Open · doi:10.1016/j.aiopen.2022.01.001 · Cited by 49
Cramming: Training a Language Model on a Single GPU in One DayPreprint
Geiping, J., & Goldstein, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.14034 · Cited by 71
The Efficiency Spectrum of Large Language Models: An Algorithmic SurveyPreprint
Ding, T., Chen, T., Zhu, H., Jiang, J., Zhong, Y., Zhou, J., Wang, G., Zhu, Z., Zharkov, I., & Liang, L. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.32142.54087/1 · Cited by 72
Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
Zhang, L., Negrinho, R., Ghosh, A., Jagannathan, V., Hassanzadeh, H. R., Schaaf, T., & Gormley, M. R. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.313 · Cited by 37
LGViT: Dynamic Early Exiting for Accelerating Vision Transformer
Xu, G., Hao, J., Shen, L., Hu, H., Luo, Y., Lin, H., & Shen, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611762 · Cited by 70
Foundation Model-Based Multimodal Remote Sensing Data Classification
He, X. J., Chen, Y., Huang, L., Hong, D., & Du, Q. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3344698 · Cited by 70
Online portfolio management via deep reinforcement learning with high-frequency data
Li, J., Zhang, Y., Yang, X., & Chen, L. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2022.103247 · Cited by 69
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Ma, X., Yang, X., Xiong, W., Chen, B., Yu, L., Zhang, H., May, J., Zettlemoyer, L., Levy, O., & Zhou, C. (2024) · neural information processing systems · doi:10.52202/079017-2294 · Cited by 69
Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes
Moro, G., Ragazzi, L., Valgimigli, L., Frisoni, G., Sartori, C., & Marfia, G. (2023) · Sensors · doi:10.3390/s23073542 · Cited by 68
MATE: Multi-view Attention for Table Transformer Efficiency
Eisenschlos, J., Gor, M., Müller, T., & Cohen, W. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.600 · Cited by 67
Robust RGB-D Fusion for Saliency Detection
Wu, Z., Gobichettipalayam, S., Tamadazte, B., Allibert, G., Paudel, D. P., & Demonceaux, C. (2022) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv57658.2022.00052 · Cited by 67
EPT-Net: Edge Perception Transformer for 3D Medical Image Segmentation
Yang, J., Jiao, L., Shang, R., Liu, X., Li, R., & Xu, L. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278461 · Cited by 67
Wang, J., Yan, J., Gu, A., & Rush, A. M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.5 · Cited by 67
TorchEEGEMO: A deep learning toolbox towards EEG-based emotion recognition
Zhang, Z., Zhong, S.-H., & Liu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123550 · Cited by 67
Adaptable butterfly accelerator for attention-based NNs via hardware and algorithm co-design
Fan, H., Chau, T., Venieris, S. I., Lee, R., Kouris, A., Luk, W., Lane, N. D., & Abdelfattah, M. S. (2022) · 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00050 · Cited by 66
“As the algorithmic complexity of the self-attention mechanism scales quadratically with respect to the input sequence length, many sparse variants have been introduced to approximate the attention-based NNs [45].”
BackgroundWhen Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
Lei, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.602 · Cited by 66
Classification of long sequential data using circular dilated convolutional neural networks
Cheng, L., Khalitov, R., Yu, T., Zhang, J., & Yang, Z. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.10.054 · Cited by 66
Deep learning for video-text retrieval: a review
Zhu, C., Jia, Q. F., Chen, W., Guo, Y., & Liu, Y. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00267-8 · Cited by 66
Deep learning for 3D human pose estimation and mesh recovery: A survey
Liu, Y., Qiu, C., & Zhang, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128049 · Cited by 67
Siu, S. C. (2024) · New frontiers in translation studies · doi:10.1007/978-981-97-2958-6_3 · Cited by 66
Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion ModelsPreprint
Fei, Z., Fan, M., Yu, C., Li, D., & Huang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.04478 · Cited by 66
Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
Xiao, S., Liu, Z., Shao, Y., Di, T., Middha, B., Wu, F., & Xie, X. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539120 · Cited by 39
Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
Wang, D., Yan, Y., Qiu, R., Zhu, Y., Guan, K., Margenot, A. J., & Tong, H. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599444 · Cited by 65
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
Zhao, L., Feng, X., Feng, X., Zhong, W., Xu, D., Yang, Q., Liu, H., Qin, B., & Liu, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.582 · Cited by 65
3D Vision with Transformers: A SurveyPreprint
Lahoud, J., Cao, J., Khan, F. S., Cholakkal, H., Anwer, R. M., Khan, S., & Yang, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.04309 · Cited by 66
Deep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 64
Lightweight transformers for clinical natural language processing
Rohanian, O., Nouriborji, M., Jauncey, H., Kouchaki, S., Nooralahzadeh, F., Clifton, L., Merson, L., & Clifton, D. A. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000542 · Cited by 64
Peng, Y., Sudo, Y., Shakeel, M., & Watanabe, S. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.549 · Cited by 64
Phyloformer: Fast, Accurate, and Versatile Phylogenetic Reconstruction with Deep Neural Networks
Nesterenko, L., Blassel, L., Veber, P., Boussau, B., & Jacob, L. (2025) · Molecular Biology and Evolution · doi:10.1093/molbev/msaf051 · Cited by 65
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2109.10686 · Cited by 63
“…(Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efficient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures…”
BackgroundSelf-citationA large-scale dataset of patient summaries for retrieval-based clinical decision support systems
Zhao, Z., Jin, Q., Chen, F., Peng, T., & Yu, S. (2023) · Scientific Data · doi:10.1038/s41597-023-02814-8 · Cited by 63
“Therefore, retrieval performance may be further enhanced by using efficient transformers [59] such as Big bird [60] and Longformer [61].”
DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization
Mao, Z., Wu, C., Ni, A., Zhang, Y., Zhang, R., Yu, T., Deb, B., Zhu, C., Awadallah, A. H., & Radev, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.118 · Cited by 63
Video Joint Modelling Based on Hierarchical Transformer for Co-summarization
Li, H., Ke, Q., Gong, M., & Zhang, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3186506 · Cited by 63
Yang, X., Zhang, K., Ni, C., Cao, H., Thé, J. V. G., Xie, G., Tan, Z., & Yu, H. (2022) · Energy · doi:10.1016/j.energy.2022.125027 · Cited by 63
Anson, D. W. J. (2024) · Higher Education Research & Development · doi:10.1080/07294360.2024.2332259 · Cited by 63
Gao, Y., Zhang, J., Wei, S., & Li, Z. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107154 · Cited by 62
Chen, X., Li, C., Bernards, M. T., Shi, Y., Shao, Q., & He, Y. (2021) · Molecular Systems Design & Engineering · doi:10.1039/d0me00161a · Cited by 61
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
Ruder, S., Vulić, I., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.184 · Cited by 61
MoA: Mixture of Sparse Attention for Automatic Large Language Model CompressionPreprint
Fu, T., Huang, H.-F., Ning, X.-F., Zhang, G., Chen, B.-J., Wu, T.-Q., Wang, H.-Y., Huang, Z.-X., Li, S.-Y., Yan, S.-G., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2406.14909 · Cited by 32
Post-Training Sparse Attention with Double SparsityPreprint
Yang, S., Sheng, Y., Gonzalez, J. E., Stoica, I. G., & Zheng, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.07092 · Cited by 62
Training language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60
Koizumi, Y., Karita, S., Wisdom, S., Erdoğan, H., Hershey, J. R., Jones, L., & Bacchiani, M. (2021) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa52581.2021.9632794 · Cited by 60
Zhao, Y., Ma, Z., Zhou, T., Ye, M., Sun, L., & Yi, Q. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615136 · Cited by 60
SpliceTransformer predicts tissue-specific splicing linked to human diseases
You, N., Liu, C., Gu, Y., Wang, R., Jia, H., Zhang, T., Jiang, S., Shi, J., Chen, M., Guan, M., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-53088-6 · Cited by 60
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Kim, G., & Cho, K. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.508 · Cited by 59
“Tay et al. (2020b) and Tay et al. (2020a) provide a survey of these efficient transformers and their benchmark comparison, respectively.”
BackgroundLiu, S., Mu, C., Jiang, H., Wang, Y., Zhang, J., Lin, F., Zhou, K., Liu, Q., & Chen, C. (2023) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2023.3337777 · Cited by 60
Transformer networks of human conceptual knowledge.
Bhatia, S., & Richie, R. (2024) · Psychological Review · doi:10.1037/rev0000319 · Cited by 59
End-to-End Trainable Multi-Instance Pose Estimation with TransformersPreprint
Stoffl, L., Vidal, M., & Mathis, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.12115 · Cited by 59
MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores
Kao, S.-C., & Krishna, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00065 · Cited by 58
“At long sequence range, attention layers becomes a heavy job owing to its quadratic complexity for both compute and memory [95], [96].”
BackgroundRevisiting Transformer-based Models for Long Document Classification
Dai, X., Chalkidis, I., Darkner, S., & Elliott, D. (2022) · arXiv · doi:10.48550/arxiv.2204.06683 · Cited by 58
Deep Generative Models for Therapeutic Peptide Discovery: A Comprehensive Review
Lai, L., Liu, Y., Song, B., Li, K., & Zeng, X. (2025) · ACM Computing Surveys · doi:10.1145/3714455 · Cited by 58
STAR: Sparse Transformer-based Action RecognitionPreprint
Shi, F., Lee, C., Qiu, L., Zhao, Y., Shen, T., Muralidhar, S., Han, T., Zhu, S. C., & Narayanan, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07089 · Cited by 58
TxPert: using multiple knowledge graphs for prediction of transcriptomic perturbation effects
Wenkel, F., Tu, W., Masschelein, C., Shirzad, H., Hodgson, L. M., Bendidi, I., Eastwood, C., Whitfield, S. T., Russell, C., Mesbahi, Y. E., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03113-4 · Cited by 57
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.