Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 801–900 of 1,010
Wang, H., Guo, M., & Tian, L. (2023) · Sensors · doi:10.3390/s23135819 · Cited by 11
Yao, Z., Jiang, Q., & Gu, X. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3562969 · Cited by 11
Efficient Transformer Inference Through Hybrid Dynamic Pruning
Jaradat, G. A., Tolba, M. F., Alsuhli, G. H., Saleh, H. H., Al‐Qutayri, M., & Stouraitis, T. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3563144 · Cited by 11
Li, Y., & Liu, C. (2025) · IISE Transactions · doi:10.1080/24725854.2025.2488340 · Cited by 11
Cottention: Linear Transformers with Cosine Attention
Mongaras, G., Dohm, T., & Larson, E. C. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-92602-0_32 · Cited by 11
The Evolution of Deep Learning: Models, Applications, and Future Directions
Elshenawy, A., Mohammed, A., & Hamouda, S. (2025) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc66482.2025.11196834 · Cited by 11
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm
Hou, Z., Gao, W., Shen, Y., Wang, F., & Liu, X. (2024) · neural information processing systems · doi:10.52202/079017-4370 · Cited by 11
FullTransNet: Full Transformer With Local-Global Attention for Video Summarization
Lan, L., Jiang, L., Yu, T., Liu, X., & He, Z. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3634718 · Cited by 11
Uncertainty-Aware Bayesian Time Series framework for probabilistic imputation
Zeng, Z.-X., Tan, Y., Deng, E., Ni, Y., Zhang, Q., & Yang, B. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.104153 · Cited by 11
Lee, I., Jiang, N., & Berg-Kirkpatrick, T. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.08049 · Cited by 11
Fast multipole attention: A scalable multilevel attention mechanism for text and images
Kang, Y., Tran, G., & Sterck, H. D. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109847 · Cited by 11
Estimating Crop Grain Yield and Seed Composition Using Deep Learning from UAV Multispectral Data
Maimaitijiang, M., Millett, B. V., Paheding, S., Khan, S. N., Dilmurat, K., Reyes-Angulo, A. A., & Kovács, P. (2023) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss52108.2023.10281802 · Cited by 11
Investigating the Effects of Sparse Attention on Cross-Encoders
Schlatt, F., Fröbe, M., & Hagen, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_11 · Cited by 11
What Makes Models Compositional? A Theoretical View
Ram, P., Klinger, T., & Gray, A. (2024) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/533 · Cited by 11
TBiSeg: A transformer-based network with bi-level routing attention for inland waterway segmentation
Fu, C., Li, M., Zhang, B., & Wang, H. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.119011 · Cited by 11
Transformers are Expressive, But Are They Expressive Enough for Regression?Preprint
Nath, S., Khadilkar, H., & Bhattacharyya, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.15478 · Cited by 11
freePruner: A Training-free Approach for Large Multimodal Model AccelerationPreprint
Xu, B., Shang, Y., Ge, Y., Lou, Q., & Yan, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.15446 · Cited by 11
A Survey on Prompt TuningPreprint
Li, Z., Su, Y., & Collier, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.06085 · Cited by 11
Music Generation through Transformers
Agarwal, S., & Sultanova, N. (2024) · International journal of data science and advanced analytics. · doi:10.69511/ijdsaa.v6i6.231 · Cited by 10
Heterogeneous Binary Pixel Difference Networks for Remote Sensing Object Detection
Zhan, J., Bai, L., Zhang, J., Liu, T., Shi, F., Liu, Y., & Liu, L. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3520161 · Cited by 10
Gui, L., Gu, X., Huang, F., Ren, S., Qin, H., & Fan, C. (2025) · Applied Sciences · doi:10.3390/app15031427 · Cited by 10
Chen, J., & Er, M. J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111067 · Cited by 10
Zhang, J., & Zhang, H. (2025) · Mathematics · doi:10.3390/math13132087 · Cited by 10
A Comprehensive Survey of Advancement in Lip Reading Models: Techniques and Future Directions
Deshpande, S., Shirsath, K., Pashte, A., Loya, P., Shingade, S., & Sambhe, V. (2025) · IET Image Processing · doi:10.1049/ipr2.70095 · Cited by 10
Chan, Y. C., Poon, H. M., & Jauw, V. L. (2025) · Energy Conversion and Management · doi:10.1016/j.enconman.2025.120638 · Cited by 10
Liu, H., & Yang, L. (2025) · Symmetry · doi:10.3390/sym17111846 · Cited by 10
Self-Retrieval from Distant Contexts for Document-Level Machine Translation
Peng, Z., Bawden, R., & Yvon, F. (2025) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2025.wmt-1.13 · Cited by 10
Optimising Time Series Forecasting Transformers to Linear Complexity
Cheng, Y., Zhang, S., & Li, H. (2025) · Energy internet. · doi:10.1049/ein2.70016 · Cited by 10
Hussain, M. A., Waris, A., Akram, M. U., Khan, M. J., Asaf, M. Z., Javaid, A., Gilani, S. O., & Hazzazi, F. (2026) · PLoS ONE · doi:10.1371/journal.pone.0341311 · Cited by 10
Fichtl, A. M., Bohn, J., Kelber, J., Mosca, E., & Groh, G. (2026) · Proceedings of The Big Picture v2: Crafting a Research Narrative · doi:10.18653/v1/2026.bigpicture-main.6 · Cited by 10
Fast Monte-Carlo Approximation of the Attention Mechanism
Kim, H., & Ko, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20679 · Cited by 10
Bruno, W., & Roth, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.369 · Cited by 10
Structure-Informed Positional Encoding for Music Generation
Agarwal, M., Wang, C., & Richard, G. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448149 · Cited by 10
Knowledge Graphs Enhanced Large Language Model Prompt for Electric Power Question Answering
Wang, C., Hua, M., Song, J., & Tang, X. (2023) · International Conference on Electronic Information Technology and Computer Engineering (EITCE) · doi:10.1145/3650400.3650405 · Cited by 10
CriticalFuzz: A critical neuron coverage-guided fuzz testing framework for deep neural networks
Bai, T., Huang, S., Huang, Y., Wang, X., Xia, C., Qu, Y., & Yang, Z. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107476 · Cited by 10
Software Development and Education: Transitioning Towards AI Enhanced Teaching
Israilidis, J., Chen, W., & Tsakalerou, M. (2024) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon60312.2024.10578564 · Cited by 10
A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
Ge, S., Lin, X., Zhang, Y., Han, J., & Peng, H. M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.01485 · Cited by 10
Swin-MGNet: Swin Transformer Based Multiview Grouping Network for 3-D Object Recognition
Ning, X., Jiang, L., Li, W., Yu, Z., Xie, J., Li, L., Tiwari, P., & Alonso‐Fernandez, F. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3492163 · Cited by 10
Zeng, Z., Guo, Q., Liu, X., Yin, Z., Shu, W., Huang, M., Wang, B., Zhou, Y., Li, L., Liu, Q., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1169 · Cited by 10
Hoang, V. T., Jeon, H.-J., & Lee, O. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3563697 · Cited by 10
SwinTSER: An Improved Bilingual Speech Emotion Recognition Using Shift Window Transformer
Akinpelu, S. A., Viriri, S., & Yousaf, M. H. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10484-4 · Cited by 10
Value-aware Approximate Attention
Gupta, A., & Berant, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.753 · Cited by 10
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Effective Pretraining Objectives for Transformer-based Autoencoders
Liello, L. D., Gabburo, M., & Moschitti, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.405 · Cited by 10
TRAMS: Training-free Memory Selection for Long-range Language Modeling
Yu, H., Wang, C., Zhang, Y., & Bi, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.331 · Cited by 10
TLS-RWKV: Real-Time Online Action Detection with Temporal Label Smoothing
Zhu, Z., Shao, W., & Jiao, D. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11540-0 · Cited by 10
Towards efficient similarity embedded temporal Transformers via extended timeframe analysis
Olorunnimbe, K., & Viktor, H. L. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01400-8 · Cited by 10
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling
Karami, M., & Ghodsi, A. (2024) · neural information processing systems · doi:10.52202/079017-2449 · Cited by 10
Gu, Y., Qin, X., Peng, Y., & Li, L. (2022) · IET Image Processing · doi:10.1049/ipr2.12575 · Cited by 10
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems
Manakul, P., & Gales, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.739 · Cited by 10
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences
Chen, Y., Zeng, Q., Hakkani-Tür, D. Z., Jin, D., Ji, H., & Yang, Y. L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.381 · Cited by 10
PoNet: Pooling Network for Efficient Token Mixing in Long SequencesPreprint
Tan, C.-H., Chen, Q., Wang, W., Zhang, Q., Zheng, S., & Ling, Z.-H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02442 · Cited by 10
Iki, T., & Aizawa, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.07828 · Cited by 10
Structured Time Series Prediction without Structural PriorPreprint
Drakulić, D., & Andreoli, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03539 · Cited by 10
Gated Recurrent Neural Networks with Weighted Time-Delay FeedbackPreprint
Erichson, N. B., Lim, S. H., & Mahoney, M. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.00228 · Cited by 10
Attention Is Not All You Need AnymorePreprint
Chen, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.07661 · Cited by 10
Scaling Laws Behind Code Understanding ModelPreprint
Lin, J., Dong, H., Xie, Y., & Zhang, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.12813 · Cited by 10
Zhang, Z., Wang, Y., Huang, X., Fang, T., Zhang, H., Deng, C., Li, S., & Yu, D. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.485 · Cited by 9
Muhetaer, M., Meng, X., Zhu, J., Aikebaier, A., Zu, L., & Bai, Y. (2025) · Symmetry · doi:10.3390/sym17111812 · Cited by 9
TextMamba: Scene Text Detector with Mamba
Zhao, Q., Yan, Y. P., & Wang, D.-H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227846 · Cited by 9
When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
Xu, Z., Zhu, S., Wang, J., Wang, J., Athiwaratkun, B., Wang, C., Zou, J., & Zhang, C. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.16411 · Cited by 9
DeRainMamba: A Frequency-Aware State Space Model With Detail Enhancement for Image Deraining
Zhu, Z., Zeng, T., Yang, T., Luo, G., & Zeng, J. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3616020 · Cited by 9
A Psychology-based Unified Dynamic Framework for Curriculum Learning
Meng, G., Zeng, Q., Lalor, J. P., & Yu, H. (2025) · Computational Linguistics · doi:10.1162/coli.a.584 · Cited by 9
Huang, Z., Chen, M., & Zheng, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132562 · Cited by 9
Layer-dependent dynamic spectral weighting for efficient transformer models
Huang, Z., Yu, Y., Zheng, S., Chen, M., & Li, Q. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114285 · Cited by 9
SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
Bashlovkina, V., Matthews, R., Kuang, Z., Baumgartner, S., & Bendersky, M. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599907 · Cited by 9
Fixed Input Parameterization for Efficient Prompting
Choi, E., Jo, Y., Jang, J., Jang, J., & Seo, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.533 · Cited by 9
Liu, Q., Chen, J., Yang, H. S., & Yin, Z. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3343743 · Cited by 9
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning
Feng, A., Ying, R., & Tassiulas, L. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.858 · Cited by 9
A Foundation Model for Wearable Movement Data in Mental Health Research
Ruan, F., Zhang, A., Oh, J. Y., Jin, S., & Jacobson, N. C. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3694809 · Cited by 9
Evaluation of predictions of disordered binding regions in the CAID2 experiment
Zhang, F., & Kurgan, L. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.009 · Cited by 9
Andronov, M. G., Andronova, N., Wand, M., Schmidhuber, J., & Clevert, D.-A. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-00974-w · Cited by 9
On the Locality of Attention in Direct Speech Translation
Alastruey, B., Ferrando, J., Gállego, G. I., & Costa‐jussà, M. R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.32 · Cited by 9
Chen, Z., Siltala-Li, L., Lassila, M., Malo, P., Vilkkumaa, E., Saaresranta, T., & Virkki, A. V. (2023) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2023.3276943 · Cited by 9
Document level contexts for neural machine translation
Rauf, S. A., & Yvon, F. (2020) · HAL · Cited by 9
Zhao, C., & Gong, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.00776 · Cited by 9
Xie, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.14517 · Cited by 9
Kandathil, S. M., Greener, J. G., Lau, A. M., & Jones, D. T. (2020) · doi:10.1101/2020.11.27.401232 · Cited by 8
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., & Santoro, A. (2024) · arXiv · doi:10.48550/arxiv.2404.02258 · Cited by 8
“This has spurred tremendous interest in making transformer architectures more efficient (Gupta and Agrawal, 2021; Tay et al., 2020).”
BackgroundHamamoto, R., Koyama, T., Takahashi, S., Yasuda, T., Kobayashi, K., Akagi, Y., Kouno, N., Sudo, K., Hirata, M., Sunami, K., et al. (2026) · Journal of Hematology & Oncology · doi:10.1186/s13045-026-01781-y · Cited by 8
ADA framework for unsupervised domain adaptation person re-identification
Zhang, W., Ye, P., Chen, D., & Su, T. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112238 · Cited by 8
Li, S., Liu, Z., Liu, X., Chen, G., Zhang, Y., Ling, H., & Huang, X. G. (2025) · IEEE Access · doi:10.1109/access.2025.3611811 · Cited by 8
Liang, H., Liang, S., Jiang, B., He, T., Tian, F., Ma, H., Xu, J., Li, W., Ma, Y., Zhang, F., et al. (2025) · Earth system science data · doi:10.5194/essd-17-5571-2025 · Cited by 8
Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers
Zhang, Y., & Zhou, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1097 · Cited by 8
Spectral attention for transformers: frequency-domain filtering of attention maps
Huang, Z., Wu, P., Chen, M., Li, Q., & Pan, M. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00599-5 · Cited by 8
Liu, Y., Jia, Z., Gao, X., Xu, K., & Xiong, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792700 · Cited by 8
Compute Cost Amortized Transformer for Streaming ASR
Xie, Y., Macoskey, J. J., Radfar, M., Chang, F.-J., King, B., Rastrow, A., Mouchtaris, A., & Strimel, G. P. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10465 · Cited by 8
Content-based and Knowledge-enriched Representations for Classification Across Modalities: A Survey
Pittaras, N., Giannakopoulos, G., Stamatopoulos, P., & Karkaletsis, V. (2023) · ACM Computing Surveys · doi:10.1145/3583682 · Cited by 8
Bhowmick, R. S., Ganguli, I., Paul, A., Paul, J., & Sil, J. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3606695 · Cited by 8
CAST: A convolutional attention spatiotemporal network for predictive learning
Sun, F., & Jin, W. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04750-x · Cited by 8
Liang, P., Chen, J., Yao, L., Yu, Y., Liang, K., & Chang, Q. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/aced79 · Cited by 8
How Much Can We Gain From Tensor Kernel Fusion on GPUs?
Sun, W., Li, A., Stuijk, S., & Corporaal, H. (2024) · IEEE Access · doi:10.1109/access.2024.3411473 · Cited by 8
CoSAEmb: Contrastive Section-aware Aspect Embeddings for Scientific Articles
Singh, S., & Singh, M. (2024) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2024.sdp-1.27 · Cited by 8
DEF-Net: A novel dual-branch efficient fusion network for polyp segmentation
Zhai, C., Song, M., Yang, L., Liang, K., & Liu, Y. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107257 · Cited by 8
Kernel approximation using analogue in-memory computing
Büchel, J., Camposampiero, G., Vasilopoulos, A., Lammie, C., Gallo, M. L., Rahimi, A., & Sebastian, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00943-2 · Cited by 8
Fang, M., Zhang, F., Zhu, D., Xiao, R., & Tao, R. (2024) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2024.2443118 · Cited by 8
DBA: Efficient Transformer With Dynamic Bilinear Low-Rank Attention
Qin, B., Li, J., Tang, S., & Zhuang, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3527046 · Cited by 8
Financial Risk Assessment via Long-term Payment Behavior Sequence Folding
Qiao, Y., Tang, Y., Ao, X., Yuan, Q., Liu, Z., Shen, C., & Zheng, X. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00048 · Cited by 8
A Survey of Long-Document Retrieval in the PLM and LLM Era
Li, M., Zhang, Y., Lv, T., Zhao, S., Luo, M., Nie, E., & Zhou, G. (2026) · ACM Transactions on Information Systems · doi:10.1145/3849382 · Cited by 8
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.