Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 901–1,000 of 1,010
Consistent and efficient long document understanding
Zeng, Q. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 8
Beyond Paragraphs: NLP for Long Sequences
Beltagy, I., Cohan, A., Hajishirzi, H., Min, S., & Peters, M. E. (2021) · doi:10.18653/v1/2021.naacl-tutorials.5 · Cited by 8
Li, Z., Ying, S., Wang, J., He, H., & Shi, J. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3238714 · Cited by 8
Can we Pretrain a SotA Legal Language Model on a Budget From Scratch?
Niklaus, J., & Giofré, D. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.11 · Cited by 8
Composition, Attention, or Both?
Yoshida, R., & Oseki, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.428 · Cited by 8
Multipath Delay Estimation in Complex Environments using Transformer
Ott, J., Stahlke, M., Kram, S., Feigl, T., & Mutschler, C. (2023) · International Conference on Indoor Positioning and Indoor Navigation · doi:10.1109/ipin57070.2023.10332470 · Cited by 8
Electrical Equipment Fault Diagnosis: A Technique Combining Fuzzy Logic and Large Language Models
Xu, T., & Tang, X. (2023) · IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA) · doi:10.1109/ispce-asia60405.2023.10365878 · Cited by 8
Contextual Dependency Vision Transformer for spectrogram-based multivariate time series analysis
Yao, J., Han, L., Yang, K., Guo, G., Liu, N., Huang, X., Zheng, Z., Zhang, D., & Han, J. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127215 · Cited by 8
A Feature-Oriented Reconstruction Method for Surface-Defect Detection on Aluminum Profiles
Shancheng, T., Zhang, Y., Jin, Z., Lu, J., Li, H., & Jiqing, Y. (2023) · Applied Sciences · doi:10.3390/app14010386 · Cited by 8
Huo, Y., Gang, S., Liang, D., & Guan, C. (2024) · Applied Sciences · doi:10.3390/app14104075 · Cited by 8
Transformer Meets Twicing: Harnessing Unattended Residual Information
Abdullaev, L. U., & Nguyen, T. M. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2503.00687 · Cited by 8
Neural Comb Filtering Using Sliding Window Attention Network for Speech Enhancement
Parvathala, V., Andhavarapu, S., Pamisetty, G., & Murty, K. S. R. (2022) · Circuits Systems and Signal Processing · doi:10.1007/s00034-022-02123-2 · Cited by 8
Once-for-All Sequence Compression for Self-Supervised Speech Models
Chen, H.-J., Meng, Y., & Lee, H.-Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095025 · Cited by 8
MSGSA: Multi-Scale Guided Self-Attention Network for Crowd Counting
Sun, Y., Li, M., Guo, H., & Zhang, L. (2023) · Electronics · doi:10.3390/electronics12122631 · Cited by 8
Purchase Intent Forecasting with Convolutional Hierarchical Transformer Networks
Huang, C., Zhao, J., & Yin, D. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00281 · Cited by 8
GroupBERT: Enhanced Transformer Architecture with Efficient Grouped StructuresPreprint
Chelombiev, I., Justus, D., Orr, D., Dietrich, A., Gressmann, F., Koliousis, A., & Luschi, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05822 · Cited by 8
Efficient Transformer for Direct Speech TranslationPreprint
Alastruey, B., Gállego, G. I., & Costa‐jussà, M. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03069 · Cited by 8
Smart Bird: Learnable Sparse Attention for Efficient and Effective TransformerPreprint
Wu, C., Wu, F., Qi, T., Jiao, B., Jiang, D., Huang, Y., & Xie, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.09193 · Cited by 8
Transformer with Fourier Integral AttentionsPreprint
Nguyen, T., Pham, M., Nguyen, T., Nguyen, K., Osher, S., & Ho, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.00206 · Cited by 8
Linear Video Transformer with Feature FixationPreprint
Lu, K., Liu, Z., Wang, J., Sun, W., Qin, Z., Li, D., Shen, X., Deng, H., Han, X., Dai, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.08164 · Cited by 8
Multiscale Attention via Wavelet Neural Operators for Vision TransformersPreprint
Nekoozadeh, A., Ahmadzadeh, M. R., & Mardani, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.12398 · Cited by 8
Efficiency Pentathlon: A Standardized Arena for Efficiency EvaluationPreprint
Peng, H., Cao, Q., Dodge, J., Peters, M. E., Fernandez, J., Sherborne, T., Lo, K., Skjonsberg, S., Strubell, E., Plessas, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.09701 · Cited by 8
Investigating Recurrent Transformers with Dynamic HaltPreprint
Chowdhury, J. R., & Caragea, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.00976 · Cited by 8
Learning to Continually Learn with the Bayesian PrinciplePreprint
Lee, S., Jeon, H., Son, J., & Kim, G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.18758 · Cited by 8
The Role of Sparsity for Length Generalization in TransformersPreprint
Golowich, N., Jelassi, S., Brandfonbrener, D., Kakade, S. M., & Malach, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.16792 · Cited by 8
Trellis: Learning to Compress Key-Value Memory in Attention ModelsPreprint
Karami, M., Behrouz, A., Kacham, P., & Mirrokni, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.23852 · Cited by 8
Utility-Guided Agent Orchestration for Efficient LLM Tool UsePreprint
Liu, B., Zhao, G., & Xu, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.19896 · Cited by 8
Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-DesignPreprint
Pepe, A., Lin, C.-Y., Magka, D., Acun, B., Foster, T. S., Wu, Y. N., Protopopov, A., Wu, C.-J., & Bachrach, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.15871 · Cited by 8
Hsieh, H.-Y., Liu, K., & Kang, S. (2025) · Journal of Civil Engineering and Management · doi:10.3846/jcem.2025.24094 · Cited by 7
A review of speaker verification: Methods, network architectures, tasks and challenges
Wang, W., Zhao, H., Yang, Y., & Yang, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112351 · Cited by 7
InEKFormer: A Hybrid State Estimator for Humanoid Robots
Hohmeyer, L., Popescu, M., Bergonzani, I., Mronga, D., & Kirchner, F. (2025) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar65334.2025.11338635 · Cited by 7
Mancini, F., Precoma, V. A. S., Oliveira, L. A. D., & Teixeira, F. O. (2026) · Electronic Commerce Research · doi:10.1007/s10660-025-10071-7 · Cited by 7
LWM-Temporal: Sparse Spatio-Temporal Attention for Wireless Channel Representation Learning
Alikhani, S., Malhotra, A., Hamidi-Rad, S., & Alkhateeb, A. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11586824 · Cited by 7
Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing
Kim, Y., Kim, J., Park, J.-H., Lee, M., & Lee, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.976 · Cited by 7
Peng, B., Chen, Z., Parthasarathy, S., & Ning, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3676560 · Cited by 7
Miao, Y., Sun, Z., Lin, C., Gu, H., Ma, C., Liang, Y., & Wang, G. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae377 · Cited by 7
DARKER: Efficient Transformer with Data-Driven Attention Mechanism for Time Series
Zuo, R., Li, G., Cao, R., Choi, B., Xu, J., & Bhowmick, S. S. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681996 · Cited by 7
Fusing Attention and Convolution: A Hybrid Model for Brain Stroke Prediction
Bhuvanya, R., Kujani, T., & Sivakumar, K. (2024) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7022 · Cited by 7
A Deep-Unfolding Approach to RIS Phase Shift Optimization Via Transformer-Based Channel Prediction
Koralege, I. R., de Sena, A. S., Mahmood, N. H., Karim, F., Lesthuruge, D. L. S., & Gunarathne, S. (2025) · Linköping electronic conference proceedings · doi:10.3384/ecp212.060 · Cited by 7
Singh, M., & Singla, S. K. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.060863 · Cited by 7
Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognition
Parcollet, T., van Dalen, R. C., Zhang, S., & Bhattacharya, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889844 · Cited by 7
Temporal focal modulation networks for sleep stage scoring
Zan, H. (2025) · Pattern Analysis and Applications · doi:10.1007/s10044-025-01475-0 · Cited by 7
Continuous Speech Tokenizer in Text To Speech
Li, Y., Xie, R., Sun, X., Cheng, Y., & Kang, Z. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.184 · Cited by 7
Temporal knowledge graph diffusion model for open-world reasoning
Wang, Y., Zhao, X., Tan, Z., Xiao, W., & Cheng, X. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4329-3 · Cited by 7
FINN-T: Compiling Custom Dataflow Accelerators for Quantized Transformers
Berganski, C., Jentzsch, F. P., Platzner, M., Kuhmichel, M., & Giefers, H. (2024) · International Conference on Field-Programmable Technology (FPT) · doi:10.1109/icfpt64416.2024.11113391 · Cited by 7
Safeguarding user data privacy in online Large Language Model services
Bai, T., Feng, Y., & Fu, S. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103555 · Cited by 7
Fast-FNet: Accelerating transformer encoder models via efficient fourier layers
Sevim, N., Özyedek, E. O., Şahinuç, F., Ozates, T., & Koç, A. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04552-3 · Cited by 7
Spectraformer: A Unified Random Feature Framework for Transformer
Nguyen, D., Yin, D., Joshi, A., & Salim, F. D. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768161 · Cited by 7
Magnetic resonance image processing transformer for general accelerated image restoration
Shen, G., Li, M., Anderson, S. W., Farris, C. W., & Zhang, X. (2025) · Scientific Reports · doi:10.1038/s41598-025-23851-w · Cited by 7
Schwenke, L., Bloemheuvel, S., & Atzmueller, M. (2023) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.36.133109 · Cited by 7
Shen, X., Lam, W. P., Ma, S., & Wang, H. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000177 · Cited by 7
Spatio-Temporal Pre-Training Enhanced Fast Pure Tansformer Network for Traffic Flow Forecasting
Zhang, J., Tang, J., Jin, J., & Qu, Z. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191072 · Cited by 7
Luan, H., Hu, T., Hu, J., Li, R., Ji, D., He, J., Duan, X., Yang, C., Gao, Y., Chen, F., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7074-2_12 · Cited by 7
Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document
Chen, X., Xiao, Q., Li, J., Dong, D., Lin, J., Liu, X., & Tang, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.107 · Cited by 7
Proxyformer: Nyström-Based Linear Transformer with Trainable Proxy Tokens
Lee, S., Lee, H., & Shin, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29244 · Cited by 7
The Energy-Efficient 10-Chiplet AI Hyperscale NPU on Large-Scale Advanced Package
Yoon, J., Kwon, Y.-S., Kim, H., Lee, J., Kim, J., Kim, S., Jang, H., Ahn, K., Kim, J., Hwang, T., et al. (2024) · IEEE Electronic Components and Technology Conference (ECTC) · doi:10.1109/ectc51529.2024.00279 · Cited by 7
Block Access Pattern Discovery via Compressed Full Tensor Transformer
Li, X., Shi, Q., Hu, G., Chen, L., Mao, H. Z., Yang, Y., Yuan, M., Zeng, J., & Cheng, Z. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482323 · Cited by 7
Transformer with Memory Replay
Liu, R., & Mozafari, B. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20722 · Cited by 7
Transformers for Object Detection in Large Point Clouds
Ruppel, F., Faion, F., Gläser, C., & Dietmayer, K. (2022) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9921840 · Cited by 7
Time-Series Forecasting With Shape Attention
Huang, F., Yi, P., Wang, J., Li, M., & Peng, J. (2022) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53654.2022.9945543 · Cited by 7
The Engage Corpus: A Social Media Dataset for Text-Based Recommender Systems
Cheng, D., Yan, K., Keung, P., & Smith, N. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5hcj5k8wy6rq · Cited by 7
Attention-based model and deep reinforcement learning for distribution of event processing tasks
Mazayev, A., AL-Tam, F., & Correia, N. (2022) · Internet of Things · doi:10.1016/j.iot.2022.100563 · Cited by 7
Malceiver: Perceiver with Hierarchical and Multi-modal Features for Android Malware DetectionPreprint
McLaughlin, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05994 · Cited by 7
Text Detection & Recognition in the Wild for Robot LocalizationPreprint
Raisi, Z., & Zelek, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08565 · Cited by 7
Treeformer: Dense Gradient Trees for Efficient Attention ComputationPreprint
Madaan, L., Bhojanapalli, S., Jain, H., & Jain, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.09015 · Cited by 7
Marefat, A., Joloudari, J. H., & Rastgarpour, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3067927/v1 · Cited by 7
Understanding Video Transformers for Segmentation: A Survey of Application and InterpretabilityPreprint
Karim, R., & Wildes, R. P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12296 · Cited by 7
Evaluation of Language Models in the Medical Context Under Resource-Constrained SettingsPreprint
Posada, A., Rueckert, D., Meissen, F., & Müller, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.16611 · Cited by 7
Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMsPreprint
Drinkall, F., Pierrehumbert, J. B., & Zohren, S. (2024) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2407.17624 · Cited by 7
Best of Both Worlds: Advantages of Hybrid Graph Sequence ModelsPreprint
Behrouz, A., Parviz, A., Karami, M., Sanford, C., Perozzi, B., & Mirrokni, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.15671 · Cited by 7
Domain Adaptation and Generalization Using Foundation Models in Healthcare ImagingPreprint
Ying, H., Lia, Y., & Fu, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5345726 · Cited by 7
Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix MultiplicationPreprint
Shi, J., Li, S., Xu, Y., Wang, X., Fu, R., Ma, Z., & Wu, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.22714 · Cited by 7
Image Recognition with Online Lightweight Vision Transformer: A SurveyPreprint
Zhang, Z., Xu, R., Zhou, J., Wang, C., Pei, X., Xu, W., Zhang, J., Guo, L., Gao, L., Xu, W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.03113 · Cited by 7
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent SpacePreprint
FSVideo Team, Chen, Q., Fang, Z., Huang, H., Huang, X., Jin, T., Lin, M., Liu, B., Liu, C., Ma, C., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.02092 · Cited by 7
Context Distillation as Latent Memory ManagementPreprint
Zheng, Z., Li, Z., Wen, X., Zhong, J., Huang, J., Chen, L., Yuan, M., & Xu, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.28889 · Cited by 7
Li, W., Yi, X., Hu, J., Sun, M., & Xie, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.745 · Cited by 6
LongT5-Mulla: LongT5 With Multi-Level Local Attention for a Longer Sequence
Zhou, L. (2023) · IEEE Access · doi:10.1109/access.2023.3340854 · Cited by 6
Decomposition Based Interference Management Framework for Local 6G Networks
Gunarathne, S., Sivalingam, T., Mahmood, N. H., Rajatheva, N., & Latva‐aho, M. (2023) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps58843.2023.10464770 · Cited by 6
Zheng, W., & Wang, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102612 · Cited by 6
Unsupervised Visual Anomaly Detection Using Self-Supervised Pre-Trained Transformer
Kim, J., & Kwon, G. (2024) · IEEE Access · doi:10.1109/access.2024.3454753 · Cited by 6
A dynamic attention mechanism for object detection in road or strip environments
Zhang, G., Zhang, W., Li, W., Wang, L., & Cui, H. (2024) · The Visual Computer · doi:10.1007/s00371-024-03653-3 · Cited by 6
Yuan, H., Hicks, P., Ahmadian, M., Johnson, K. A., Valtadoros, L., & Krishnan, A. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae652 · Cited by 6
Ullah, R., Zhang, S., Asif, M., & Wahab, F. E. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110082 · Cited by 6
A Skillful Prediction of Monsoon Intraseasonal Oscillation Using Deep Learning
Anirudh, K. M., Raj, P., Sandeep, S., Kodamana, H., & Sabeerali, C. T. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2024jh000504 · Cited by 6
Cui, Z., Zang, D., Zhang, J., & Tang, K. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3577826 · Cited by 6
Dynamic Mixture of Experts for Adaptive Computation in Character-Level Transformers
Huang, Z., Chen, M., & Zheng, S. (2025) · Information · doi:10.3390/info16060483 · Cited by 6
Entropy-Guided KV Caching for Efficient LLM Inference
Kim, H.-K., & Jung, Y. (2025) · Mathematics · doi:10.3390/math13152366 · Cited by 6
BRP-SpMM: Block-Row Partition Based Sparse Matrix Multiplication with Tensor and CUDA Cores
Dong, Y., Jiang, W., Shen, X., Guo, H., Shao, Z., & Jin, H. (2025) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps64566.2025.00085 · Cited by 6
Ahmadi, M. A. (2025) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2025.150885 · Cited by 6
Nwaiwu, S., Jongsawat, N., & Tungkasthan, A. (2025) · Results in Engineering · doi:10.1016/j.rineng.2025.107783 · Cited by 6
Accelerating training speed of tiny recursive models with curriculum guided adaptive recursion
Qasim, K. U., Zhang, J., Alharith, R., & Shaheen, M. K. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109720 · Cited by 6
ARC-Encoder: learning compressed text representations for large language models
Pilchen, H., Grave, É., & Pérez, P. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2510.20535 · Cited by 6
Prabhu, O., & Navada, S. G. (2025) · Systems Science & Control Engineering · doi:10.1080/21642583.2025.2600795 · Cited by 6
Simple linear attention language models balance the recall-throughput tradeoff
Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zinsley, D., Zou, J., Rudra, A., & Ré, C. (2024) · arXiv · doi:10.48550/arxiv.2402.18668 · Cited by 5
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
Yuan, Z., Shang, Y., Song, Y., Yang, D., Wu, Q., Yan, Y., & Sun, G. (2023) · arXiv · doi:10.48550/arxiv.2312.05821 · Cited by 4
“A significant amount of memory and computational overhead is incurred by Transformers’ linear weight matrices, especially in its multi-head attention (MHA) and feed-forward layers [Tay et al., 2022].”
BackgroundZoology: Measuring and Improving Recall in Efficient Language Models
Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., & Ré, C. (2023) · arXiv · doi:10.48550/arxiv.2312.04927 · Cited by 4
Characterizing verbatim short-term memory in neural language models
Armeni, K., Honey, C., & Linzen, T. (2022) · Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2022.conll-1.28 · Cited by 4
A Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU
Li, H., Choi, J., & Ahn, J. H. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00019 · Cited by 3
“Sparse transformers alleviated these problems through SA, and recently numerous studies have been actively conducted, reporting state-of-the-art (SOTA) results [4, 35, 46].”
BackgroundLanguage Model Cascades: Token-level uncertainty and beyond
Gupta, N., Narasimhan, H., Jitkrittum, W., Rawat, A. S., Menon, A. K., & Kumar, S. (2024) · arXiv · doi:10.48550/arxiv.2404.10136 · Cited by 3
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.