Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 501–600 of 1,010
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer
Wang, N., Gan, G., Zhang, P., Zhang, S., Wei, V. J., Liu, Q., & Jiang, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.170 · Cited by 28
Local–Global Transformer Neural Network for temporal action segmentation
Tian, X., Jin, Y., & Tang, X. (2022) · Multimedia Systems · doi:10.1007/s00530-022-00998-4 · Cited by 28
Semi-Dense Feature Matching With Transformers and its Applications in Multiple-View Geometry
Shen, Z., Sun, J., Wang, Y., He, X., Bao, H., & Zhou, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3223530 · Cited by 28
RawFormer: An Efficient Vision Transformer for Low-Light RAW Image Enhancement
Xu, W., Dong, X., Ma, L., Teoh, A. B. J., & Lin, Z. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3233005 · Cited by 28
Detection of Epileptic Seizures Based-on Channel Fusion and Transformer Network in EEG Recordings
Yauri, J., Lagos, M., Vega-Huerta, H., de Villa, P. E. D. L. C. V., Maquen-Niño, G. L. E., & Tinoco, E. E. C. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.01405110 · Cited by 28
Reducing Sequence Length by Predicting Edit Spans with Large Language Models
Kaneko, M., & Okazaki, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.619 · Cited by 28
Dodo: Dynamic Contextual Compression for Decoder-only LMs
Qin, G., Rosset, C., Chau, E. C., Rao, N., & Durme, B. V. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.536 · Cited by 28
Synthetic Aperture Radar Imaging Meets Deep Unfolded Learning: A comprehensive review
Wang, M., Hu, Y., Wei, S., Shi, J., Cui, G., Kong, L., & Guo, Y. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3494754 · Cited by 27
Conotoxins: Classification, Prediction, and Future Directions in Bioinformatics
Li, R., Yu, J., Ye, D.-X., Liu, S., Zhang, H., Lin, H., Feng, J., & Deng, K. (2025) · Toxins · doi:10.3390/toxins17020078 · Cited by 27
STL-ELM: A computationally efficient hybrid approach for predicting high volatility stock market
Kehinde, T. O., Adedokun, O. J., Kareem, M. K., Akpan, J. S., & Olanrewaju, O. A. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200564 · Cited by 27
Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians
Higham, C. F., Higham, D. J., & Grindrod, P. (2025) · SIAM Review · doi:10.1137/23m1626232 · Cited by 27
Xu, W., Ding, Z., Wang, Z., Cui, Z., Hu, Y., & Jiang, F. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3612417 · Cited by 27
Li, X.-J., Yang, J., & Zhang, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19818-2_31 · Cited by 27
Sparse Binary Transformers for Multivariate Time Series Modeling
Gorbett, M., Shirazi, H. M., & Ray, I. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599508 · Cited by 27
Peng, D., Liu, C., Liu, Y., & Jin, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28245 · Cited by 27
ResMatch: Residual Attention Learning for Feature Matching
Deng, Y., Zhang, K., Zhang, S., Li, Y., & Ma, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27915 · Cited by 27
Neural Architecture Search on Efficient Transformers and BeyondPreprint
Liu, Z., Li, D., Lu, K., Qin, Z., Sun, W., Xu, J., & Zhong, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.13955 · Cited by 27
Mobile Vision Transformer-based Visual Object TrackingPreprint
Gopal, G. Y., & Amer, M. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05829 · Cited by 27
Transformer: A Survey and ApplicationPreprint
Song, X., Niu, Q., Liu, J., Peng, B., Zhang, S., Liu, M., Li, M., Wang, T., Pan, X., Xu, J., et al. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/5p2hu · Cited by 27
Principe, R. A. A., Chiarini, N., & Viviani, M. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70019 · Cited by 26
MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
Ding, R., Xu, T., Shen, X., Ding, A. A., & Fei, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765174 · Cited by 26
EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation
Shi, Y., Cai, H., Ansari, A., & Porikli, F. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00017 · Cited by 26
Chen, W., Zhou, H., Cheng, L., & Xia, M. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3329105 · Cited by 26
MotherNet: Fast Training and Inference via Hyper-Network Transformers
Müller, A., Curino, C., & Ramakrishnan, R. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2312.08598 · Cited by 26
A Convolutional Transformer Architecture for Remaining Useful Life Estimation
Ding, Y., & Jia, M. (2021) · Global Reliability and Prognostics and Health Management Conference (PHM) · doi:10.1109/phm-nanjing52125.2021.9612814 · Cited by 26
Sumformer: Universal Approximation for Efficient TransformersPreprint
Alberti, S., Dern, N., Thesing, L., & Kutyniok, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02301 · Cited by 26
Xie, H. M., Dong, Z., Yang, H., Luo, Y., Ren, S., Zhang, P., He, J., Jia, C., Yang, Y., Jiang, M., et al. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112631 · Cited by 25
AI Computing Systems for Large Language Models Training
Zhang, Z., Wen, Y., Lyu, H.-Q., Liu, C., Zhang, R., Li, X., Wang, C., Du, Z., Guo, Q., Li, L., et al. (2025) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4178-1 · Cited by 25
Transformers in Natural Language Processing
Yvon, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24349-3_6 · Cited by 25
Data Ambiguity Profiling for the Generation of Training Examples
Veltri, E., Badaro, G., Saeed, M., & Papotti, P. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00041 · Cited by 25
Towards Query Performance Prediction for Neural Information Retrieval: Challenges and Opportunities
Faggioli, G., Formal, T., Lupart, S., Marchesin, S., Clinchant, S., Ferro, N., & Piwowarski, B. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605142 · Cited by 25
IMP: Iterative Matching and Pose Estimation with Adaptive Pooling
Xue, F., Budvytis, I., & Cipolla, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02042 · Cited by 25
SwipeFormer: Transformers for mobile touchscreen biometrics
Delgado-Santos, P., Tolosana, R., Guest, R., Lamb, P., Khmelnitsky, A., Coughlan, C., & Fiérrez, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121537 · Cited by 25
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
Amos, I., Berant, J., & Gupta, A. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.02980 · Cited by 25
Optimizing a transformer-based network for a deep-learning seismic processing workflow
Harsuko, R., & Alkhalifah, T. A. (2024) · Geophysics · doi:10.1190/geo2023-0403.1 · Cited by 25
TACTiS: Transformer-Attentional Copulas for Time SeriesPreprint
Drouin, A., Marcotte, É., & Chapados, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03528 · Cited by 25
Progressive Gradient Flow for Robust N:M Sparsity Training in TransformersPreprint
Bambhaniya, A. R., Yazdanbakhsh, A., Subramanian, S., Kao, S.-C., Agrawal, S., Evci, U., & Krishna, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04744 · Cited by 25
NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference TimePreprint
Chen, Y., Wang, G., Shang, J., Cui, S., Zhang, Z., Liu, T., Wang, S., Sun, Y., Yu, D., & Wu, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.03675 · Cited by 25
Xie, W. C., Li, Y., Wei, H., & Zhang, L. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3394475 · Cited by 24
Vehicle-to-everything mode of operation technologies: A state-of-art systematic review
Safari, A., & Rahimi, A. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126382 · Cited by 24
Pre-trained Language Model-based Retrieval and Ranking for Web Search
Zou, L., Lu, W., Liu, Y., Cai, H., Chu, X., Ma, D., Shi, D., Sun, Y., Cheng, Z., Gu, S., et al. (2022) · ACM Transactions on the Web · doi:10.1145/3568681 · Cited by 24
Transformer-Based Context Condensation for Boosting Feature Pyramids in Object Detection
Chen, Z., Zhang, J., Xu, Y., & Tao, D. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01830-w · Cited by 24
Mamalis, M. E., Kalampokis, E., Karamanou, A., Brimos, P., & Tarabanis, K. A. (2023) · Pan-Hellenic Conference on Progress in Computing and Informatics (PCI) · doi:10.1145/3635059.3635068 · Cited by 24
Parcollet, T., van Dalen, R. C., Zhang, S., & Bhattacharya, S. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-40 · Cited by 24
Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation
Li, X., Zmigrod, R., Ma, Z., Liu, X., & Zhu, X. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.491 · Cited by 24
Lifting Transformer for 3D Human Pose Estimation in Video.Preprint
Li, W., Liu, H., Ding, R., Liu, M., & Wang, P. (2021) · arXiv (Cornell University) · Cited by 24
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for SequencesPreprint
Zhu, Z., & Soricut, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11906 · Cited by 24
An Empirical Analysis of SMS Scam Detection SystemsPreprint
Salman, M., Ikram, M., & Kâafar, M. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10451 · Cited by 24
Do Efficient Transformers Really Save Computation?Preprint
Yang, K., Ackermann, J., He, Z., Feng, G., Zhang, B., Feng, Y., Ye, Q., He, D., & Wang, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.13934 · Cited by 24
Serialized Point Mamba: A Serialized Point Cloud Mamba Segmentation ModelPreprint
Wang, T., Wen, W., Zhai, J., Xu, K., & Luo, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.12319 · Cited by 24
Data-driven differentiable model for dynamic prediction and control in wastewater treatment
Song, Y., Wang, W., Wang, Y.-Q., Yin, W.-X., Chen, J.-J., Xu, H.-R., Cheng, H.-Y., Ma, F., Wang, H.-T., Wang, A.-J., et al. (2025) · Water Research · doi:10.1016/j.watres.2025.123772 · Cited by 23
Zhu, X., Peng, S., Guo, J., Chen, Y., Guo, Q., Wen, Y., Qin, H., Chen, R., Zhou, Q., Gao, K., et al. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i34.40155 · Cited by 23
A Survey on Sparsity Exploration in Transformer-Based Accelerators
Fuad, K. A. A., & Chen, L. (2023) · Electronics · doi:10.3390/electronics12102299 · Cited by 23
Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset
Amalvy, A., Labatut, V., & Dufour, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.642 · Cited by 23
CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference
Mohtashami, A., Pagliardini, M., & Jaggi, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.10845 · Cited by 23
Ahmad, P. N., Liu, Y., Khan, K., Jiang, T., & Burhan, U. (2023) · Sensors · doi:10.3390/s23239355 · Cited by 23
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.331 · Cited by 23
PermuteFormer: Efficient Relative Position Encoding for Long Sequences
Chen, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.828 · Cited by 23
Compositional Attention: Disentangling Search and RetrievalPreprint
Mittal, S., Raparthy, S. C., Rish, I., Bengio, Y., & Lajoie, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09419 · Cited by 23
Rethinking Data Augmentation for Tabular Data in Deep LearningPreprint
Onishi, S., & Meguro, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10308 · Cited by 23
Curve Your Attention: Mixed-Curvature Transformers for Graph Representation LearningPreprint
Cho, S., Cho, S., Park, S., Lee, H., Lee, H., & Lee, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.04082 · Cited by 23
SPECTRE: An FFT-Based Efficient Drop-In Replacement to Self-Attention for Long ContextsPreprint
Fein-Ashley, J., Gupta, N., Kannan, R., & Prasanna, V. K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18394 · Cited by 23
Dong, Y., Noci, L., Khodak, M., & Li, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.01115 · Cited by 23
Language models for document understanding
Douzon, T. (2023) · theses.fr (ABES) · doi:10.70675/66ce60f8zcccaz476eza4d6zf801a2256b86 · Cited by 23
Sparse attention is all you need for pre-training on tabular data
Isomura, T., Shimizu, R., & Goto, M. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10698-x · Cited by 22
Token Weighting for Long-Range Language Modeling
Helm, F., Daheim, N., & Gurevych, I. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.79 · Cited by 22
Scaling up DNA digital data storage by efficiently predicting DNA hybridisation using deep learning
Buterez, D. (2021) · Scientific Reports · doi:10.1038/s41598-021-97238-y · Cited by 22
Control of Dual-Sourcing Inventory Systems Using Recurrent Neural Networks
Böttcher, L., Asikis, T., & Fragkos, I. (2023) · INFORMS journal on computing · doi:10.1287/ijoc.2022.0136 · Cited by 22
Vagliano, I., Dormosh, N., Rios, M., Luik, T. T., Buonocore, T. M., Elbers, P., Dongelmans, D. A., Schut, M. C., & Abu–Hanna, A. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104504 · Cited by 22
Is ChatGPT the way toward artificial general intelligence
Emmert‐Streib, F. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00126-3 · Cited by 22
Conformer-Based Speech Recognition On Extreme Edge-Computing Devices
Xu, M., Jin, A., Wang, S., Su, M., Ng, T., Mason, H., Han, S., Lei, Z., Deng, Y., Huang, Z., et al. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.12 · Cited by 22
My body is a cage: the role of morphology in graph−based incompatible control
Kurin, V., Igl, M., Rocktaschel, T., Boehmer, W., & Whiteson, S. (2021) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 22
Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient ConversationsPreprint
Zhang, L., Negrinho, R. M. P., Ghosh, A., Jagannathan, V., Hassanzadeh, H. R., Schaaf, T., & Gormley, M. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12174 · Cited by 22
Fundamental Components of Deep Learning: A category-theoretic approachPreprint
Gavranović, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.13001 · Cited by 22
Language Model Embeddings Can Be Sufficient for Bayesian OptimizationPreprint
Nguyễn, T. L., Zhang, Q., Yang, B., Lee, C., Bornschein, J., Miao, Y., Perel, S., Chen, Y., & Song, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.10190 · Cited by 22
Lattice: Learning to Efficiently Compress the MemoryPreprint
Karami, M., Pascanu, R., & Mirrokni, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.05646 · Cited by 22
Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., Lu, Y., Ibarz, J., & Arnab, A. (2022) · arXiv · doi:10.48550/arxiv.2211.09119 · Cited by 21
Salomão, P. E. A. (2026) · Journal International Review of Research Studies · doi:10.66104/kxf7hk05 · Cited by 21
Resource-efficient Hybrid X-formers for Vision
Pranav Jeevan, P., & Sethi, A. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00361 · Cited by 21
Yulita, I. N., Wijaya, V., Rosadi, R., Sarathan, I., Djuyandi, Y., & Prabuwono, A. S. (2023) · Data · doi:10.3390/data8030046 · Cited by 21
Linear Complexity Self-Attention with $3^{\text{rd}}$ Order Polynomials
Babiloni, F., Marras, I., Deng, J., Kokkinos, F., Maggioni, M., Chrysos, G. G., Torr, P. H. S., & Zafeiriou, S. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3231971 · Cited by 21
TS2ARCformer: A Multi-Dimensional Time Series Forecasting Framework for Short-Term Load Prediction
Li, S., Zhang, W., & Wang, P. (2023) · Energies · doi:10.3390/en16155825 · Cited by 21
Zeng, J., Meng, F., Yin, Y., & Zhou, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.786 · Cited by 21
Zhu, X., Sun, S., Lin, L., Wu, Y., & Ma, X. (2024) · Reviews in the Neurosciences · doi:10.1515/revneuro-2024-0088 · Cited by 21
A Diffusion Modeling-Based System for Teaching Dance to Digital Human
Zhou, L., Zhao, J., & He, J. (2024) · Applied Sciences · doi:10.3390/app14199084 · Cited by 21
Boytsov, L., & Kolter, J. Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_5 · Cited by 21
Textual Pre-Trained Models for Gender Identification Across Community Question-Answering Members
Schwarzenberg, P., & Figueroa, A. (2023) · IEEE Access · doi:10.1109/access.2023.3235735 · Cited by 21
Literature review on vulnerability detection using NLP technologyPreprint
Wu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11230 · Cited by 21
Not All Memories are Created Equal: Learning to Forget by ExpiringPreprint
Sukhbaatar, S., Ju, D., Poff, S., Roller, S., Szlam, A. D., Weston, J., & Fan, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.06548 · Cited by 21
Retrieval meets Long Context Large Language Models
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., & Catanzaro, B. (2023) · arXiv · doi:10.48550/arxiv.2310.03025 · Cited by 20
Liu, X., Xu, Y., Xu, H., He, L., Long, S., Huang, Y., Wang, Y., Lu, Y., Huang, Y., Wu, J., et al. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110675 · Cited by 20
Exploring Scaling Laws of CTR Model for Online Performance Improvement
Lai, W., Jin, B., Zhang, J., Zheng, Y., Dong, J., Cheng, J., Lei, J., & Wang, X. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748046 · Cited by 20
Language Modeling by Language Models
Cheng, J., Clark, P. E., & Richardson, K. (2025) · neural information processing systems · doi:10.52202/085713-5704 · Cited by 20
Improved Transformer With Multi-Head Dense Collaboration
Wang, H., Shen, X., Tu, M., Zhuang, Y., & Liu, Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3199648 · Cited by 20
Multi-view fusion neural network for traffic demand prediction
Zhang, D., & Li, J. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119303 · Cited by 20
Rogers, B., Noman, N., Chalup, S. K., & Moscato, P. (2023) · Evolutionary Intelligence · doi:10.1007/s12065-023-00874-8 · Cited by 20
Zhu, R. H., Liang, S., Song, Y., & Cai, Z. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610876 · Cited by 20
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition
Liu, L., Liu, L., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3363446 · Cited by 21
Continuous-time Autoencoders for Regular and Irregular Time Series Imputation
Wi, H., Shin, Y., & Park, N. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635831 · Cited by 20
Yang, X., Yang, L., Li, Y., Zhao, Z., & Zhang, Y. (2024) · Electric Power Systems Research · doi:10.1016/j.epsr.2024.111049 · Cited by 20
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.