Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 701–800 of 1,010
Universal Hopfield networks: a general framework for single-shot associative memory models
Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., & Bogacz, R. (2022) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 15
Occluded Text Detection and Recognition in the Wild
Raisi, Z., & Zelek, J. (2022) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv55824.2022.00026 · Cited by 15
Exploiting stacked embeddings with LSTM for multilingual humor and irony detection
Tasnia, R., Ayman, N., Sultana, A., Chy, A. N., & Aono, M. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01049-0 · Cited by 15
Efficient Transformer Knowledge Distillation: A Performance Review
Brown, N., Williamson, A., Anderson, T., & Lawrence, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.6 · Cited by 15
Franke, J. K. H., Runge, F., & Hutter, F. (2022) · neural information processing systems · doi:10.52202/068431-1947 · Cited by 15
Show Me What You’re Looking For
Schwenke, L., & Atzmueller, M. (2021) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.v34i1.128399 · Cited by 15
Deep Convolutional Network with Pixel-Aware Attention for Smoke Recognition
Cheng, G., Chen, X., & Gong, J. (2022) · Fire Technology · doi:10.1007/s10694-022-01231-4 · Cited by 15
Chilkuri, N., Hunsberger, E., Voelker, A. R., Malik, G. S., & Eliasmith, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02402 · Cited by 15
Linear Complexity Randomized Self-attention MechanismPreprint
Lin, Z., Wang, C., & Kong, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.04667 · Cited by 15
Feng, L., Senapati, J., & Liu, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.11187 · Cited by 15
A Survey of Learning on Small Data: Generalization, Optimization, and ChallengePreprint
Cao, X., Bu, W., Huang, S.-J., Zhang, M., Tsang, I. W.-H., Ong, Y. S., & Kwok, J. T.-Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.14443 · Cited by 15
Expediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning
Yuan, Y., Liang, W., Ding, H., Liang, Z., Zhang, C., & Hu, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327511 · Cited by 14
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., & Sun, M. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3801 · Cited by 14
Munthuli, A., Socatiyanurak, V., Sangchocanonta, S., Kovudhikulrungsri, L., Saksakulkunakorn, N., Chairuangsri, P., & Tantibundhit, C. (2023) · IEEE Access · doi:10.1109/access.2023.3296261 · Cited by 14
Piankarnka, V., Lertbumroongchai, K., & Piriyasurawong, P. (2023) · Advances in Human-Computer Interaction · doi:10.1155/2023/5230762 · Cited by 14
Distilling dynamical knowledge from stochastic reaction networks
Liu, C., & Wang, J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317422121 · Cited by 14
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
Wang, H., Fan, Y., Naeem, M. F., Xian, Y., Lenssen, J. E., Wang, L., Tombari, F., & Schiele, B. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2410.23168 · Cited by 14
Giri, S., Du, D., & Beruvides, M. G. (2025) · Cogent Economics & Finance · doi:10.1080/23322039.2025.2468387 · Cited by 14
An Effective Masked Transformer Model for Automatic Modulation Recognition
Kong, W., Jiao, X., Xu, Y., & Yang, Q. (2025) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2025.3550729 · Cited by 14
Zhu, L., Lu, W., & Luo, C. (2025) · Journal of Hydrology · doi:10.1016/j.jhydrol.2025.133237 · Cited by 14
UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering
Liu, L., Liu, S., Yuan, Y., Zhang, Y., Yan, B., Zeng, Z., Wang, Z., Liu, J., Wang, D., Su, W., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737385 · Cited by 14
Are vision transformers replacing convolutional neural networks in scene interpretation?: A review
Rosy, N. A., Balasubadra, K., & Deepa, K. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07574-1 · Cited by 14
Bispo, D. A., Leite, G. D. N. P., Droguett, E. L., de Souza, O. V. C., Lisboa, L. A., Cavalcanti, G. D. C., Ochoa, Á. A. V., da Costa, A. C. A., Vilela, O. D. C., Brennand, L. J. D. P., et al. (2025) · Energies · doi:10.3390/en18236162 · Cited by 14
How does momentum benefit deep neural networks architecture design? A few case studies
Wang, B., Xia, H., Nguyen, T., & Osher, S. (2022) · Research in the Mathematical Sciences · doi:10.1007/s40687-022-00352-0 · Cited by 14
Long-Document Cross-Lingual Summarization
Zheng, S., Li, Z., Wang, J., Qu, J., Liu, A., Zhao, L., & Chen, Z. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570479 · Cited by 14
Attention And Positional Encoding Are (Almost) All You Need For Shape Matching
Raganato, A., Pasi, G., & Melzi, S. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14912 · Cited by 14
Design tactics for tailoring transformer architectures to cybersecurity challenges
Salma, C. A., Tekinerdoğan, B., & Catal, C. (2024) · Cluster Computing · doi:10.1007/s10586-024-04355-0 · Cited by 14
Do we really need a large number of visual prompts?
Kim, Y., Li, Y., Moitra, A., Yin, R., & Panda, P. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106390 · Cited by 14
Recasting Continual Learning as Sequence Modeling
Lee, S., Son, J., & Kim, G. (2023) · neural information processing systems · doi:10.52202/075280-3086 · Cited by 14
Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy NetworksPreprint
Yao, J., Köttering, P., Gundlach, H., Lin, L., & Bukov, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06701 · Cited by 14
Adaptive Filters and Aggregator Fusion for Efficient Graph Convolutions.Preprint
Tailor, S. A., Opolka, F. L., Lió, P., & Lane, N. D. (2021) · arXiv (Cornell University) · Cited by 14
Continual Transformers: Redundancy-Free Attention for Online InferencePreprint
Hedegaard, L., Bakhtiarnia, A., & Iosifidis, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2201.06268 · Cited by 14
Benchmarking General-Purpose In-Context LearningPreprint
Wang, F., Lin, C., Cao, Y., & Kang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.17234 · Cited by 14
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Du, X., Ferdinan, T., Hou, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.05892 · Cited by 13
Can Language Models Be Used in Multistep Commonsense Planning Domains?
Tang, Z., & Kejriwal, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33469-6_28 · Cited by 13
A Rewiring Contrastive Patch PerformerMixer Framework for Graph Representation Learning
Sun, Z., Harit, A., Cristea, A. I., Wang, J., & Lió, P. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386951 · Cited by 13
Automatic Graph Topology-Aware Transformer
Wang, C., Zhao, J., Li, L., Jiao, L., Liu, F., & Yang, S. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3440269 · Cited by 13
A framework for measuring the training efficiency of a neural architecture
Cueto-Mendoza, E., & Kelleher, J. D. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10943-8 · Cited by 13
Xu, W. M., Fan, J., Wang, H., & Cai, L. (2025) · Sensors · doi:10.3390/s25061950 · Cited by 13
Adaptive learning network for detecting pavement distresses in complex environments
Wang, D. W., Zhang, A. A., Peng, Y. L., Wei, Y.-J., Cheng, H., & Shang, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110784 · Cited by 13
iTransformer-FFC: A Frequency-Aware Transformer Framework for Multi-Scale Time Series Forecasting
Tang, Y., & Cai, Z. (2025) · Electronics · doi:10.3390/electronics14122378 · Cited by 13
Xie, Z.-A., Chow, C., Chuah, J. H., & Raymond, W. J. K. (2025) · IEEE Access · doi:10.1109/access.2025.3595679 · Cited by 13
Exploring the Limitations of Mamba in COPY and CoT Reasoning
Ren, R., Li, Z., & Liu, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.634 · Cited by 13
A survey of foundation models for IoT: taxonomy and criteria-based analysis
Wei, H., Lee, D. Y., Rohal, S., Hu, Z., Rossi, R. A., Fang, S., & Pan, S. (2025) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-025-00208-w · Cited by 13
Knowledge Aware State-Space Capsule Network for Multivariate Time Series Classification
Xiao, Z., Wan, Q., Ding, W., Song, F., & Tong, H. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3692700 · Cited by 13
MiTU-Net: An Efficient Mix Transformer U-like Network for Forward-looking Sonar Image Segmentation
Liang, Y., Zhu, X., & Zhang, J. (2022) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai55564.2022.9807763 · Cited by 13
Maneeroj, S., & Sritrakool, N. (2022) · IEEE Access · doi:10.1109/access.2022.3182390 · Cited by 13
Multivariate Geostatistical Simulation and Deep Q-Learning to Optimize Mining Decisions
Avalos, S., & Ortíz, J. M. (2023) · Mathematical Geosciences · doi:10.1007/s11004-023-10049-x · Cited by 13
Role of Bias Terms in Dot-Product Attention
Namazifar, M., Hazarika, D., & Hakkani-Tur, D. Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097125 · Cited by 13
HRCTNet: a hybrid network with high-resolution representation for object detection in UAV image
Xing, W., Cui, Z., & Qi, J. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01076-6 · Cited by 13
PSST! Prosodic Speech Segmentation with Transformers
Roll, N., Graham, C., & Todd, S. W. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.31 · Cited by 13
Song, T., Garza, P., Meo, M., & Munafò, M. M. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110507 · Cited by 13
Coneheads: Hierarchy Aware Attention
Tseng, A., Yu, T., Liu, T., & De, C. (2023) · neural information processing systems · doi:10.52202/075280-2239 · Cited by 13
Patch-based stochastic attention for image editing
Cherel, N., Almansa, A., Gousseau, Y., & Newson, A. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103866 · Cited by 13
Infrared and visible image fusion using a guiding network to leverage perceptual similarity
Kim, J., & Hwang, Y. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103598 · Cited by 13
Li, Y., Shi, Z., & Liu, C. (2023) · IISE Transactions · doi:10.1080/24725854.2023.2193257 · Cited by 13
Focus Your Attention (with Adaptive IIR Filters)
Lutati, S., Zimerman, I., & Wolf, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.772 · Cited by 13
Tempo: Accelerating Transformer-Based Model Training Through Memory Footprint Reduction
Andoorveedu, M., Zhu, Z., Zheng, B., & Pekhimenko, G. G. (2022) · neural information processing systems · doi:10.52202/068431-0891 · Cited by 13
Zheng, W., Guo, Q., Yang, H., Wang, P., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15926 · Cited by 13
Understanding Optimization of Deep Learning via Jacobian Matrix and Lipschitz ConstantPreprint
Qi, X., Wang, J., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.09338 · Cited by 13
Differential Privacy Under Class Imbalance: Methods and Empirical InsightsPreprint
Rosenblatt, L., Lut, Y., Turok, E., Avella-Medina, M., & Cummings, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05733 · Cited by 13
PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented GenerationPreprint
Khaledian, A., Ghadiridehkordi, A., & Khaledian, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.08386 · Cited by 13
Schäfke, H., Lampe, N., & Kortmann, K.-P. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10384175 · Cited by 12
Deep context transformer: bridging efficiency and contextual understanding of transformer models
Ghaith, S. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05453-7 · Cited by 12
Two-Dimensional Laplace NMR Reconstruction through Deep Learning Enhancement
Chen, B., Fang, Z., Zhang, Y., Guan, X., Lin, E., Feng, H. B., Zeng, Y., Cai, S., Yang, Y., Huang, Y., et al. (2024) · Journal of the American Chemical Society · doi:10.1021/jacs.4c05211 · Cited by 12
StampOne: Addressing Frequency Balance in Printer-proof Steganography
Shadmand, F., Medvedev, I. O., Schirmer, L., Marcos, J., & Gonçalves, N. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00440 · Cited by 12
Transformer-Based Approach for Automatic Semantic Financial Document Verification
Toprak, A., & Turan, M. (2024) · IEEE Access · doi:10.1109/access.2024.3477270 · Cited by 12
Visual State Space Model for Image Super-Resolution
Zhang, J., Gao, M., Li, W., Fang, D., & Li, C. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3522426 · Cited by 12
Machine learning tools match physician accuracy in multilingual text annotation
Zielonka, M., Czyżewski, A., Szplit, D., Graff, B., Szyndler, A., Budzisz, M., & Narkiewicz, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-89754-y · Cited by 12
A scalable machine learning strategy for resource allocation in database
Manhary, F. N., Mohamed, M. H., & Farouk, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-14962-5 · Cited by 12
Lin, C., Zhang, X., & Zhao, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-16268-y · Cited by 12
Milyani, A. H., & Attar, E. T. (2025) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2025.1668935 · Cited by 12
Attention Mechanism, Max-Affine Partition, and Universal Approximation
Liu, H., Hu, J. Y.-C., Song, Z., & Liu, H. (2025) · neural information processing systems · doi:10.52202/085713-4020 · Cited by 12
Tracking with attention: A review of transformer-based object tracking
Khoshnevis, S. A., & Amirkhani, A. (2025) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2025.102263 · Cited by 12
Yang, G., Jin, N., Ai, W., Zheng, Z., He, Y., & He, Y. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111495 · Cited by 12
From modality-specific to compositional foundation models for cell biology
Bahrami, M., Richter, T., Schmacke, N. A., Lavandera, A. E., & Theis, F. J. (2026) · Cell Systems · doi:10.1016/j.cels.2026.101534 · Cited by 12
A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Code Smell Detection
Zhang, B., Liang, P., Zhou, X., Zhou, X., Lo, D., Feng, Q., Li, Z., & Li, L. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3806390 · Cited by 12
Long-Document RAG for Contractual and Insurance Clause Analysis in Receivables RWA Structures
Zhou, S., Li, Z., & Wang, E. (2024) · Journal of Advanced Computing Systems · doi:10.69987/jacs.2024.40810 · Cited by 12
Wang, J., Zhang, Q., & Liu, G. (2022) · Measurement Science and Technology · doi:10.1088/1361-6501/ac6fb2 · Cited by 12
Locality-Sensitive Hashing for Long Context Neural Machine Translation
Petrick, F., Rosendahl, J., Herold, C., & Ney, H. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.4 · Cited by 12
ASSET: autoregressive semantic scene editing with transformers at high resolutions
Liu, D., Shetty, S., Hinz, T., Fisher, M., Zhang, R., Park, T., & Kalogerakis, E. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530172 · Cited by 12
Couplformer: Rethinking Vision Transformer with Coupling Attention
Lan, H. P., Wang, X., Shen, H., Liang, P., & Wei, X. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00641 · Cited by 12
PMA-Net: A parallelly mixed attention network for person re-identification
Qu, J., Zhang, Y., & Zhang, Z. (2023) · Displays · doi:10.1016/j.displa.2023.102437 · Cited by 12
MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers
Nouriborji, M., Rohanian, O., Kouchaki, S., & Clifton, D. A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.83 · Cited by 12
Zhao, X., Wang, S., Wang, H., He, T., Zhang, D., & Wang, G. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614918 · Cited by 12
Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers
Rau, D., Dehghani, M., & Kamps, J. (2024) · ACM Transactions on Information Systems · doi:10.1145/3640460 · Cited by 12
Self-citationChain and Causal Attention for Efficient Entity Tracking
Fagnou, E., Caillon, P., Delattre, B., & Allauzen, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.731 · Cited by 12
Investigations into the Efficiency of Computer-Aided Synthesis Planning
Hartog, P. B. R., Westerlund, A. M., Tetko, I. V., & Genheden, S. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01821 · Cited by 12
Nielsen, S. K., Abdullaev, L. U., Teo, R. S. Y., & Nguyen, T. M. (2024) · neural information processing systems · doi:10.52202/079017-3482 · Cited by 12
Schwenke, L., & Atzmueller, M. (2021) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa53316.2021.9564126 · Cited by 12
LSTN:Long Short-Term Traffic Flow Forecasting with Transformer Networks
Yang, L., Ren, Q., Hu, J., & Han, M. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956632 · Cited by 12
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction
Milbauer, J., Louis, A. P., Hosseini, M. J., Fabrikant, A., Metzler, D., & Schuster, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.571 · Cited by 12
Self-citationFMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field AttentionPreprint
Nguyen, T. M., Suliafu, V., Osher, S., Chen, L., & Bao, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02347 · Cited by 12
A Comprehensive Survey of Transformers for Computer VisionPreprint
Jamil, S., Piran, M. J., & Kwon, O. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.06004 · Cited by 12
Blockwise Parallel Transformer for Large Context ModelsPreprint
Liu, H., & Abbeel, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.19370 · Cited by 12
DE-HNN: An effective neural model for Circuit Netlist representationPreprint
Luo, Z., Hy, T. S., Tabaghi, P., Koh, D., Defferrard, M., Rezaei, E., Carey, R., Davis, W. R., Jain, R., & Wang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.00477 · Cited by 12
Bio-Eng-LMM AI Assist chatbot: A Comprehensive Tool for Research and EducationPreprint
Forootani, A., Aliabadi, D. E., & Thraen, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.07110 · Cited by 12
The Linear Attention Resurrection in Vision TransformerPreprint
Zheng, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.16182 · Cited by 12
(2025) · doi:10.52202/085713-0602 · Cited by 11
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., & Metzler, D. (2022) · arXiv · doi:10.48550/arxiv.2207.10551 · Cited by 11
Self-citation
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.