Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
Items 601–700 of 1,010
Linear attention based spatiotemporal multi graph GCN for traffic flow prediction
Zhang, Y., Xu, W., Ma, B., Zhang, D., Zeng, F., Yao, J., Yang, H., & Du, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-93179-y · Cited by 20
Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning
She, J., Gupta, J. K., & Kochenderfer, M. J. (2022) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/ybxd5829 · Cited by 20
Hypoformer: Hybrid Decomposition Transformer for Edge-friendly Neural Machine Translation
Li, S., Zhang, P., Gan, G., Lv, X., Wang, B., Wei, V. J., & Jiang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.475 · Cited by 20
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model
Liu, Z., Dettmers, T., Lin, X., Stoyanov, V., & Li, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.930 · Cited by 20
On the Long Range Abilities of TransformersPreprint
Zimerman, I., & Wolf, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.16620 · Cited by 20
Xu, C., Huang, Y.-C., Hu, J. Y.-C., Li, W., Gilani, A., Goan, H., & Liu, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.03830 · Cited by 20
PRAGMA: A Profiling-Reasoned Multi-Agent Framework for Automatic Kernel OptimizationPreprint
Lei, K., Yang, H., Zhang, H., You, X., Zhang, K., Luan, Z., Liu, Y., & Qian, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.06345 · Cited by 20
An Empirical Study of Mamba-based Language Models
Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07887 · Cited by 19
Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models
Millidge, B., Salvatori, T., Song, Y., Lukasiewicz, T., & Bogacz, R. (2022) · arXiv · doi:10.48550/arxiv.2202.04557 · Cited by 19
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
Huang, Z., Ling, G., Zhong, S., Wu, H., & Lin, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.560 · Cited by 19
Wei, D., Wang, Z., Kang, H., Sha, X., Xie, Y., Dai, A., & Ouyang, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-14610-y · Cited by 19
Ibáñez, M. B., Estrada, M. L. B., & Cabada, R. Z. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70080 · Cited by 19
Exploiting neuro-inspired dynamic sparsity for energy-efficient intelligent perception
Zhou, S., Gao, C., Delbrück, T., Verhelst, M., & Liu, S. (2025) · Nature Communications · doi:10.1038/s41467-025-65387-7 · Cited by 19
Self-Supervised Bi-Channel Transformer Networks for Computer-Aided Diagnosis
Gong, R., Han, X., Wang, J., Ying, S., & Shi, J. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3153902 · Cited by 19
Long-range Sequence Modeling with Predictable Sparse Attention
Zhuang, Y., Zhang, J., & Tu, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.19 · Cited by 19
On Compressing Sequences for Self-Supervised Speech Models
Meng, Y., Chen, H.-J., Shi, J., Watanabe, S., Garcia, L. P., Lee, H.-Y., & Tang, H. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022991 · Cited by 19
Hybridformer: Improving Squeezeformer with Hybrid Attention and NSR Mechanism
Yang, Y., Yu, P., Yin, J., Han, J., Ma, L., & Lu, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096467 · Cited by 19
Classification of Alzheimer's using Deep-learning Methods on Webcam-based Gaze Data
Harisinghani, A., Sriram, H., Conati, C., Carenini, G., Field, T. S., Jang, H., & Murray, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3591126 · Cited by 19
MUSTIE: Multimodal Structural Transformer for Web Information Extraction
Wang, Q., Wang, J., Quan, X., Feng, F., Xu, Z., Nie, S., Wang, S., Khabsa, M., Firooz, H., & Liu, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.135 · Cited by 19
Fading memory as inductive bias in residual recurrent networks
Dubinin, I., & Effenberger, F. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106179 · Cited by 19
Modeling and prediction of key parameters of circulating fluidized bed boiler based on Transformer
Ye, Y., Lin, H., & Zhou, H. (2024) · Journal of the Taiwan Institute of Chemical Engineers · doi:10.1016/j.jtice.2024.105622 · Cited by 19
Compare, M., & Zio, E. (2024) · Journal of Reliability Science and Engineering · doi:10.1088/3050-2454/ad9f63 · Cited by 19
ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-Term Tracking
Liu, H., Gao, M., Luo, X., Wang, Z., Qin, G., Wu, J., & Jin, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05127-1_42 · Cited by 19
Score Transformer: Generating Musical Score from Note-level Representation
Suzuki, M. (2021) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3469877.3490612 · Cited by 19
Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models
Ang, P., Dhingra, B., & Wills, L. W. (2022) · doi:10.18653/v1/2022.nlppower-1.12 · Cited by 19
BERTuit : Understanding Spanish language in Twitter with transformers
Huertas‐Tato, J., Martín, A. G., & Camacho, D. (2023) · Expert Systems · doi:10.1111/exsy.13404 · Cited by 19
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?Preprint
Niklaus, J., & Giofré, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.17135 · Cited by 19
Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clustersPreprint
Shyam, V., Pilault, J., Shepperd, E., Anthony, Q. G., & Millidge, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.04093 · Cited by 19
Training Report of TeleChat3-MoEPreprint
Liu, X., Wang, C., Yang, Z., Jiang, Z., Zhao, X., Wang, H., Li, L., He, D., Liu, L., Yuan, K., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.24157 · Cited by 19
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
Gim, I., Chen, G., Lee, S.-S., Sarda, N., Khandelwal, A., & Zhong, L. (2023) · arXiv · doi:10.48550/arxiv.2311.04934 · Cited by 18
“…becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size. prompts.”
BackgroundHiRID-ICU-Benchmark - A Comprehensive Machine Learning Benchmark on High-resolution ICU Data
Yèche, H., Kuznetsova, R., Zimmermann, M., Hüser, M., Lyu, X., Faltys, M., & Rätsch, G. (2021) · arXiv · doi:10.48550/arxiv.2111.08536 · Cited by 18
Weiss, G., Goldberg, Y., & Yahav, E. (2021) · arXiv · doi:10.48550/arxiv.2106.06981 · Cited by 18
Fan, J., Yu, G.-A., Zhao, M., & Zong, H. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.131081 · Cited by 18
Karita, S., Sproat, R., & Ishikawa, H. (2023) · doi:10.18653/v1/2023.cawl-1.8 · Cited by 18
Lutakamale, A. S., Myburgh, H. C., & Freitas, A. D. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108349 · Cited by 18
ETransCap: efficient transformer for image captioning
Mundu, A., Singh, S. K., & Dubey, S. R. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05739-w · Cited by 18
Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural Network
Li, Z., & Ma, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3512352 · Cited by 18
Lu, Z., Zhou, H., Wang, L., Kong, D., Lyu, H., Wu, H., Chen, B., Chen, F., Dong, N., & Yang, G. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3560812 · Cited by 18
Language Modeling with Editable External Knowledge
Li, B. Z., Liu, E., Ross, A., Zeitoun, A., Neubig, G., & Andreas, J. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.168 · Cited by 18
HGPflow: extending hypergraph particle flow to collider event reconstruction
Kakati, N., Dreyer, E., Ivina, A., Bello, F. A. D., Heinrich, L., Kado, M., & Gross, E. (2025) · The European Physical Journal C · doi:10.1140/epjc/s10052-025-14443-z · Cited by 18
MCaM : Efficient LLM Inference with Multi-tier KV Cache Management
Chu, K., Shen, Z., Cheng, S.-R., Xiang, D., Liu, Z., & Zhang, W. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00062 · Cited by 18
Plug-and-Play Document Modules for Pre-trained Models
Xiao, C., Zhang, Z., Han, X., Chan, C.-M., Lin, Y., Liu, Z., Li, X., Li, Z., Cao, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.875 · Cited by 18
Expressing Multivariate Time Series as Graphs with Time Series Attention TransformerPreprint
Ng, W. T., Siu, K., Cheung, A. C., & Ng, M. K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.09300 · Cited by 18
ChordMixer: A Scalable Neural Attention Model for Sequences with Different LengthsPreprint
Khalitov, R., Yu, T., Cheng, L., & Yang, Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.05852 · Cited by 18
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
“Transformer models emerged as a powerful alternative due to their ability to handle both local and long-range dependencies and their capability for parallelized training (Tay et al., 2022).”
BackgroundHardware Accelerator for Approximation-Based Softmax and Layer Normalization in Transformers
Kim, R., Lee, D., Kim, J., Park, J., & Lee, S. E. (2025) · Electronics · doi:10.3390/electronics14122337 · Cited by 17
GraphShield: Advanced dynamic graph-based malware detection using graph neural networks
Amer, E., El–Sappagh, S., Abuhamad, T., Al‐rimy, B. A. S., & Mohasseb, A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129812 · Cited by 17
Constituent-constrained word prediction during language comprehension
Zou, J., Poeppel, D., & Ding, N. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02272-6 · Cited by 17
Modular and Parameter-Efficient Fine-Tuning for NLP Models
Ruder, S., Pfeiffer, J., & Vulić, I. (2022) · doi:10.18653/v1/2022.emnlp-tutorials.5 · Cited by 17
Yoo, J., Kim, S., Lee, D., Kim, C., & Hong, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02192 · Cited by 17
Deep Learning Approaches for Big Data-Driven Metadata Extraction in Online Job Postings
Skondras, P., Ζώτος, Ν., Lagios, D., Zervas, P., Giotopoulos, K. C., & Tzimas, G. E. (2023) · Information · doi:10.3390/info14110585 · Cited by 17
Asymptotic Consistent Graph Structure Learning for Multivariate Time-Series Anomaly Detection
Pang, H., Wei, S., Li, Y., Liu, T., Zhang, H., Qin, Y. H., & Zhao, Y. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3369159 · Cited by 17
Yang, Y., Zheng, Y., Zou, Q., Li, J., & Feng, H. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012340 · Cited by 17
Aeroelastic force prediction via temporal fusion transformers
Montoya, M. C., Mishra, A., Verma, S., Mures, O. A., & Rubio-Medrano, C. E. (2024) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13381 · Cited by 17
SPLAT: A Framework for Optimised GPU Code-Generation for SParse reguLar ATtention
Gupta, A., Yuan, Y., Jain, D., Ge, Y., Aponte, D., Zhou, Y., & Mendis, C. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3720503 · Cited by 17
Towards multimodal graph large language model
Wang, X. Y., Zhang, Z., Xiao, L., Chen, H., Ge, C., & Zhu, W. (2025) · Science China Information Sciences · doi:10.1007/s11432-025-4627-3 · Cited by 17
Zhang, G., & Abdulla, W. H. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.113947 · Cited by 18
Towards efficient and reliable artificial intelligence through neuromorphic principles
Rajendran, B., Simeone, O., & Al‐Hashimi, B. M. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2024.0522 · Cited by 17
Improving reference mining in patents with BERT
Voskuil, K. S., & Verberne, S. (2021) · Leiden Repository (Leiden University) · doi:10.48550/arxiv.2101.01039 · Cited by 17
ReadTwice: Reading Very Large Documents with Memories
Zemlyanskiy, Y., Ainslie, J., de Jong, M., Pham, P., Eckstein, I., & Sha, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.408 · Cited by 17
Mask usage recognition using vision transformer with transfer learning and data augmentation
Jahja, H. D., Yudistira, N., & Sutrisno (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200186 · Cited by 17
BERT for Long Documents: A Case Study of Automated ICD Coding
Afkanpour, A., Adeel, S., Bassani, H. F., Epshteyn, A., Fan, H., Jones, I., Malihi, M., Nauth, A., Sinha, R. K., Woonna, S., et al. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.12 · Cited by 17
SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts
Kimura, M., Nakamura, T. K. M., & Saito, Y. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00355 · Cited by 17
HyperConformer: Multi-head HyperMixer for Efficient Speech Recognition
Mai, F., Zuluaga-Gómez, J., Parcollet, T., & Motlíček, P. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1611 · Cited by 17
Designing Robust Transformers using Robust Kernel Density Estimation
Han, X., Ren, T., Nguyen, T., Nguyen, K., Ghosh, J., & Ho, N. (2023) · neural information processing systems · doi:10.52202/075280-2323 · Cited by 17
Reservoir TransformersPreprint
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15045 · Cited by 17
LoCoCo: Dropping In Convolutions for Long Context CompressionPreprint
Cai, R., Tian, Y., Wang, Z., & Chen, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.05317 · Cited by 17
Wavelets Are All You Need for Autoregressive Image GenerationPreprint
Mattar, W., Levy, I., Sharon, N., & Dekel, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.19997 · Cited by 17
LongCoder: A Long-Range Pre-trained Language Model for Code Completion
Guo, D., Xu, C., Duan, N., Yin, J., & McAuley, J. (2023) · arXiv · doi:10.48550/arxiv.2306.14893 · Cited by 16
Small and Fast LLMs on Commodity Hardware: Post-Training Quantization in llama. cpp
Sparrenberg, L., Deußer, T., Berger, A., & Sifa, R. (2025) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa65442.2025.11247985 · Cited by 16
Mahmud, M. I., & Banerjee, T. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag229 · Cited by 16
FastSeq: Make Sequence Generation Faster
Yan, Y., Hu, F., Chen, J., Bhendawade, N., Ye, T., Gong, Y., Duan, N., Cui, D., Chi, B., & Zhang, R. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.26 · Cited by 16
Sepúlveda-Torres, R., Vicente, M., Saquete, E., Lloret, E., & Palomar, M. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102176 · Cited by 16
Attention and self-attention in random forests
Utkin, L. V., Konstantinov, A. V., & Kirpichenko, S. R. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00301-0 · Cited by 16
High-Quality Reconstruction for Laplace NMR Based on Deep Learning
Chen, B., Wu, L., Cui, X., Lin, E., Cao, S., Zhan, H., Huang, Y., Yang, Y., & Chen, Z. (2023) · Analytical Chemistry · doi:10.1021/acs.analchem.3c00537 · Cited by 16
TransMap: Pinpointing Mistakes in Neural Code Translation
Wang, B., Li, R., Li, M., & Saxena, P. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616322 · Cited by 16
Train Long and Test Long:Leveraging Full Document Contexts in Speech Processing
Chen, W., Kano, T., Ogawa, A., Delcroix, M., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446727 · Cited by 16
HDT: Hierarchical Document Transformer
He, H., Flicke, M., Buchmann, J., Gurevych, I., & Geiger, A. (2024) · TUbilio (Technical University of Darmstadt) · doi:10.26083/tuprints-00030063 · Cited by 16
From Darkness to Detail: Frequency-Aware SSMs for Low-Light Vision
Adhikarla, E., Zhang, K., Chen, G., Nicholson, J., & Davison, B. D. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00672 · Cited by 16
Feature-driven hybrid attention learning for accurate water quality prediction
Yao, X., Xu, Z., Ren, T., & Zeng, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127160 · Cited by 16
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
Shihab, I. F., Akter, S., & Sharma, A. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.562 · Cited by 16
Exploiting ICD Hierarchy for Classification of EHRs in Spanish Through Multi-Task Transformers
Blanco, A., Pérez, A., & Casillas, A. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3112130 · Cited by 16
SensorFormer: Efficient Many-to-Many Sensor Calibration With Learnable Input Subsampling
Cheng, Y., Saukh, O., & Thiele, L. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3177948 · Cited by 16
Chiplet Heterogeneous-Integration AI Processor
Kwon, Y.-S., Han, J., Cho, Y. P., Kim, J., Chung, J., Choi, J., Park, S., Kim, I., Kwon, H., Kim, J., et al. (2023) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic57457.2023.10049867 · Cited by 16
Cahyawijaya, S. (2021) · arXiv.org · doi:10.14711/thesis-991012980421703412 · Cited by 16
Ligandformer: A Graph Neural Network for Predicting Compound Property with Robust InterpretationPreprint
Guo, J., Liu, Q., Guo, H. W., & Lu, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.10873 · Cited by 16
Advances in Transformers for Robotic Applications: A ReviewPreprint
Sanghai, N., & Brown, N. B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.10599 · Cited by 16
Investigating Length Issues in Document-level Machine TranslationPreprint
Peng, Z., Bawden, R., & Yvon, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.17592 · Cited by 16
∞-former: Infinite Memory Transformer-former: Infinite Memory Transformer
Martins, P. H., Marinho, Z., & Martins, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.375 · Cited by 15
Prediction of Currency Exchange Rate Based on Transformers
Zhao, L., & Yan, W. (2024) · Journal of risk and financial management · doi:10.3390/jrfm17080332 · Cited by 15
Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
Ling, T., Qian, C., & Schiele, G. (2024) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot63253.2024.00017 · Cited by 15
Li, Y., Liu, S., Zhang, Y., Liu, X., & Xia, M. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3478249 · Cited by 15
Reparameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
Zhang, Z., Dong, Z., Xu, W., & Han, J. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3483232 · Cited by 15
Lıu, M., Kong, X., Xiong, K., Wang, J., & Lin, Q. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.124963 · Cited by 15
Rahman, M. H., Uddin, M. A., Ria, Z. F., & Rahman, R. M. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.058329 · Cited by 15
Artificial Intelligence in Medical Metaverse: Applications, Challenges, and Future Prospects
Yang, J., Chen, B., Li, R., Huang, B.-Q., Zhao, M.-H., Liu, P., Zhang, J., & Ye, Z. (2024) · Current Medical Science · doi:10.1007/s11596-024-2960-5 · Cited by 15
A Mamba-based foundation model for materials
Soares, E. A., Brazil, E. V., Shirasuna, V. Y., Zubarev, D. Y., Cerqueira, R., & Schmidt, K. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00009-7 · Cited by 15
Document Structure in Long Document Transformers
Buchmann, J., Eichler, M., Bodensohn, J.-M., Kuznetsov, I., & Gurevych, I. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.64 · Cited by 15
FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
Cai, K., Zhang, J., Ma, S., Lu, B., Wang, J., & Wang, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37086-0_33 · Cited by 15
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.