Efficient Transformers: A Survey
Open accessThe DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Abstract
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of “X-former” models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“One key development in the area is the use of transformer architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of autoregressive 1 self-supervised 2 language models to handle longrange dependencies in natural-language texts.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [3]
In the citing paper
“We note that the performance advantage becomes more pronounced as the size of cached segments grows since the computation overhead of attention states scales quadratically with input sequence size (Keles et al., 2022; Tay et al., 2023) while the space and compute complexity of Prompt Cache scales linearly with the size.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“The modular KV cache construction in Prompt Cache bears resemblance to the approximations observed in locally masked attention (Beltagy et al., 2020; Tay et al., 2023), which optimizes computations by setting a limited window for attention score calculations rather than spanning its attention across every token in its input sequence.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Tay et al., 2023
In the citing paper
“Conversely, models that rely on premature embeddings demonstrate inferior performance compared to models that utilize manually acquired features (Borgeaud et al. 2022; Schwartz and Dodge 2020; Tay et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Tay et al. 2022
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4×
“Transformers are pervasive and have made tremendous impact in many fields such as language understanding (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2019) and image processing (Parmar et al., 2018; Carion et al., 2020).”
BackgroundInfluentialEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015) · arXiv · doi:10.48550/arxiv.1511.07289 · Cited by 4,513
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 2×
“Similarly, Albert [42] does the same parameter sharing across layers.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 6×
“…al., 2020) Linformer (Wang et al., 2020b) Linear Transformer (Katharopouloset al., 2020) Set Transformer (Lee et al., 2019) Transformer-XL (Dai et al., 2019) Memory Compressed (Liu et al., 2018) ETC (Ainslie et al., 2020) Sparse Transformer (Child et al., 2019) Image Transformer (Parmar…”
BackgroundMethodInfluentialCCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
Convergence Results for Neural Networks via Electrodynamics
Panigrahy, R., Rahimi, A., Sachdeva, S., & Zhang, Q. (2018) · Leibniz international proceedings in informatics · doi:10.4230/lipics.itcs.2018.22 · Cited by 2,761
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Self-citationAdvances in Neural Information Processing Systems 32 (NeurIPS 2019)
Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché–Buc, F., Fox, E. B., & Garnett, R. (2020) · HAL · Cited by 2,451
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · cited 1×
“An example is to prune attention heads during inference (Voita et al., 2019; Michel et al., 2019).”
BackgroundParmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085 · cited 7×
“Examples of models that do this include Blockwise (Qiu et al., 2019) and/or Local Attention (Parmar et al., 2018).”
BackgroundMethodInfluentialGenerating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices
Sinkhorn, R. (1964) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177703591 · Cited by 1,032
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 976
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 8×
“On the other hand, the Linformer (Wang et al., 2020c) focuses on subsets of the GLUE (Wang et al., 2018) benchmark.”
BackgroundMethodInfluentialTransformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827
Efficiently Modeling Long Sequences with Structured State Spaces
Gu, A., Goel, K., & Ré, C. (2021) · arXiv · doi:10.48550/arxiv.2111.00396 · Cited by 805
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 13×
“…et al., 2020) O ( n ( k + m )) FP+M ETC (Ainslie et al., 2020) O ( N 2 g + NN g ) FP+M Synthesizer (Tay et al., 2020a) O ( N 2 ) LR+LP Performer (Choromanski et al., 2020a) O ( N ) KR Funnel Transformer (Dai et al., 2020) O ( N 2 ) FP+DS Linformer (Wang et al., 2020c) O ( N ) LR Linear…”
BackgroundMethodInfluentialfairseq: A Fast, Extensible Toolkit for Sequence ModelingPreprint
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01038 · Cited by 563
FNet: Mixing Tokens with Fourier Transforms
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontanon, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.319 · Cited by 519
Efficient Content-Based Sparse Attention with Routing Transformers
Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509 · cited 10×
“For example, the k -means clustering in Routing Transformer (Roy et al., 2020) can also be interpreted as a form of global model memory approach, since one can view the centroids as parameterized model memory.”
BackgroundMethodInfluentialDehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv · doi:10.48550/arxiv.1807.03819 · Cited by 467
Self-citationQ-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6409 · Cited by 437
MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
Pfeiffer, J., Vulić, I., Gurevych, I., & Ruder, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.617 · Cited by 437
Pointer Sentinel Mixture Models
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07843 · Cited by 427
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404
Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376
Axial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370
FNet: Mixing Tokens with Fourier TransformsPreprint
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontañón, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03824 · Cited by 338
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 297
Random Feature AttentionPreprint
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 293
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingPreprint
Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03236 · Cited by 286
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, E., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1032 · Cited by 279
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277
ETC: Encoding Long and Structured Inputs in Transformers
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S., Wang, Q., & Yang, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.19 · Cited by 268
Synthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251
Self-citationLong Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241 · cited 2×
“There have been recent attempts to unify evaluation on Efficient Transformers, namely Long Range Arena, i.e., LRA, (Tay et al., 2021a) that benchmarked 10 different xformer variants on long range modeling tasks.”
BackgroundMethodSelf-citationSparse Sinkhorn AttentionPreprint
Tay, Y., Bahri, D., Liu, Y., Metzler, D., & Juan, D.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11296 · Cited by 237
Self-citationDepthwise Separable Convolutions for Neural Machine Translation
Kaiser, Ł., Gomez, A. N., & Chollet, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03059 · Cited by 234
BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
Stickland, A. C., & Murray, I. (2019) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 231
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Wang, H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., & Han, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.686 · Cited by 227
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1223 · Cited by 198
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, X. D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14795 · Cited by 198
Fast Transformers with Clustered AttentionPreprint
Vyas, A., Katharopoulos, A., & Fleuret, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04825 · Cited by 192
Long-Short Transformer: Efficient Transformers for Language and Vision
Zhu, C., Ping, W., Xiao, C., Shoeybi, M., Goldstein, T., Anandkumar, A., & Catanzaro, B. (2021) · CaltechAUTHORS (California Institute of Technology) · doi:10.48550/arxiv.2107.02192 · Cited by 189
BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task LearningPreprint
Stickland, A. C., & Murray, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02671 · Cited by 163
Blockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 161
Ranking via Sinkhorn PropagationPreprint
Adams, R. P., & Zemel, R. S. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.1925 · Cited by 160
Adaptive Attention Span in TransformersPreprint
Sukhbaatar, S., Grave, É., Bojanowski, P., & Joulin, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.07799 · Cited by 146
Training with Quantization Noise for Extreme Model Compression
Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2004.07320 · Cited by 145
TinyBERT: Distilling BERT for Natural Language UnderstandingPreprint
Jiao, X., Yin, Y., Shang, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10351 · Cited by 133
The Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 129
Self-citationTransformer Memory as a Differentiable Search Index
Tay, Y., Tran, V., Dehghani, M., Ni, J., Bahri, D., Mehta, H., Qin, Z., Hui, K., Zhao, Z., Gupta, J., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1587 · Cited by 128
Self-citationMasked Language Modeling for Proteins via Linearly Scalable Long-Context TransformersPreprint
Choromański, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J. Q., Belanger, D., Colwell, L. J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03555 · Cited by 127
Block Pruning For Faster Transformers
Lagunas, F., Charlaix, E., Sanh, V., & Rush, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.829 · Cited by 123
Sparse is Enough in Scaling TransformersPreprint
Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, Ł., Gajewski, W., Michalewski, H., & Kanerva, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12763 · Cited by 122
Weighted Transformer Network for Machine Translation
Ahmed, K., Keskar, N. S., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02132 · Cited by 118
Adaptively Sparse TransformersPreprint
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00015 · Cited by 115
Poolingformer: Long Document Modeling with Pooling AttentionPreprint
Zhang, H., Gong, Y., Shen, Y., Li, W., Lv, J., Duan, N., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04371 · Cited by 107
Scaling Autoregressive Video Models
Weissenborn, D., Täckström, O., & Uszkoreit, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02634 · Cited by 93
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2106.12672 · Cited by 92 · cited 2×
“Even though many existing ‘memory’ models utilize a form of downsampling to achieve a speed and efficiency gain, we erected a new categoriation of ‘downsampling’ to better reflect this new emerging trend (Dai et al., 2020; Jaegle et al., 2021; Tay et al., 2021c; Ryoo et al., 2021).”
BackgroundMethodSelf-citationBASE Layers: Simplifying Training of Large, Sparse Models
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2103.16716 · Cited by 88
Transformer Memory as a Differentiable Search IndexPreprint
Tay, Y., Tran, V. Q., Dehghani, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06991 · Cited by 80
Self-citationConsistent Accelerated Inference via Confident Adaptive Transformers
Schuster, T., Fisch, A., Jaakkola, T., & Barzilay, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.406 · Cited by 80
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse TransformersPreprint
Yun, C., Chang, Y.-W., Bhojanapalli, S., Rawat, A. S., Reddi, S. J., & Kumar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04862 · Cited by 78
Augmenting Self-attention with Persistent Memory
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75 · cited 5×
“This can be thought of as a form of parameter attention (Sukhbaatar et al., 2019).”
BackgroundInfluentialLightweight and Efficient End-To-End Speech Recognition Using Low-Rank Transformer
Winata, G. I., Cahyawijaya, S., Lin, Z., Liu, Z., & Fung, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053878 · Cited by 73
Large Memory Layers with Product Keys
Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H. (2019) · arXiv · doi:10.48550/arxiv.1907.05242 · Cited by 72 · cited 7×
“…2021) O ( kN ) M TokenLearner (Ryoo et al., 2021) O ( k 2 ) DS Adaptive Sparse Transformer (Correia et al., 2019) O ( N 2 ) Sparse Product Key Memory (Lample et al., 2019) O ( N 2 ) Sparse Switch Transformer (Fedus et al., 2021) O ( N 2 ) Sparse ST-MoE (Zoph et al., 2022) O ( N 2 ) Sparse…”
BackgroundMethodInfluentialHash Layers For Large Sparse Models
Roller, S., Sukhbaatar, S., Szlam, A., & Weston, J. (2021) · arXiv · doi:10.48550/arxiv.2106.04426 · Cited by 66
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Tay, Y., Zhang, A., Luu, A. T., Rao, J., Zhang, S., Wang, S., Fu, J., & Hui, S. C. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1145 · Cited by 65
Self-citationAre Pretrained Convolutions Better than Pretrained Transformers?
Tay, Y., Dehghani, M., Gupta, J. P., Aribandi, V., Bahri, D., Qin, Z., & Metzler, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.335 · Cited by 56
Self-citationCluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency EncodingPreprint
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06097 · Cited by 36
Are Pre-trained Convolutions Better than Pre-trained Transformers?Preprint
Tay, Y., Dehghani, M., Gupta, J. P., Bahri, D., Aribandi, V., Qin, Z., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03322 · Cited by 33
Self-citationLightweight and Efficient Neural Natural Language Processing with Quaternion NetworksPreprint
Tay, Y., Zhang, A., Tuan, L. A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04393 · Cited by 16
Self-citationBlockwise Self-Attention for Long Document UnderstandingPreprint
Qiu, J., Ma, H., Levy, O., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02972 · Cited by 15
Training with Quantization Noise for Extreme Fixed-Point CompressionPreprint
Fan, A., Stock, P., Graham, B., Grave, É., Gribonval, R., Jeǵou, H., & Joulin, A. (2020) · arXiv (Cornell University) · Cited by 9
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper ProjectionsPreprint
Tay, Y., Zhao, Z., Bahri, D., Metzler, D., & Juan, D.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05891 · Cited by 9
Self-citationTransformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv · doi:10.48550/arxiv.1901.02860
Research works printed but not linked: 14
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016). · arXiv
- Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]. · arXiv
- Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).
- Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737–748.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=rkgNKkHtvB.
- Alex Krizhevsky Geoffrey Hinton et al. 2009. Learning multiple layers of features from tiny images. Master thesis.
- Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744–3753.
- Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).
- Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021.
- Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68–80.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https://openreview.net/forum?id=SylKikSYDH.
- Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS).
- Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems.
Cited by
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
“Onekeydevelopment in the area is the use of trans-former architectures [2, 3] and the underlying attention mechanism [4], which have greatly improved the ability of auto-regressive 1 self-supervised 2 language models to handle long-1 Auto-regressive because the model uses its previous predictions…”
MethodDeformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027
“Recently, many efforts have been made to address this problem (Tay et al., 2020b), which can be roughly divided into three categories in practice.”
BackgroundTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Metadata
- Authors
- Tay, Y., Dehghani, M., Bahri, D., & Metzler, D.
- Affiliations
- Google (United States) (US)
- Venue
- ACM Computing Surveys
- Published
- 2022-12-07
- Volume
- 55
- Issue
- 6
- Pages
- 1-28
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3085139254
- Cited by
- 1,267
- References
- 98
- Citations by year
- 2026: 240 · 2025: 288 · 2024: 192 · 2023: 258 · 2022: 135
- FWCI
- 71.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Computing Surveys (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- transformer models, efficient transformers, reformers, Linformer, performers, Longformer, computational efficiency, memory efficiency, natural language processing, vision transformer, reinforcement learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.