CoChat
Sign Up
PreprintCited by 1,420Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

A Survey of Transformers

Open access
Lin, Tianyang, Wang, Yuxin, Liu, Xiangyang, Qiu, Xipeng·2021·arXiv
DOI 10.48550/arxiv.2106.04554arXiv 2106.04554
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.04554

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The self-attention mechanism has transformed deep learning by generating context-aware models that can pay attention to large input spaces and learn which elements are most important in each context, boosting predictions.

    “The advent of the self-attention mechanism1,2 has further transformed the deep learning field by generating context-aware models that are able to pay attention to large input spaces and learn which elements are most important to focus on in each context, boosting predictions in a wide realm of applications2,8.”

    From Transfer learning enables predictions in network biology (2023) · cites this paper as [8]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 171

  1. Ainslie, Joshua, Ontanon, Santiago, Alberti, Chris, Cvicek, Vaclav, Fisher, Zachary, Pham, Philip, Ravula, Anirudh, Sanghai, Sumit, Wang, Qifan, Yang, Li, 2020. ETC: Encoding long and structured inputs in transformers. In: Proceedings of EMNLP. pp. 268-284. http://dx.doi.org/10.18653/v1/2020.emnlp-main.19, Online. · DOI
  2. Al-Rfou, Rami, Choe, Dokook, Constant, Noah, Guo, Mandy, Jones, Llion, 2019. Character-level language modeling with deeper self-attention. In: Proceedings of AAAI. pp. 3159-3166. http://dx.doi.org/10.1609/aaai.v33i01.33013159. · DOI · Paper page
  3. Arnab, Anurag, Dehghani, Mostafa, Heigold, Georg, Sun, Chen, Lučić, Mario, Schmid, Cordelia, 2021. Vivit: A video vision transformer. arXiv:2103.15691. · arXiv
  4. Ba, Lei Jimmy, Kiros, Jamie Ryan, Hinton, Geoffrey E., 2016. Layer normalization. CoRR arXiv:1607.06450. · arXiv
  5. Bachlechner, Thomas, Majumder, Bodhisattwa Prasad, Mao, Huanru Henry, Cot- trell, Garrison W., McAuley, Julian J., 2020. Rezero is all you need: Fast convergence at large depth. CoRR arXiv:2003.04887. · arXiv
  6. Baevski, Alexei, Auli, Michael, 2019. Adaptive input representations for neural lan- guage modeling. In: Proceedings of ICLR. URL https://openreview.net/forum?id= ByxZX20qFQ.
  7. Bapna, Ankur, Arivazhagan, Naveen, Firat, Orhan, 2020. Controlling computation versus quality for neural sequence models. arXiv:2002.07106, [cs.LG]. · arXiv
  8. Bapna, Ankur, Chen, Mia, Firat, Orhan, Cao, Yuan, Wu, Yonghui, 2018. Training deeper neural machine translation models with transparent attention. In: Proceedings of EMNLP. Brussels, Belgium, pp. 3028-3033. http://dx.doi.org/10.18653/v1/D18- 1338. · DOI
  9. Battaglia, Peter W., Hamrick, Jessica B., Bapst, Victor, Sanchez-Gonzalez, Alvaro, Zambaldi, Vinicius, Malinowski, Mateusz, Tacchetti, Andrea, Raposo, David, San- toro, Adam, Faulkner, Ryan, Gulcehre, Caglar, Song, Francis, Ballard, Andrew, Gilmer, Justin, Dahl, George, Vaswani, Ashish, Allen, Kelsey, Nash, Charles, Langston, Victoria, Dyer, Chris, Heess, Nicolas, Wierstra, Daan, Kohli, Pushmeet, Botvinick, Matt, Vinyals, Oriol, Li, Yujia, Pascanu, Razvan, 2018. Relational inductive biases, deep learning, and graph networks. arXiv:1806.01261, [cs.LG]. · arXiv
  10. Beltagy, Iz, Peters, Matthew E., Cohan, Arman, 2020. Longformer: The long-document transformer. arXiv:2004.05150, [cs.]. · arXiv
  11. Bhojanapalli, Srinadh, Yun, Chulhee, Rawat, Ankit Singh, Reddi, Sashank J., Ku- mar, Sanjiv, 2020. Low-rank bottleneck in multi-head attention models. In: Proceedings of ICML. pp. 864-873, URL http://proceedings.mlr.press/v119/ bhojanapalli20a.html.
  12. Brown, Tom, Mann, Benjamin, Ryder, Nick, Subbiah, Melanie, Kaplan, Jared D, Dhari- wal, Prafulla, Neelakantan, Arvind, Shyam, Pranav, Sastry, Girish, Askell, Amanda, Agarwal, Sandhini, Herbert-Voss, Ariel, Krueger, Gretchen, Henighan, Tom, Child, Rewon, Ramesh, Aditya, Ziegler, Daniel, Wu, Jeffrey, Winter, Clemens, Hesse, Chris, Chen, Mark, Sigler, Eric, Litwin, Mateusz, Gray, Scott, Chess, Ben- jamin, Clark, Jack, Berner, Christopher, McCandlish, Sam, Radford, Alec, Sutskever, Ilya, Amodei, Dario, 2020. Language models are few-shot learners. In: Proceedings of NeurIPS. pp. 1877-1901, URL https://proceedings.neurips.cc/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  13. Carion, Nicolas, Massa, Francisco, Synnaeve, Gabriel, Usunier, Nicolas, Kirillov, Alexan- der, Zagoruyko, Sergey, 2020. End-to-end object detection with transformers. In: Proceedings of ECCV. pp. 213-229. http://dx.doi.org/10.1007/978-3-030-58452- 8_13. · DOI · Paper page
  14. Chen, Ziye, Gong, Mingming, Ge, Lingjuan, Du, Bo, 2020a. Compressed self-attention for deep metric learning with low-rank approximation. In: Proceedings of IJCAI. pp. 2058-2064. http://dx.doi.org/10.24963/ijcai.2020/285. · DOI
  15. Chen, Mark, Radford, Alec, Child, Rewon, Wu, Jeffrey, Jun, Heewoo, Luan, David, Sutskever, Ilya, 2020b. Generative pretraining from pixels. In: Proceedings of ICML. pp. 1691-1703, URL http://proceedings.mlr.press/v119/chen20s.html.
  16. Chen, Xie, Wu, Yu, Wang, Zhenghao, Liu, Shujie, Li, Jinyu, 2021. Developing real- time streaming transformer transducer for speech recognition on large-scale dataset. arXiv:2010.11395, [cs.CL]. · arXiv
  17. Child, Rewon, Gray, Scott, Radford, Alec, Sutskever, Ilya, 2019. Generating long sequences with sparse transformers. arXiv:1904.10509, [cs.LG]. · arXiv
  18. Choromanski, Krzysztof, Likhosherstov, Valerii, Dohan, David, Song, Xingyou, Gane, An- dreea, Sarlos, Tamas, Hawkins, Peter, Davis, Jared, Belanger, David, Colwell, Lucy, Weller, Adrian, 2020a. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv:2006.03555, [cs.LG]. · arXiv
  19. Choromanski, Krzysztof, Likhosherstov, Valerii, Dohan, David, Song, Xingyou, Gane, Andreea, Sarlos, Tamas, Hawkins, Peter, Davis, Jared, Mohiuddin, Afroz, Kaiser, Lukasz, Belanger, David, Colwell, Lucy, Weller, Adrian, 2020b. Rethinking attention with performers. arXiv:2009.14794, [cs.LG]. · arXiv
  20. Chu, Xiangxiang, Tian, Zhi, Zhang, Bo, Wang, Xinlong, Wei, Xiaolin, Xia, Huaxia, Shen, Chunhua, 2021. Conditional positional encodings for vision transformers. arXiv:2102.10882, [cs.CV]. · arXiv
  21. Cordonnier, Jean-Baptiste, Loukas, Andreas, Jaggi, Martin, 2020. Multi-head attention: Collaborate instead of concatenate. CoRR arXiv:2006.16362. · arXiv
  22. Cornia, Marcella, Stefanini, Matteo, Baraldi, Lorenzo, Cucchiara, Rita, 2020. Meshed- memory transformer for image captioning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, pp. 10575-10584. http://dx.doi.org/10.1109/CVPR42600.2020. 01059. · DOI · Paper page
  23. Dai, Zihang, Lai, Guokun, Yang, Yiming, Le, Quoc, 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. In: Proceedings of NeurIPS. URL https://proceedings.neurips.cc/paper/2020/hash/ 2cd2915e69546904e4e5d4a2ac9e1652-Abstract.html.
  24. Dai, Zihang, Yang, Zhilin, Yang, Yiming, Carbonell, Jaime, Le, Quoc, Salakhut- dinov, Ruslan, 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In: Proceedings of ACL. Florence, Italy, pp. 2978-2988. http: //dx.doi.org/10.18653/v1/P19-1285. · DOI · Paper page
  25. Dauphin, Yann N., Fan, Angela, Auli, Michael, Grangier, David, 2017. Language modeling with gated convolutional networks. In: Proceedings of ICML. pp. 933-941, URL http://proceedings.mlr.press/v70/dauphin17a.html.
  26. Dehghani, Mostafa, Gouws, Stephan, Vinyals, Oriol, Uszkoreit, Jakob, Kaiser, Lukasz, 2019. Universal transformers. In: Proceedings of ICLR. URL https://openreview. net/forum?id=HyzdRiR9Y7.
  27. Deshpande, Ameet, Narasimhan, Karthik, 2020. Guiding attention for self-supervised learning with transformers. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 4676-4686. http://dx.doi.org/10.18653/v1/2020. findings-emnlp.419, Online. · DOI
  28. Devlin, Jacob, Chang, Ming-Wei, Lee, Kenton, Toutanova, Kristina, 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of HLT-NAACL. Minneapolis, Minnesota, pp. 4171-4186. http://dx. doi.org/10.18653/v1/N19-1423. · DOI · Paper page
  29. Ding, Siyu, Shang, Junyuan, Wang, Shuohuan, Sun, Yu, Tian, Hao, Wu, Hua, Wang, Haifeng, 2020. ERNIE-DOC: The retrospective long-document modeling transformer. arXiv:2012.15688, [cs.. · arXiv
  30. Ding, Ming, Yang, Zhuoyi, Hong, Wenyi, Zheng, Wendi, Zhou, Chang, Yin, Da, Lin, Junyang, Zou, Xu, Shao, Zhou, Yang, Hongxia, Tang, Jie, 2021. CogView: Mastering text-to-image generation via transformers. arXiv:2105.13290, [cs.CV]. · arXiv
  31. Dong, Yihe, Cordonnier, Jean-Baptiste, Loukas, Andreas, 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. CoRR arXiv: 2103.03404. · arXiv
  32. Dong, Linhao, Xu, Shuang, Xu, Bo, 2018. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In: Proceedings of ICASSP. pp. 5884-5888. http://dx.doi.org/10.1109/ICASSP.2018.8462506. · DOI · Paper page
  33. Dosovitskiy, Alexey, Beyer, Lucas, Kolesnikov, Alexander, Weissenborn, Dirk, Zhai, Xiaohua, Unterthiner, Thomas, Dehghani, Mostafa, Minderer, Matthias, Heigold, Georg, Gelly, Sylvain, Uszkoreit, Jakob, Houlsby, Neil, 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, [cs.CV]. · arXiv
  34. Fan, Zhihao, Gong, Yeyun, Liu, Dayiheng, Wei, Zhongyu, Wang, Siyuan, Jiao, Jian, Duan, Nan, Zhang, Ruofei, Huang, Xuanjing, 2021a. Mask attention networks: Rethinking and strengthen transformer. In: Proceedings of NAACL. pp. 1692-1701, URL https://www.aclweb.org/anthology/2021.naacl-main.135.
  35. Fan, Angela, Lavril, Thibaut, Grave, Edouard, Joulin, Armand, Sukhbaatar, Sainbayar, 2021b. Addressing some limitations of transformers with feedback memory. URL https://openreview.net/forum?id=OCm0rwa1lx1.
  36. Fedus, William, Zoph, Barret, Shazeer, Noam, 2022. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), 1-39, http://jmlr.org/papers/v23/21-0998.html.
  37. Graves, Alex, 2016. Adaptive computation time for recurrent neural networks. CoRR arXiv:1603.08983. · arXiv
  38. Gu, Shuhao, Feng, Yang, 2019. Improving multi-head attention with capsule networks. In: Proceedings of NLPCC. pp. 314-326. http://dx.doi.org/10.1007/978-3-030- 32233-5_25. · DOI
  39. Gu, Jiatao, Liu, Qi, Cho, Kyunghyun, 2019. Insertion-based decoding with automatically inferred generation order. Trans. Assoc. Comput. Linguist. 7, 661-676, URL https: //transacl.org/ojs/index.php/tacl/article/view/1732.
  40. Gulati, Anmol, Qin, James, Chiu, Chung-Cheng, Parmar, Niki, Zhang, Yu, Yu, Jiahui, Han, Wei, Wang, Shibo, Zhang, Zhengdong, Wu, Yonghui, Pang, Ruoming, 2020. Conformer: Convolution-augmented transformer for speech recognition. In: Proceed- ings of Interspeech. pp. 5036-5040. http://dx.doi.org/10.21437/Interspeech.2020- 3015. · DOI · Paper page
  41. Guo, Qipeng, Qiu, Xipeng, Liu, Pengfei, Shao, Yunfan, Xue, Xiangyang, Zhang, Zheng, 2019a. Star-transformer. In: Proceedings of HLT-NAACL. pp. 1315-1325, URL https://www.aclweb.org/anthology/N19-1133.
  42. Guo, Qipeng, Qiu, Xipeng, Liu, Pengfei, Xue, Xiangyang, Zhang, Zheng, 2020. Multi- scale self-attention for text classification. In: Proceedings of AAAI. pp. 7847-7854, URL https://aaai.org/ojs/index.php/AAAI/article/view/6290.
  43. Guo, Qipeng, Qiu, Xipeng, Xue, Xiangyang, Zhang, Zheng, 2019b. Low-rank and locality constrained self-attention for sequence modeling. IEEE/ACM Trans. Audio Speech Lang. Proc. 27 (12), 2213-2222. http://dx.doi.org/10.1109/TASLP.2019.2944078. · DOI
  44. Guo, Maosheng, Zhang, Yu, Liu, Ting, 2019c. Gaussian transformer: A lightweight approach for natural language inference. In: Proceedings of AAAI. pp. 6489-6496. http://dx.doi.org/10.1609/aaai.v33i01.33016489. · DOI
  45. Han, Kai, Wang, Yunhe, Chen, Hanting, Chen, Xinghao, Guo, Jianyuan, Liu, Zhenhua, Tang, Yehui, Xiao, An, Xu, Chunjing, Xu, Yixing, Yang, Zhaohui, Zhang, Yiman, Tao, Dacheng, 2021a. A survey on visual transformer. arXiv:2012.12556, [cs.CV]. · arXiv
  46. Han, Chi, Wang, Mingxuan, Ji, Heng, Li, Lei, 2021b. Learning shared semantic space for speech-to-text translation. arXiv:2105.03095, [cs.]. · arXiv
  47. Han, Kai, Xiao, An, Wu, Enhua, Guo, Jianyuan, Xu, Chunjing, Wang, Yunhe, 2021c. Transformer in transformer. arXiv:2103.00112, [cs.CV]. · arXiv
  48. He, Pengcheng, Liu, Xiaodong, Gao, Jianfeng, Chen, Weizhu, 2020a. Deberta: Decoding-enhanced BERT with disentangled attention. arXiv:2006.03654. · arXiv
  49. He, Ruining, Ravula, Anirudh, Kanagal, Bhargav, Ainslie, Joshua, 2020b. RealFormer: Transformer likes residual attention. arXiv:2012.11747, [cs.LG]. · arXiv
  50. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, Sun, Jian, 2016. Deep residual learning for image recognition. In: Proceedings CVPR. pp. 770-778. http://dx.doi.org/10. 1109/CVPR.2016. · DOI · Paper page
  51. Hendrycks, Dan, Gimpel, Kevin, 2020. Gaussian error linear units (GELUs). arXiv: 1606.08415, [cs.LG]. · DOI · Paper page
  52. Hinton, Geoffrey E., Sabour, Sara, Frosst, Nicholas, 2018. Matrix capsules with EM routing. In: Proceedings of ICLR. URL https://openreview.net/forum?id= HJWLfGWRb.
  53. Ho, Jonathan, Kalchbrenner, Nal, Weissenborn, Dirk, Salimans, Tim, 2019. Axial attention in multidimensional transformers. CoRR arXiv:1912.12180. · arXiv
  54. Hu, Ronghang, Singh, Amanpreet, Darrell, Trevor, Rohrbach, Marcus, 2020. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 9989-9999. http://dx.doi.org/10. 1109/CVPR42600.2020.01001. · DOI
  55. Huang, Cheng-Zhi Anna, Vaswani, Ashish, Uszkoreit, Jakob, Simon, Ian, Hawthorne, Curtis, Shazeer, Noam, Dai, Andrew M., Hoffman, Matthew D., Dinculescu, Monica, Eck, Douglas, 2019. Music transformer. In: Proceedings of ICLR. URL https://openreview.net/forum?id=rJe4ShAcF7.
  56. Ihm, Hyeong Rae, Lee, Joun Yeop, Choi, Byoung Jin, Cheon, Sung Jun, Kim, Nam Soo, 2020. Reformer-TTS: Neural speech synthesis with reformer network. In: Meng, He- len, Xu, Bo, Zheng, Thomas Fang (Eds.), Proceedings of Interspeech. pp. 2012-2016. http://dx.doi.org/10.21437/Interspeech.2020-2189. · DOI
  57. Ioffe, Sergey, Szegedy, Christian, 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of ICML. pp. 448-456, URL http://proceedings.mlr.press/v37/ioffe15.html.
  58. Irie, Kazuki, Zeyer, Albert, Schlüter, Ralf, Ney, Hermann, 2019. Language modeling with deep transformers. In: Proceedings of Interspeech. pp. 3905-3909. http: //dx.doi.org/10.21437/Interspeech.2019-2225. · DOI
  59. Islam, Md. Amirul, Jia, Sen, Bruce, Neil D.B., 2020. How much position information do convolutional neural networks encode? In: Proceedings of ICLR. URL https: //openreview.net/forum?id=rJeB36NKvB.
  60. Jiang, Yifan, Chang, Shiyu, Wang, Zhangyang, 2021. Transgan: Two transformers can make one strong GAN. arXiv:2102.07074, [cs.CV]. · arXiv
  61. Katharopoulos, Angelos, Vyas, Apoorv, Pappas, Nikolaos, Fleuret, François, 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. In: Proceedings of ICML. pp. 5156-5165, URL http://proceedings.mlr.press/v119/ katharopoulos20a.html.
  62. Ke, Guolin, He, Di, Liu, Tie-Yan, 2020. Rethinking positional encoding in language pre-training. arXiv:2006.15595, [cs.. · arXiv
  63. Khan, Salman, Naseer, Muzammal, Hayat, Munawar, Zamir, Syed Waqas, Khan, Fa- had Shahbaz, Shah, Mubarak, 2021. Transformers in vision: A survey. arXiv: 2101.01169, [cs.CV]. · arXiv
  64. Kim, Jaeyoung, El-Khamy, Mostafa, Lee, Jungwon, 2020. T-GSA: transformer with Gaussian-weighted self-attention for speech enhancement. In: 2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020. IEEE, pp. 6649-6653. http://dx.doi.org/10.1109/ ICASSP40776.2020.9053591. · DOI
  65. Kitaev, Nikita, Kaiser, Lukasz, Levskaya, Anselm, 2020. Reformer: The efficient transformer. In: Proceedings of ICLR. URL https://openreview.net/forum?id= rkgNKkHtvB.
  66. Klein, Guillaume, Kim, Yoon, Deng, Yuntian, Senellart, Jean, Rush, Alexander, 2017. OpenNMT: Open-source toolkit for neural machine translation. In: Proceedings of ACL. pp. 67-72, URL https://www.aclweb.org/anthology/P17-4012.
  67. Kovaleva, Olga, Romanov, Alexey, Rogers, Anna, Rumshisky, Anna, 2019. Revealing the dark secrets of BERT. In: Proceedings of EMNLP-IJCNLP. pp. 4364-4373. http://dx.doi.org/10.18653/v1/D19-1445. · DOI
  68. Lample, Guillaume, Sablayrolles, Alexandre, Ranzato, Marc'Aurelio, Denoyer, Ludovic, Jégou, Hervé, 2019. Large memory layers with product keys. In: Proceedings of NeurIPS. pp. 8546-8557, URL https://proceedings.neurips.cc/paper/2019/hash/ 9d8df73a3cfbf3c5b47bc9b50f214aff-Abstract.html.
  69. Lee, Juho, Lee, Yoonho, Kim, Jungtaek, Kosiorek, Adam R., Choi, Seungjin, Teh, Yee Whye, 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In: Proceedings of ICML. pp. 3744-3753, URL http://proceedings.mlr.press/v97/lee19d.html.
  70. Lepikhin, Dmitry, Lee, HyoukJoong, Xu, Yuanzhong, Chen, Dehao, Firat, Orhan, Huang, Yanping, Krikun, Maxim, Shazeer, Noam, Chen, Zhifeng, 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. CoRR arXiv:2006.16668. · arXiv
  71. Lewis, Mike, Liu, Yinhan, Goyal, Naman, Ghazvininejad, Marjan, Mohamed, Abdelrah- man, Levy, Omer, Stoyanov, Veselin, Zettlemoyer, Luke, 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of ACL. pp. 7871-7880. http://dx.doi.org/10. 18653/v1/2020.acl-main.703. · DOI · Paper page
  72. Li, Wei, Gao, Can, Niu, Guocheng, Xiao, Xinyan, Liu, Hao, Liu, Jiachen, Wu, Hua, Wang, Haifeng, 2020a. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv preprint arXiv:2012.15409. · arXiv
  73. Li, Naihan, Liu, Shujie, Liu, Yanqing, Zhao, Sheng, Liu, Ming, 2019a. Neural speech synthesis with transformer network. In: Proceedings of AAAI. pp. 6706-6713. http://dx.doi.org/10.1609/aaai.v33i01.33016706. · DOI · Paper page
  74. Li, Xiaoya, Meng, Yuxian, Zhou, Mingxin, Han, Qinghong, Wu, Fei, Li, Jiwei, 2020b. SAC: accelerating and structuring self-attention via sparse adaptive connection. In: Proceedings of NeurIPS. URL https://proceedings.neurips.cc/paper/2020/hash/ c5c1bda1194f9423d744e0ef67df94ee-Abstract.html.
  75. Li, Xiaonan, Shao, Yunfan, Sun, Tianxiang, Yan, Hang, Qiu, Xipeng, Huang, Xuanjing, 2021. Accelerating BERT inference for sequence labeling via early-exit. arXiv: 2105.13878, [cs.. · arXiv
  76. Li, Jian, Tu, Zhaopeng, Yang, Baosong, Lyu, Michael R., Zhang, Tong, 2018. Multi-head attention with disagreement regularization. In: Proceedings of EMNLP. Brussels, Belgium, pp. 2897-2903. http://dx.doi.org/10.18653/v1/D18-1317. · DOI
  77. Li, Xiaonan, Yan, Hang, Qiu, Xipeng, Huang, Xuanjing, 2020c. FLAT: Chinese NER using flat-lattice transformer. In: Proceedings of ACL. pp. 6836-6842. http://dx. doi.org/10.18653/v1/2020.acl-main.611. · DOI
  78. Li, Jian, Yang, Baosong, Dou, Zi-Yi, Wang, Xing, Lyu, Michael R., Tu, Zhaopeng, 2019b. Information aggregation for multi-head attention with routing-by-agreement. In: Proceedings of HLT-NAACL. pp. 3566-3575. http://dx.doi.org/10.18653/v1/N19- 1359. · DOI
  79. Li, Liunian Harold, Yatskar, Mark, Yin, Da, Hsieh, Cho-Jui, Chang, Kai-Wei, 2019c. Visualbert: A simple and performant baseline for vision and language. arXiv: 1908.03557, [cs.CV]. · arXiv
  80. Lin, Junyang, Men, Rui, Yang, An, Zhou, Chang, Ding, Ming, Zhang, Yichang, Wang, Peng, Wang, Ang, Jiang, Le, Jia, Xianyan, Zhang, Jie, Zhang, Jianwei, Zou, Xu, Li, Zhikang, Deng, Xiaodong, Liu, Jie, Xue, Jinbao, Zhou, Huiling, Ma, Jianxin, Yu, Jin, Li, Yong, Lin, Wei, Zhou, Jingren, Tang, Jie, Yang, Hongxia, 2021. M6: A Chinese multimodal pretrainer. arXiv:2103.00823, [cs.. · arXiv
  81. Liu, Yang, Lapata, Mirella, 2019. Hierarchical transformers for multi-document sum- marization. In: Proceedings of ACL. Florence, Italy, pp. 5070-5081. http://dx.doi. org/10.18653/v1/P19-1500. · DOI
  82. Liu, Ze, Lin, Yutong, Cao, Yue, Hu, Han, Wei, Yixuan, Zhang, Zheng, Lin, Stephen, Guo, Baining, 2021. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030, [cs.CV]. · arXiv
  83. Liu, Liyuan, Liu, Xiaodong, Gao, Jianfeng, Chen, Weizhu, Han, Jiawei, 2020a. Un- derstanding the difficulty of training transformers. In: Proceedings of EMNLP. pp. 5747-5763. http://dx.doi.org/10.18653/v1/2020.emnlp-main.463. · DOI
  84. Liu, Yinhan, Ott, Myle, Goyal, Naman, Du, Jingfei, Joshi, Mandar, Chen, Danqi, Levy, Omer, Lewis, Mike, Zettlemoyer, Luke, Stoyanov, Veselin, 2019a. Roberta: A robustly optimized BERT pretraining approach. arXiv:1907.11692, [cs.. · arXiv
  85. Liu, Peter J., Saleh, Mohammad, Pot, Etienne, Goodrich, Ben, Sepassi, Ryan, Kaiser, Lukasz, Shazeer, Noam, 2018. Generating wikipedia by summarizing long sequences. In: Proceedings of ICLR. URL https://openreview.net/forum?id= Hyg0vbWC-.
  86. Liu, Hanxiao, Simonyan, Karen, Yang, Yiming, 2019b. DARTS: Differentiable archi- tecture search. In: Proceedings of ICLR. URL https://openreview.net/forum?id= S1eYHoC5FX.
  87. Liu, Xuanqing, Yu, Hsiang-Fu, Dhillon, Inderjit S., Hsieh, Cho-Jui, 2020b. Learning to encode position for transformer with continuous dynamical model. In: Proceedings of ICML. pp. 6327-6335, URL http://proceedings.mlr.press/v119/liu20n.html.
  88. Lu, Yiping, Li, Zhuohan, He, Di, Sun, Zhiqing, Dong, Bin, Qin, Tao, Wang, Liwei, Liu, Tie-Yan, 2020. Understanding and improving transformer from a multi-particle dynamic system point of view. URL https://openreview.net/forum?id=SJl1o2NFwS.
  89. Ma, Xuezhe, Kong, Xiang, Wang, Sinong, Zhou, Chunting, May, Jonathan, Ma, Hao, Zettlemoyer, Luke, 2021. Luna: Linear unified nested attention. arXiv:2106.01540, [cs.LG]. · arXiv
  90. Mehta, Sachin, Ghazvininejad, Marjan, Iyer, Srinivasan, Zettlemoyer, Luke, Ha- jishirzi, Hannaneh, 2020. Delight: Very deep and light-weight transformer. arXiv: 2008.00623, [cs.LG]. · arXiv
  91. Miculicich, Lesly, Ram, Dhananjay, Pappas, Nikolaos, Henderson, James, 2018. Document-level neural machine translation with hierarchical attention networks. In: Proceedings of EMNLP. Brussels, Belgium, pp. 2947-2954. http://dx.doi.org/ 10.18653/v1/D18-1325. · DOI
  92. Nguyen, Toan Q., Salazar, Julian, 2019. Transformers without tears: Improving the normalization of self-attention. CoRR arXiv:1910.05895. · arXiv
  93. Parmar, Niki, Vaswani, Ashish, Uszkoreit, Jakob, Kaiser, Lukasz, Shazeer, Noam, Ku, Alexander, Tran, Dustin, 2018. Image transformer. In: Proceedings of ICML. pp. 4052-4061, URL http://proceedings.mlr.press/v80/parmar18a.html.
  94. Peng, Hao, Pappas, Nikolaos, Yogatama, Dani, Schwartz, Roy, Smith, Noah, Kong, Ling- peng, 2021. Random feature attention. In: Proceedings of ICLR. URL https:// openreview.net/forum?id=QtTKTdVrFBB.
  95. Perez, Ethan, Strub, Florian, de Vries, Harm, Dumoulin, Vincent, Courville, Aaron C., 2018. Film: Visual reasoning with a general conditioning layer. In: Proceedings of AAAI. pp. 3942-3951, URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/ paper/view/16528.
  96. Pham, Ngoc-Quan, Nguyen, Thai-Son, Niehues, Jan, Müller, Markus, Waibel, Alex, 2019. Very deep self-attention networks for end-to-end speech recognition. In: Proceedings of Interspeech. pp. 66-70. http://dx.doi.org/10.21437/Interspeech. 2019-2702. · DOI
  97. Pilault, Jonathan, hattami, Amine El, Pal, Christopher, 2021. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. In: Proceedings of ICLR. URL https://openreview.net/forum?id= de11dbHzAMF.
  98. Press, Ofir, Smith, Noah A., Levy, Omer, 2020. Improving transformer models by reordering their sublayers. In: Proceedings of ACL. pp. 2996-3005. http://dx.doi. org/10.18653/v1/2020.acl-main.270, Online. · DOI · Paper page
  99. Qiu, Xipeng, Sun, TianXiang, Xu, Yige, Shao, Yunfan, Dai, Ning, Huang, Xuanjing, 2020. Pre-trained models for natural language processing: A survey. Sci. China Technol. Sci. 63 (10), 1872-1897. http://dx.doi.org/10.1007/s11431-020-1647-3. · DOI · Paper page
  100. Rae, Jack W., Potapenko, Anna, Jayakumar, Siddhant M., Hillier, Chloe, Lillicrap, Tim- othy P., 2020. Compressive transformers for long-range sequence modelling. In: Proceedings of ICLR. URL https://openreview.net/forum?id=SylKikSYDH.
  101. Raffel, Colin, Shazeer, Noam, Roberts, Adam, Lee, Katherine, Narang, Sharan, Matena, Michael, Zhou, Yanqi, Li, Wei, Liu, Peter J., 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683, [cs.LG]. · arXiv
  102. Rahimi, Ali, Recht, Benjamin, 2007. Random features for large-scale kernel machines. In: Proceedings of NeurIPS. pp. 1177-1184, URL https://proceedings.neurips.cc/ paper/2007/hash/013a006f03dbc5392effeb8f18fda755-Abstract.html.
  103. Ramachandran, Prajit, Zoph, Barret, Le, Quoc V., 2018. Searching for activa- tion functions. In: Proceedings of ICLR. URL https://openreview.net/forum?id= Hkuq2EkPf.
  104. Ramesh, Aditya, Pavlov, Mikhail, Goh, Gabriel, Gray, Scott, Voss, Chelsea, Rad- ford, Alec, Chen, Mark, Sutskever, Ilya, 2021. Zero-shot text-to-image generation. arXiv:2102.12092, [cs.CV]. · arXiv
  105. Rebuffi, Sylvestre-Alvise, Bilen, Hakan, Vedaldi, Andrea, 2017. Learning multiple visual domains with residual adapters. In: Proceedings of NeurIPS. pp. 506-516, URL https://proceedings.neurips.cc/paper/2017/hash/ e7b24b112a44fdd9ee93bdf998c6ca0e-Abstract.html.
  106. Rives, Alexander, Meier, Joshua, Sercu, Tom, Goyal, Siddharth, Lin, Zeming, Liu, Jason, Guo, Demi, Ott, Myle, Zitnick, C. Lawrence, Ma, Jerry, Fergus, Rob, 2021. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118 (15), http://dx.doi.org/ 10.1073/pnas.2016239118. · DOI · Paper page
  107. Roller, Stephen, Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, 2021. Hash layers for large sparse models. arXiv:2106.04426, [cs.LG]. · arXiv
  108. Roy, Aurko, Saffar, Mohammad, Vaswani, Ashish, Grangier, David, 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics 9, 53-68.
  109. Sabour, Sara, Frosst, Nicholas, Hinton, Geoffrey E., 2017. Dynamic routing between capsules. In: Proceedings of NeurIPS. pp. 3856-3866, URL https://proceedings. neurips.cc/paper/2017/hash/2cad8fa47bbef282badbb8de5374b894-Abstract.html.
  110. Schlag, Imanol, Irie, Kazuki, Schmidhuber, Jürgen, 2021. Linear transformers are secretly fast weight memory systems. CoRR arXiv:2102.11174. · arXiv
  111. Schwaller, Philippe, Laino, Teodoro, Gaudin, Théophile, Bolgar, Peter, Hunter, Christo- pher A., Bekas, Costas, Lee, Alpha A., 2019. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Central Sci. 5 (9), 1572-1583. http://dx.doi.org/10.1021/acscentsci.9b00576. · DOI
  112. Shao, Jie, Wen, Xin, Zhao, Bingchen, Xue, Xiangyang, 2021. Temporal context aggre- gation for video retrieval with contrastive learning. In: Proceedings of WACV. pp. 3268-3278.
  113. Shaw, Peter, Uszkoreit, Jakob, Vaswani, Ashish, 2018. Self-attention with relative position representations. In: Proceedings of HLT-NAACL. New Orleans, Louisiana, pp. 464-468. http://dx.doi.org/10.18653/v1/N18-2074. · DOI · Paper page
  114. Shazeer, Noam, 2019. Fast transformer decoding: One write-head is all you need. CoRR arXiv:1911.02150. · arXiv
  115. Shazeer, Noam, 2020. GLU variants improve transformer. arXiv:2002.05202, [cs.LG]. · arXiv
  116. Shazeer, Noam, Lan, Zhenzhong, Cheng, Youlong, Ding, Nan, Hou, Le, 2020. Talking-heads attention. CoRR arXiv:2003.02436. · arXiv
  117. Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc V., Hinton, Geoffrey E., Dean, Jeff, 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In: Proceedings of ICLR. URL https:// openreview.net/forum?id=B1ckMDqlg.
  118. Shen, Sheng, Yao, Zhewei, Gholami, Amir, Mahoney, Michael W., Keutzer, Kurt, 2020. PowerNorm: Rethinking batch normalization in transformers. In: Proceedings of ICML. pp. 8741-8751, URL http://proceedings.mlr.press/v119/shen20e.html.
  119. Shoeybi, Mohammad, Patwary, Mostofa, Puri, Raul, LeGresley, Patrick, Casper, Jared, Catanzaro, Bryan, 2020. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053, [cs.]. · arXiv
  120. So, David R., Le, Quoc V., Liang, Chen, 2019. The evolved transformer. In: Proceedings of ICML. pp. 5877-5886, URL http://proceedings.mlr.press/v97/so19a.html.
  121. Su, Jianlin, Lu, Yu, Pan, Shengfeng, Wen, Bo, Liu, Yunfeng, 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864. · arXiv
  122. Su, Weijie, Zhu, Xizhou, Cao, Yue, Li, Bin, Lu, Lewei, Wei, Furu, Dai, Jifeng, 2020. VL-BERT: pre-training of generic visual-linguistic representations. In: Proceedings of ICLR. URL https://openreview.net/forum?id=SygXPaEYvH.
  123. Sukhbaatar, Sainbayar, Grave, Edouard, Bojanowski, Piotr, Joulin, Armand, 2019a. Adaptive attention span in transformers. In: Proceedings of ACL. Florence, Italy, pp. 331-335. http://dx.doi.org/10.18653/v1/P19-1032. · DOI
  124. Sukhbaatar, Sainbayar, Grave, Edouard, Lample, Guillaume, Jegou, Herve, Joulin, Ar- mand, 2019b. Augmenting self-attention with persistent memory. arXiv:1907. 01470, [cs.LG].
  125. Sun, Chen, Myers, Austin, Vondrick, Carl, Murphy, Kevin, Schmid, Cordelia, 2019. Videobert: A joint model for video and language representation learning. In: Proceedings of ICCV. pp. 7463-7472. http://dx.doi.org/10.1109/ICCV.2019.00756. · DOI
  126. Sun, Tianxiang, Zhou, Yunhua, Liu, Xiangyang, Zhang, Xinyu, Jiang, Hao, Cao, Zhao, Huang, Xuanjing, Qiu, Xipeng, 2021. Early exiting with ensemble internal classifiers. arXiv:2105.13792, [cs.]. · arXiv
  127. Sutskever, Ilya, Vinyals, Oriol, Le, Quoc V., 2014. Sequence to sequence learning with neural networks. In: Proceedings of NeurIPS. pp. 3104-3112, URL https: //proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2- Abstract.html.
  128. Tay, Yi, Bahri, Dara, Metzler, Donald, Juan, Da-Cheng, Zhao, Zhe, Zheng, Che, 2020a. Synthesizer: Rethinking self-attention in transformer models. CoRR arXiv:2005. 00743.
  129. Tay, Yi, Bahri, Dara, Yang, Liu, Metzler, Donald, Juan, Da-Cheng, 2020b. Sparse sinkhorn attention. In: Proceedings of ICML. pp. 9438-9447, URL http:// proceedings.mlr.press/v119/tay20a.html.
  130. Tsai, Yao-Hung Hubert, Bai, Shaojie, Yamada, Makoto, Morency, Louis-Philippe, Salakhutdinov, Ruslan, 2019. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In: Proceedings of EMNLP-IJCNLP.
  131. Hong Kong, China, pp. 4344-4353. http://dx.doi.org/10.18653/v1/D19-1443. · DOI · Paper page
  132. van den Oord, Aäron, Dieleman, Sander, Zen, Heiga, Simonyan, Karen, Vinyals, Oriol, Graves, Alex, Kalchbrenner, Nal, Senior, Andrew W., Kavukcuoglu, Koray, 2016. WaveNet: A generative model for raw audio. In: Proceedings of ISCA. p. 125, URL http://www.isca-speech.org/archive/SSW_2016/abstracts/ssw9_DS-4_van_den_ Oord.html.
  133. van den Oord, Aäron, Li, Yazhe, Vinyals, Oriol, 2018. Representation learning with contrastive predictive coding. CoRR arXiv:1807.03748. · arXiv
  134. Vaswani, Ashish, Bengio, Samy, Brevdo, Eugene, Chollet, Francois, Gomez, Aidan, Gouws, Stephan, Jones, Llion, Kaiser, Łukasz, Kalchbrenner, Nal, Parmar, Niki, Sepassi, Ryan, Shazeer, Noam, Uszkoreit, Jakob, 2018. Tensor2Tensor for neural machine translation. In: Proceedings of AMTA. pp. 193-199, URL https://www. aclweb.org/anthology/W18-1819.
  135. Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, Polosukhin, Illia, 2017. Attention is all you need. In: Proceedings of NeurIPS. pp. 5998-6008, URL https://proceedings.neurips.cc/ paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  136. Vyas, Apoorv, Katharopoulos, Angelos, Fleuret, François, 2020. Fast transformers with clustered attention. arXiv:2007.04825, [cs.LG]. · arXiv
  137. Wang, Sinong, Li, Belinda Z., Khabsa, Madian, Fang, Han, Ma, Hao, 2020a. Linformer: Self-attention with linear complexity. arXiv:2006.04768, [cs.LG]. · arXiv
  138. Wang, Qiang, Li, Bei, Xiao, Tong, Zhu, Jingbo, Li, Changliang, Wong, Derek F., Chao, Lidia S., 2019a. Learning deep transformer models for machine translation. In: Proceedings of ACL. pp. 1810-1822. http://dx.doi.org/10.18653/v1/p19-1176. · DOI · Paper page
  139. Wang, Zhiwei, Ma, Yao, Liu, Zitao, Tang, Jiliang, 2019b. R-transformer: Recurrent neural network enhanced transformer. CoRR arXiv:1907.05572. · arXiv
  140. Wang, Benyou, Shang, Lifeng, Lioma, Christina, Jiang, Xin, Yang, Hao, Liu, Qun, Simonsen, Jakob Grue, 2021a. On position embeddings in BERT. In: Proceedings of ICLR. URL https://openreview.net/forum?id=onxoVA9FxMw.
  141. Wang, Yujing, Yang, Yaming, Bai, Jiangang, Zhang, Mingliang, Bai, Jing, Yu, Jing, Zhang, Ce, Tong, Yunhai, 2021b. Predictive attention transformer: Improving transformer with attention map prediction. URL https://openreview.net/forum?id= YQVjbJPnPc9.
  142. Wang, Benyou, Zhao, Donghao, Lioma, Christina, Li, Qiuchi, Zhang, Peng, Simon- sen, Jakob Grue, 2020b. Encoding word order in complex embeddings. In: Proceedings of ICLR. URL https://openreview.net/forum?id=Hke-WTVtwr.
  143. Wu, Felix, Fan, Angela, Baevski, Alexei, Dauphin, Yann N., Auli, Michael, 2019. Pay less attention with lightweight and dynamic convolutions. In: Proceedings of ICLR. URL https://openreview.net/forum?id=SkVhlh09tX.
  144. Wu, Qingyang, Lan, Zhenzhong, Gu, Jing, Yu, Zhou, 2020a. Memformer: The memory-augmented transformer. arXiv:2010.06891, [cs.. · arXiv
  145. Wu, Zhanghao, Liu, Zhijian, Lin, Ji, Lin, Yujun, Han, Song, 2020b. Lite transformer with long-short range attention. In: Proceedings of ICLR. URL https://openreview. net/forum?id=ByeMPlHKPH.
  146. Wu, Zonghan, Pan, Shirui, Chen, Fengwen, Long, Guodong, Zhang, Chengqi, Yu, Philip S., 2021a. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 32 (1), 4-24. http://dx.doi.org/10.1109/TNNLS. 2020.2978386. · DOI · Paper page
  147. Wu, Chuhan, Wu, Fangzhao, Qi, Tao, Huang, Yongfeng, 2021b. Hi-transformer: Hierar- chical interactive transformer for efficient and effective long document modeling. arXiv:2106.01040, [cs.]. · arXiv
  148. Xin, Ji, Tang, Raphael, Lee, Jaejun, Yu, Yaoliang, Lin, Jimmy, 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In: Proceedings of ACL. pp. 2246-2251. http://dx.doi.org/10.18653/v1/2020.acl-main.204. · DOI
  149. Xiong, Ruibin, Yang, Yunchang, He, Di, Zheng, Kai, Zheng, Shuxin, Xing, Chen, Zhang, Huishuai, Lan, Yanyan, Wang, Liwei, Liu, Tie-Yan, 2020. On layer normal- ization in the transformer architecture. In: Proceedings of ICML. pp. 10524-10533, URL http://proceedings.mlr.press/v119/xiong20b.html.
  150. Xiong, Yunyang, Zeng, Zhanpeng, Chakraborty, Rudrasis, Tan, Mingxing, Fung, Glenn, Li, Yin, Singh, Vikas, 2021. Nyströmformer: A nyström-based algorithm for approximating self-attention. In: Proceedings of AAAI.
  151. Xu, Jingjing, Sun, Xu, Zhang, Zhiyuan, Zhao, Guangxiang, Lin, Junyang, 2019. Understanding and improving layer normalization. In: Proceedings of NeurIPS. pp. 4383-4393, URL https://proceedings.neurips.cc/paper/2019/hash/ 2f4fe03d77724a7217006e5d16728874-Abstract.html.
  152. Yan, Hang, Deng, Bocao, Li, Xiaonan, Qiu, Xipeng, 2019. TENER: Adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474. · arXiv
  153. Yang, An, Lin, Junyang, Men, Rui, Zhou, Chang, Jiang, Le, Jia, Xianyan, Wang, Ang, Zhang, Jie, Wang, Jiamang, Li, Yong, Zhang, Di, Lin, Wei, Qu, Lin, Zhou, Jingren, Yang, Hongxia, 2021. Exploring sparse expert models and beyond. arXiv:2105. 15082, [cs.LG].
  154. Yang, Baosong, Tu, Zhaopeng, Wong, Derek F., Meng, Fandong, Chao, Lidia S., Zhang, Tong, 2018. Modeling localness for self-attention networks. In: Proceedings of EMNLP. Brussels, Belgium, pp. 4449-4458. http://dx.doi.org/10.18653/v1/D18- 1475. · DOI
  155. Yang, Yilin, Wang, Longyue, Shi, Shuming, Tadepalli, Prasad, Lee, Stefan, Tu, Zhaopeng, 2020. On the sub-layer functionalities of transformer decoder. In: Findings of EMNLP. pp. 4799-4811. http://dx.doi.org/10.18653/v1/2020.findings-emnlp.432, Online. · DOI
  156. Ye, Zihao, Guo, Qipeng, Gan, Quan, Qiu, Xipeng, Zhang, Zheng, 2019. BP-transformer: Modelling long-range context via binary partitioning. arXiv:1911.04070, [cs.. · arXiv
  157. Ying, Chengxuan, Cai, Tianle, Luo, Shengjie, Zheng, Shuxin, Ke, Guolin, He, Di, Shen, Yanming, Liu, Tie-Yan, 2021a. Do transformers really perform bad for graph representation?. arXiv:2106.05234, [cs.LG]. · arXiv
  158. Ying, Chengxuan, Ke, Guolin, He, Di, Liu, Tie-Yan, 2021b. LazyFormer: Self attention with lazy update. CoRR arXiv:2102.12702. · arXiv
  159. Yoshida, Davis, Ettinger, Allyson, Gimpel, Kevin, 2020. Adding recurrence to pretrained transformers for improved efficiency and context size. CoRR arXiv:2008.07027. · arXiv
  160. You, Weiqiu, Sun, Simeng, Iyyer, Mohit, 2020. Hard-coded Gaussian attention for neural machine translation. In: Proceedings of ACL. pp. 7689-7700. http://dx.doi.org/10. 18653/v1/2020.acl-main.687, Online. · DOI
  161. Yu, Weiwei, Zhou, Jian, Wang, HuaBin, Tao, Liang, 2021. Setransformer: Speech enhancement transformer. Cognit. Comput. http://dx.doi.org/10.1007/s12559-020- 09817-2. · DOI
  162. Zaheer, Manzil, Guruganesh, Guru, Dubey, Avinava, Ainslie, Joshua, Alberti, Chris, Ontanon, Santiago, Pham, Philip, Ravula, Anirudh, Wang, Qifan, Yang, Li, Ahmed, Amr, 2020. Big bird: Transformers for longer sequences. arXiv:2007.14062, [cs.LG]. · arXiv
  163. Zhang, Hang, Gong, Yeyun, Shen, Yelong, Li, Weisheng, Lv, Jiancheng, Duan, Nan, Chen, Weizhu, 2021. Poolingformer: Long document modeling with pooling attention. arXiv:2105.04371. · arXiv
  164. Zhang, Xingxing, Wei, Furu, Zhou, Ming, 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In: Proceed- ings of ACL. Florence, Italy, pp. 5059-5069. http://dx.doi.org/10.18653/v1/P19- 1499. · DOI
  165. Zhang, Biao, Xiong, Deyi, Su, Jinsong, 2018. Accelerating neural transformer via an average attention network. In: Proceedings of ACL. Melbourne, Australia, pp. 1789-1798. http://dx.doi.org/10.18653/v1/P18-1166. · DOI
  166. Zhao, Yuekai, Dong, Li, Shen, Yelong, Zhang, Zhihua, Wei, Furu, Chen, Weizhu, 2021. Memory-efficient differentiable transformer architecture search. arXiv:2105.14669, [cs.LG]. · arXiv
  167. Zheng, Minghang, Gao, Peng, Wang, Xiaogang, Li, Hongsheng, Dong, Hao, 2020a. End-to-end object detection with adaptive clustering transformer. CoRR arXiv: 2011.09315. · arXiv
  168. Zheng, Yibin, Li, Xinhui, Xie, Fenglong, Lu, Li, 2020b. Improving end-to-end speech syn- thesis with local recurrent neural network enhanced transformer. In: Proceedings of ICASSP. pp. 6734-6738. http://dx.doi.org/10.1109/ICASSP40776.2020.9054148. · DOI
  169. Zhou, Wangchunshu, Xu, Canwen, Ge, Tao, McAuley, Julian, Xu, Ke, Wei, Furu, 2020. BERT loses patience: Fast and robust inference with early exit. arXiv:2006.04152. · arXiv
  170. Zhou, Haoyi, Zhang, Shanghang, Peng, Jieqi, Zhang, Shuai, Li, Jianxin, Xiong, Hui, Zhang, Wancai, 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of AAAI.
  171. Zhu, Xizhou, Su, Weijie, Lu, Lewei, Li, Bin, Wang, Xiaogang, Dai, Jifeng, 2020. Deformable DETR: deformable transformers for end-to-end object detection. CoRR arXiv:2010.04159. · arXiv

Cited by

Sorted by citations

Authors
Lin, T., Wang, Y., Liu, X., & Qiu, X.
Affiliations
Fudan University (CN)
Corresponding author
Qiu, Xipeng
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
3
Pages
111-132
ISSN
2666-6510 (ISSN-L)
DOI
10.48550/arxiv.2106.04554
arXiv
2106.04554
OpenAlex
W3171649327
Cited by
1,420
References
1
Citations by year
2026: 218 · 2025: 423 · 2024: 438 · 2023: 261 · 2022: 69 · 2021: 10 · 2020: 1
FWCI
98.02
Type
journal article
Language
it
Open access
Licence
Locations
  • AI Open (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (62022027)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review