A survey of transformers

Deep Residual Learning for Image Recognition

He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608 · see citing works

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works

Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works

End-to-End Object Detection with Transformers

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works

Sequence to Sequence Learning with Neural Networks

Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

Layer NormalizationPreprint

Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works

A Comprehensive Survey on Graph Neural Networks

Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17325 · Cited by 7,752 · see citing works

Representation Learning with Contrastive Predictive Coding

Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works

WaveNet: a generative model for raw audio

Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works

Untitled work

Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works

Dynamic Routing Between Capsules

Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167 · see citing works

Transformers in Vision: A Survey

Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · see citing works

Gaussian Error Linear Units (GELUs)

Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works

DARTS: Differentiable architecture search

Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248 · see citing works

A Comprehensive Survey on Graph Neural Networks

Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 3,174 · see citing works

Transformer-XL: Attentive language models beyond a fixed-length context

Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works

Conformer: Convolution-augmented Transformer for Speech Recognition

Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works

Language Modeling with Gated Convolutional NetworksPreprint

Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1612.08083 · Cited by 3,097 · see citing works

Convolutional Sequence to Sequence Learning

Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · see citing works

Random Features for Large-Scale Kernel Machines

Rahimi, A., & Recht, B. (2007) · Cited by 2,783 · see citing works

Self-Attention with Relative Position Representations

Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · see citing works

Film: Visual reasoning with a general conditioning layer

Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490 · see citing works

Zero-shot text-to-image generation

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works

Searching for activation functions

Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335 · see citing works

Longformer: The Long-Document Transformer

Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works

RoFormer: Enhanced transformer with rotary position embedding

Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957 · see citing works

RoFormer: Enhanced Transformer with Rotary Position Embedding

Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952 · see citing works

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · cited 1× · see citing works

OpenNMT: Open-Source Toolkit for Neural Machine Translation

Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,771 · see citing works

On Layer Normalization in the Transformer ArchitecturePreprint

Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H.-S., Lan, Y.-Y., Wang, L.-W., & Liu, T.-Y. (2020) · International Conference on Machine Learning · doi:10.48550/arxiv.2002.04745 · Cited by 1,709 · see citing works

Pre-trained models for natural language processing: A survey

Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works

Reformer: The Efficient Transformer

Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works

Transformer in Transformer

Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · see citing works

Big Bird: Transformers for Longer Sequences

Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works

Meshed-Memory Transformer for Image Captioning

Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259 · see citing works

DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint

He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,224 · see citing works

VideoBERT: A Joint Model for Video and Language Representation Learning

Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · see citing works

VisualBERT: A Simple and Performant Baseline for Vision and Language

Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works

Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition

Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100 · see citing works

Image transformer

Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085 · see citing works

Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction

Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., & Lee, A. A. (2019) · ACS Central Science · doi:10.1021/acscentsci.9b00576 · Cited by 1,078 · see citing works

Generating Long Sequences with Sparse Transformers

Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works

Untitled work

(2020) · International Conference on Machine Learning · Cited by 1,012 · see citing works

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933 · see citing works

Matrix capsules with EM routing

Hinton, G. E., Sabour, S., & Frosst, N. (2018) · International Conference on Learning Representations · Cited by 918 · see citing works

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works

Linformer: Self-Attention with Linear Complexity

Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827 · see citing works

Untitled work

(2017) · International Conference on Machine Learning · Cited by 758 · see citing works

Neural Speech Synthesis with Transformer Network

Li, N., Liu, S., Liu, Y., Zhao, S., & Liu, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016706 · Cited by 754 · see citing works

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear UnitsPreprint

Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · Cited by 753 · see citing works

Rethinking Attention with Performers

Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 9× · see citing works

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., & Teh, Y. W. (2018) · arXiv · doi:10.48550/arxiv.1810.00825 · Cited by 691 · see citing works

Learning Deep Transformer Models for Machine Translation

Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673 · see citing works

Pay Less Attention with Lightweight and Dynamic ConvolutionsPreprint

Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1901.10430 · Cited by 669 · see citing works

Music Transformer: Generating Music with Long-Term StructurePreprint

Huang, C., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M., Dinculescu, M., & Eck, D. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1809.04281 · Cited by 635 · see citing works

Learning multiple visual domains with residual adapters

Rebuffi, S.-A., Bilen, H., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1705.08045 · Cited by 605 · see citing works

Generating Wikipedia by Summarizing Long SequencesPreprint

Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 578 · see citing works

Revealing the Dark Secrets of BERT

Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1445 · Cited by 537 · see citing works

Efficient Content-Based Sparse Attention with Routing Transformers

Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509 · see citing works

Conditional Positional Encodings for Vision Transformers

Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 4× · see citing works

Universal Transformers

Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv · doi:10.48550/arxiv.1807.03819 · Cited by 467 · see citing works

FLAT: Chinese NER Using Flat-Lattice Transformer

Li, X., Yan, H. R., Qiu, X., & Huang, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.611 · Cited by 463 · see citing works

CogView: Mastering Text-to-Image Generation via Transformers

Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450 · see citing works

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412 · see citing works

HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Zhang, X., Wei, F., & Zhou, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1499 · Cited by 390 · see citing works

Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention

Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376 · see citing works

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale UpPreprint

Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07074 · Cited by 373 · see citing works

Axial Attention in Multidimensional Transformers

Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370 · see citing works

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

Xin, J., Tang, R., Lee, J., Yu, Y., & Lin, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.204 · Cited by 353 · see citing works

Adaptive Computation Time for Recurrent Neural Networks

Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335 · see citing works

Hierarchical Transformers for Multi-Document Summarization

Liu, Y., & Lapata, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1500 · Cited by 332 · see citing works

TENER: Adapting Transformer Encoder for Named Entity RecognitionPreprint

Yan, H. R., Deng, B., Li, X., & Qiu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04474 · Cited by 317 · see citing works

GLU Variants Improve Transformer

Shazeer, N. (2020) · arXiv · doi:10.48550/arxiv.2002.05202 · Cited by 303 · see citing works

ETC: Encoding Long and Structured Inputs in Transformers

Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S. K., Wang, Q., & Li, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.19 · Cited by 293 · see citing works

Adaptive Attention Span in Transformers

Sukhbaatar, S., Grave, É., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1032 · Cited by 286 · see citing works

Lite Transformer with Long-Short Range AttentionPreprint

Wu, Z., Liu, Z., Lin, J., Lin, Y., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11886 · Cited by 279 · see citing works

The evolved transformer

So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · see citing works

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276 · see citing works

Document-Level Neural Machine Translation with Hierarchical Attention Networks

Miculicich, L., Ram, D., Pappas, N., & Henderson, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1325 · Cited by 258 · see citing works

Adaptive Input Representations for Neural Language ModelingPreprint

Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255 · see citing works

Understanding and Improving Layer NormalizationPreprint

Xu, J., Sun, X., Zhang, Z., Zhao, G., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07013 · Cited by 253 · see citing works

How Much Position Information Do Convolutional Neural Networks Encode?

Islam, M. A., Jia, S., & Bruce, N. D. B. (2020) · arXiv · doi:10.48550/arxiv.2001.08248 · Cited by 253 · see citing works

Synthesizer: Rethinking Self-Attention in Transformer Models

Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251 · see citing works

Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA

Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01001 · Cited by 229 · see citing works

Rethinking Positional Encoding in Language Pre-trainingPreprint

Ke, G., He, D., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15595 · Cited by 223 · see citing works

Do Transformers Really Perform Bad for Graph Representation?Preprint

Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05234 · Cited by 221 · see citing works

A Survey on Visual TransformerPreprint

Han, K., Wang, Y., Chen, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12556 · Cited by 214 · see citing works

BERT Loses Patience: Fast and Robust Inference with Early ExitPreprint

Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., & Wei, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04152 · Cited by 207 · see citing works

Compressive Transformers for Long-Range Sequence Modelling

Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works

Modeling Localness for Self-Attention Networks

Yang, B., Tu, Z., Wong, D. F., Meng, F.-D., Chao, L., & Zhang, T. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1475 · Cited by 196 · see citing works

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

Dong, Y., Cordonnier, J.-B., & Loukas, A. (2021) · arXiv · doi:10.48550/arxiv.2103.03404 · Cited by 195 · see citing works

T-GSA: Transformer with Gaussian-Weighted Self-Attention for Speech Enhancement

Kim, J., El‐Khamy, M., & Lee, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053591 · Cited by 190 · see citing works

Understanding the Difficulty of Training Transformers

Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.463 · Cited by 190 · see citing works

Multi-Head Attention: Collaborate Instead of ConcatenatePreprint

Cordonnier, J.-B., Loukas, A., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16362 · Cited by 180 · see citing works

Multi-Head Attention with Disagreement Regularization

Li, J., Tu, Z., Yang, B., Lyu, M. R., & Zhang, T. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1317 · Cited by 176 · see citing works

Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset

Chen, X., Wu, Y., Wang, Z., Liu, S., & Li, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413535 · Cited by 175 · see citing works

Very Deep Self-Attention Networks for End-to-End Speech Recognition

Pham, N.-Q., Nguyen, T., Niehues, J., Müller, M., & Waibel, A. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2702 · Cited by 173 · see citing works

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1443 · Cited by 173 · see citing works

Random Feature AttentionPreprint

Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 169 · see citing works

ReZero is All You Need: Fast Convergence at Large DepthPreprint

Bachlechner, T. C., Majumder, B. P., Mao, H. H., Cottrell, G. W., & McAuley, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04887 · Cited by 167 · see citing works

Language Modeling with Deep Transformers

Irie, K., Zeyer, A., Schlüter, R., & Ney, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2225 · Cited by 166 · see citing works

End-to-End Object Detection with Adaptive Clustering TransformerPreprint

Zheng, M., Gao, P., Zhang, R., Li, K., Wang, X., Li, H., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09315 · Cited by 166 · see citing works

Sparse Sinkhorn AttentionPreprint

Tay, Y., Bahri, D., Liu, Y., Metzler, D., & Juan, D.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11296 · Cited by 162 · see citing works

Training Deeper Neural Machine Translation Models with Transparent Attention

Bapna, A., Chen, M., Firat, O., Cao, Y., & Wu, Y.-H. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1338 · Cited by 160 · see citing works

TransGAN: Two Transformers Can Make One Strong GANPreprint

Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · Cited by 147 · see citing works

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of ViewPreprint

Lu, Y., Li, Z., He, D., Sun, Z., Dong, B., Qin, T., Wang, L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02762 · Cited by 146 · see citing works

Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingPreprint

Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03236 · Cited by 139 · see citing works

Accelerating Neural Transformer via an Average Attention Network

Zhang, B., Xiong, D., & Su, J. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1166 · Cited by 129 · see citing works

Gaussian Transformer: A Lightweight Approach for Natural Language Inference

Guo, M., Zhang, Y., & Liu, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016489 · Cited by 126 · see citing works

Transformers without Tears: Improving the Normalization of Self-AttentionPreprint

Nguyen, T. Q., & Salazar, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05895 · Cited by 124 · see citing works

Poolingformer: Long Document Modeling with Pooling AttentionPreprint

Zhang, H., Gong, Y., Shen, Y.-L., Li, W., Lv, J., Duan, N., & Chen, W.-Z. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2105.04371 · Cited by 123 · see citing works

RealFormer: Transformer Likes Residual Attention

He, R., Ravula, A., Kanagal, B., & Ainslie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.81 · Cited by 114 · see citing works

R-Transformer: Recurrent Neural Network Enhanced TransformerPreprint

Wang, Z., Ma, Y., Liu, Z., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05572 · Cited by 103 · see citing works

Fast Transformer Decoding: One Write-Head is All You Need

Shazeer, N. (2019) · arXiv · doi:10.48550/arxiv.1911.02150 · Cited by 101 · see citing works

DeLighT: Deep and Light-weight TransformerPreprint

Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.00623 · Cited by 101 · see citing works

Learning to Encode Position for Transformer with Continuous Dynamical ModelPreprint

Liu, X., Yu, H., Dhillon, I. S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09229 · Cited by 98 · see citing works

Talking-Heads AttentionPreprint

Shazeer, N., Lan, Z., Cheng, Y., Ding, N., & Hou, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02436 · Cited by 91 · see citing works

Do We Really Need Explicit Position Encodings for Vision TransformersPreprint

Chu, X., Zhang, B., Tian, Z., Wei, X. K., & Xia, H. (2021) · arXiv (Cornell University) · Cited by 89 · see citing works

Fast Transformers with Clustered AttentionPreprint

Vyas, A., Katharopoulos, A., & Fleuret, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04825 · Cited by 86 · see citing works

Encoding word order in complex embeddingsPreprint

Wang, B., Zhao, D., Lioma, C., Li, Q., Zhang, P., & Simonsen, J. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12333 · Cited by 85 · see citing works

Insertion-based Decoding with Automatically Inferred Generation Order

Gu, J., Liu, Q., & Cho, K. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00292 · Cited by 83 · see citing works

Masked Language Modeling for Proteins via Linearly Scalable Long-Context TransformersPreprint

Choromański, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J. Q., Belanger, D., Colwell, L. J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03555 · Cited by 83 · see citing works

M6: A Chinese Multimodal PretrainerPreprint

Lin, J., Men, R., An, Y., Zhou, C., Ding, M., Zhang, Y., Wang, P., Wang, A., Jiang, L., Jia, X., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00823 · Cited by 83 · see citing works

Multi-Scale Self-Attention for Text Classification

Guo, Q., Qiu, X., Liu, P., Xue, X., & Zhang, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6290 · Cited by 77 · see citing works

On Position Embeddings in BERT

Wang, B., Shang, L., Lioma, C., Jiang, X., Yang, H., Liu, Q., & Simonsen, J. G. (2021) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 77 · see citing works

Augmenting Self-attention with Persistent Memory

Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75 · see citing works

PowerNorm: Rethinking Batch Normalization in TransformersPreprint

Shen, S., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07845 · Cited by 73 · see citing works

Large Memory Layers with Product Keys

Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H. (2019) · arXiv · doi:10.48550/arxiv.1907.05242 · Cited by 72 · see citing works

Temporal Context Aggregation for Video Retrieval with Contrastive Learning

Shao, J., Wen, X., Zhao, B., & Xue, X. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00331 · Cited by 72 · see citing works

Luna: Linear Unified Nested AttentionPreprint

Ma, X., Kong, X., Wang, S., Zhou, C., May, J., Ma, H., & Zettlemoyer, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01540 · Cited by 67 · see citing works

Hash Layers For Large Sparse Models

Roller, S., Sukhbaatar, S., Szlam, A., & Weston, J. (2021) · arXiv · doi:10.48550/arxiv.2106.04426 · Cited by 66 · see citing works

Improving Transformer Models by Reordering their Sublayers

Press, O., Smith, N. A., & Levy, O. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.270 · Cited by 64 · see citing works

BP-Transformer: Modelling Long-Range Context via Binary PartitioningPreprint

Ye, Z., Guo, Q., Gan, Q., Qiu, X., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04070 · Cited by 61 · see citing works

Mask Attention Networks: Rethinking and Strengthen Transformer

Fan, Z., Gong, Y., Liu, D., Wei, Z., Wang, S., Jiao, J., Duan, N., Zhang, R., & Huang, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.135 · Cited by 61 · see citing works

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less DataPreprint

Pilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 61 · see citing works

Hard-Coded Gaussian Attention for Neural Machine Translation

You, W., Sun, S., & Iyyer, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.687 · Cited by 60 · see citing works

Linear Transformers Are Secretly Fast Weight Programmers

Schlag, I., Irie, K., & Schmidhuber, J. (2021) · arXiv · doi:10.48550/arxiv.2102.11174 · Cited by 60 · see citing works

Character-Level Language Modeling with Deeper Self-Attention

Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works

SETransformer: Speech Enhancement Transformer

Yu, W., Zhou, J., Wang, H.-B., & Tao, L. (2021) · Cognitive Computation · doi:10.1007/s12559-020-09817-2 · Cited by 55 · see citing works

Information Aggregation for Multi-Head Attention with Routing-by-Agreement

Li, J., Yang, B., Dou, Z.-Y., Wang, X., Lyu, M. R.-T., & Tu, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1359 · Cited by 50 · see citing works

Low-Rank Bottleneck in Multi-head Attention Models

Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S. J., & Kumar, S. (2020) · arXiv · doi:10.48550/arxiv.2002.07028 · Cited by 46 · see citing works

Low-Rank and Locality Constrained Self-Attention for Sequence Modeling

Guo, Q., Qiu, X., Xue, X., & Zhang, Z. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2944078 · Cited by 45 · see citing works

M6-T: Exploring Sparse Expert Models and BeyondPreprint

An, Y., Lin, J., Men, R., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Wang, J., Li, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15082 · Cited by 35 · see citing works

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive LearningPreprint

Li, W., Gao, C., Niu, G., Xiao, X., Líu, H., Liu, J., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15409 · Cited by 29 · see citing works

Improving End-to-End Speech Synthesis with Local Recurrent Neural Network Enhanced Transformer

Zheng, Y., Li, X., Xie, F., & Lü, L. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054148 · Cited by 28 · see citing works

Early Exiting with Ensemble Internal ClassifiersPreprint

Sun, T., Zhou, Y., Liu, X., Zhang, X., Jiang, H., Cao, Z., Huang, X., & Qiu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13792 · Cited by 26 · see citing works

On the Sub-layer Functionalities of Transformer Decoder

Yang, Y., Wang, L., Shi, S., Tadepalli, P. V., Lee, S., & Tu, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.432 · Cited by 25 · see citing works

DeLighT: Very Deep and Light-weight TransformerPreprint

Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · Cited by 21 · see citing works

RealFormer: Transformer Likes Residual AttentionPreprint

He, R., Ravula, A., Kanagal, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11747 · Cited by 20 · see citing works

Controlling Computation versus Quality for Neural Sequence ModelsPreprint

Bapna, A., Arivazhagan, N., & Fırat, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07106 · Cited by 18 · see citing works

SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive ConnectionPreprint

Li, X., Meng, Y., Zhou, M., Han, Q., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09833 · Cited by 15 · see citing works

Guiding Attention for Self-Supervised Learning with Transformers

Deshpande, A., & Narasimhan, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.419 · Cited by 15 · see citing works

Linear Transformers Are Secretly Fast Weight Memory SystemsPreprint

Schlag, I., Irie, K., & Schmidhuber, J. (2021) · arXiv (Cornell University) · Cited by 14 · see citing works

Improving Multi-head Attention with Capsule Networks

Gu, S., & Feng, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_25 · Cited by 12 · see citing works

Memformer: The Memory-Augmented TransformerPreprint

Wu, Q., Lan, Z., Gu, J., & Yu, Z. (2021) · arXiv (Cornell University) · Cited by 11 · see citing works

ERNIE-Doc: A Retrospective Long-Document Modeling TransformerPreprint

Ding, S., Shang, J., Wang, S., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15688 · Cited by 11 · see citing works

LazyFormer: Self Attention with Lazy UpdatePreprint

Ying, C., Ke, G., He, D., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12702 · Cited by 11 · see citing works

Reformer-TTS: Neural Speech Synthesis with Reformer Network

Ihm, H. R., Lee, J. Y., Choi, B. J., Cheon, S. J., & Kim, N. S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2189 · Cited by 10 · see citing works

Compressed Self-Attention for Deep Metric Learning with Low-Rank Approximation

Chen, Z., Gong, M., Ge, L., & Du, B. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/285 · Cited by 8 · see citing works

Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale DatasetPreprint

Chen, X., Wu, Y., Wang, Z., Liu, S., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11395 · Cited by 8 · see citing works

Learning Shared Semantic Space for Speech-to-Text TranslationPreprint

Han, C., Wang, M., Ji, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03095 · Cited by 7 · see citing works

Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context SizePreprint

Yoshida, D., Ettinger, A., & Gimpel, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07027 · Cited by 5 · see citing works

Accelerating BERT Inference for Sequence Labeling via Early-ExitPreprint

Li, X., Shao, Y., Sun, T., Yan, H. R., Qiu, X., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13878 · Cited by 3 · see citing works

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document ModelingPreprint

Wu, C., Wu, F., Qi, T., & Huang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01040 · Cited by 3 · see citing works

Memory-Efficient Differentiable Transformer Architecture SearchPreprint

Zhao, Y., Li, D., Shen, Y., Zhang, Z., Wei, F., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14669 · Cited by 2 · see citing works

Untitled work

(2018) · doi:10.4230/lipics.cp.2025.27

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

Pre-trained models for natural language processing: A survey

Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works

Transfer learning enables predictions in network biology

Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193 · see citing works

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works

A Review of the Role of Artificial Intelligence in Healthcare

Al Kuwaiti, A., Nazer, K., Al-Reedy, A., Al-Shehri, S., Al-Muhanna, A., Subbarayalu, A. V., Al Muhanna, D., & Al-Muhanna, F. A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13060951 · Cited by 915 · see citing works

ChatGPT: Jack of all trades, master of none

Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works

A Survey of Visual Transformers

Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J. S., Tian, J., Zhang, Y. S., Shi, Z., Jianping Fan, B., & He, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 614 · see citing works

Advances in medical image analysis with vision Transformers: A comprehensive review

Azad, R., Kazerouni, A., Heidari, M., Aghdam, E. K., Molaei, A., Jia, Y., Jose, A., Roy, R., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103000 · Cited by 523 · see citing works

A comprehensive survey on applications of transformers for deep learning tasks

Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 522 · see citing works

Swin-UMamba: Mamba-Based UNet with ImageNet-Based Pretraining

Liu, J., Yang, H., Zhou, H.-Y., Xi, Y., Yu, L., Li, C., Liang, Y., Shi, G., Yu, Y., Zhang, S., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72114-4_59 · Cited by 383 · see citing works

Attention is all you need: utilizing attention in AI-enabled drug discovery

Zhang, Y., Liu, C., Liu, M., Liu, T., Lin, H., Cheng-bing, H., & Lin, N. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad467 · Cited by 385 · see citing works

Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers

Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2023) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 286 · see citing works

Comparison of Vision Transformers and Convolutional Neural Networks in Medical Image Analysis: A Systematic Review

Takahashi, S., Sakaguchi, Y., Kouno, N., Takasawa, K., Ishizu, K., Akagi, Y., Aoyama, R., Teraya, N., Bolatkan, A., Shinkai, N., et al. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02105-8 · Cited by 356 · see citing works

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works

PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer

Yu, Z., Shen, Y., Shi, J.-G., Zhao, H., Torr, P., & Zhao, G.-Y. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00415 · Cited by 349 · see citing works

Transformers in Time-Series Analysis: A Tutorial

Ahmed, S., Nielsen, I. E., Tripathi, A. G., Siddiqui, S., Ramachandran, R. P., & Rasool, G. (2023) · Circuits Systems and Signal Processing · doi:10.1007/s00034-023-02454-8 · Cited by 334 · see citing works

Machine Learning With Data Assimilation and Uncertainty Quantification for Dynamical Systems: A Review

Cheng, S. L., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., et al. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123537 · Cited by 320 · see citing works

Dual-Aspect Self-Attention Based on Transformer for Remaining Useful Life Prediction

Zhang, Z., Song, W. W., & Li, Q. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3160561 · Cited by 273 · see citing works

Deep learning for depression recognition with audiovisual cues: A review

He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 259 · see citing works

UAV Trajectory Planning for AoI-Minimal Data Collection in UAV-Aided IoT Networks by Transformer

Zhu, B., Bedeer, E. E., Nguyen, H. H., Barton, R., & Gao, Z. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3204438 · Cited by 249 · see citing works

Transformer-Based Visual Segmentation: A Survey

Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., & Loy, C. C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3434373 · Cited by 272 · see citing works

Generative artificial intelligence empowers digital twins in drug discovery and clinical trials

Bordukova, M., Makarov, N., Rodriguez-Esteban, R., Schmich, F., & Menden, M. P. (2024) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2023.2273839 · Cited by 235 · see citing works

A guide to artificial intelligence for cancer researchers

Pérez-López, R., Laleh, N. G., Mahmood, F., & Kather, J. N. (2024) · Nature reviews. Cancer · doi:10.1038/s41568-024-00694-7 · Cited by 220 · see citing works

Machine Learning-Guided Protein Engineering

Kouba, P., Kohout, P., Haddadi, F., Bushuiev, A., Samusevich, R., Sedlář, J., Damborský, J., Pluskal, T., Šivic, J., & Mazurenko, S. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c02743 · Cited by 214 · see citing works

A Survey on Large Language Models for Code Generation

Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251 · see citing works

Avoiding common machine learning pitfalls

Lones, M. A. (2024) · Patterns · doi:10.1016/j.patter.2024.101046 · Cited by 199 · see citing works

Unleashing the power of generative AI in drug discovery

Gangwal, A., & Lavecchia, A. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.103992 · Cited by 195 · see citing works

Applications of transformer-based language models in bioinformatics: a survey

Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176 · see citing works

AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint

Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 240 · see citing works

Video Transformers: A Survey

Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapés, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3243465 · Cited by 177 · see citing works

HoVer-Trans: Anatomy-Aware HoVer-Transformer for ROI-Free Breast Cancer Diagnosis in Ultrasound Images

Mo, Y., Han, C., Liu, Y., Liu, M., Shi, Z., Lin, J., Zhao, B., Huang, C., Qiu, B., Cui, Y., et al. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3236011 · Cited by 173 · see citing works

Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda

Schneider, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10916-x · Cited by 184 · see citing works

Variational Attention-Based Interpretable Transformer Network for Rotary Machine Fault Diagnosis

Li, Y., Zhou, Z., Sun, C., Chen, X., & Yan, R. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3202234 · Cited by 160 · see citing works

EMTCAL: Efficient Multiscale Transformer and Cross-Level Attention Learning for Remote Sensing Scene Classification

Tang, X., Li, M., Ma, J., Zhang, X., Liu, F., & Jiao, L. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3194505 · Cited by 147 · see citing works

PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference Transformer

Yu, Z., Shen, Y., Shi, J.-G., Zhao, H., Cui, Y., Zhang, J., Torr, P., & Zhao, G.-Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01758-1 · Cited by 161 · see citing works

Deep Learning in EEG-Based BCIs: A Comprehensive Review of Transformer Models, Advantages, Challenges, and Applications

Abibullaev, B., Keutayeva, A., & Zollanvari, A. (2023) · IEEE Access · doi:10.1109/access.2023.3329678 · Cited by 147 · see citing works

CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Li, Z., Xu, B. H., Zhu, C., & Zhao, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.175 · Cited by 149 · see citing works

A Survey on Explainable Anomaly Detection

Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139 · see citing works

A survey of deep learning-based intrusion detection in automotive applications

Lampe, B., & Meng, W. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119771 · Cited by 130 · see citing works

Contrast and Generation Make BART a Good Dialogue Emotion Recognizer

Li, S., Yan, H., & Qiu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21348 · Cited by 129 · see citing works

SOH prediction for Lithium-Ion batteries by using historical state and future load information with an AM-seq2seq model

Qian, C., Xu, B., Xia, Q., Ren, Y., Sun, B., & Wang, Z. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.120793 · Cited by 130 · see citing works

PMVT: a lightweight vision transformer for plant disease identification on mobile devices

Li, G., Wang, Y., Zhao, Q. Q., Yuan, P., & Chang, B. (2023) · Frontiers in Plant Science · doi:10.3389/fpls.2023.1256773 · Cited by 139 · see citing works

Automatic assessment of pain based on deep learning methods: A systematic review

Gkikas, S., & Tsiknakis, M. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107365 · Cited by 129 · see citing works

Transformer models in biomedicine

Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 133 · see citing works

Towards an understanding of large language models in software engineering tasks

Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 149 · see citing works

Machine learning heralding a new development phase in molecular dynamics simulations

Prašnikar, E., Ljubič, M., Perdih, A., & Borišek, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10731-4 · Cited by 115 · see citing works

Accelerating Biocatalysis Discovery with Machine Learning: A Paradigm Shift in Enzyme Engineering, Discovery, and Design

Markus, B., Gruber Christian, C., Andreas, K., Kummer, A., Stefan, L., Oberdorfer, G., Elina, S., & Šnajdrová, R. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c03417 · Cited by 116 · see citing works

Neural Architecture Search for Transformers: A Survey

Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118 · see citing works

Predicting drug–target binding affinity through molecule representation block based on multi-head attention and skip connection

Zhang, L., Wang, C.-C., & Chen, X. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac468 · Cited by 111 · see citing works

A Comprehensive Survey of Transformers for Computer Vision

Jamil, S., Piran, M. J., & Kwon, O. (2023) · Drones · doi:10.3390/drones7050287 · Cited by 111 · see citing works

Artificial Intelligence for Management of Variable Renewable Energy Systems: A Review of Current Status and Future Directions

Yousef, L. A., Yousef, H., & Rocha‐Meneses, L. (2023) · Energies · doi:10.3390/en16248057 · Cited by 122 · see citing works

Glycoinformatics in the Artificial Intelligence Era

Bojar, D., & Lisacek, F. (2022) · Chemical Reviews · doi:10.1021/acs.chemrev.2c00110 · Cited by 103 · see citing works

Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning

Li, L., Song, D., Li, X., Zeng, J., Ma, R., & Qiu, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.241 · Cited by 112 · see citing works

Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semisupervised Skeleton-Based Action Recognition

Xu, B., Shu, X., Zhang, J., Dai, G., & Song, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3247103 · Cited by 100 · see citing works

An Overview on Generative AI at Scale With Edge–Cloud Computing

Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 117 · see citing works

Transformer in reinforcement learning for decision-making: a survey

Yuan, W., Chen, J., Chen, S., Lu, L., Hu, Z., Li, P., Feng, D., Liu, F., & Chen, J. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300548 · Cited by 112 · see citing works

Computer vision and deep learning-based approaches for detection of food nutrients/nutrition: New insights and advances

Kaushal, S., Tammineni, D. K., Rana, P., Sharma, M., Sridhar, K., & Chen, H. (2024) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2024.104408 · Cited by 111 · see citing works

When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?

Chen, C., Su, J., Chen, J., Wang, Y., Bi, T., Yu, J., Wang, Y., Lin, X., Chen, T., & Zheng, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702973 · Cited by 118 · see citing works

Vision Transformer Hashing for Image Retrieval

Dubey, S. R., Singh, S. K., & Chu, W.-T. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859900 · Cited by 92 · see citing works

Multimodal transformer with adaptive modality weighting for multimodal sentiment analysis

Wang, Y., He, J., Wang, D., Wang, Q., Wan, B., & Luo, X. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127181 · Cited by 91 · see citing works

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Omar, M., Choi, S., Nyang, D., & Mohaisen, A. (2022) · IEEE Access · doi:10.1109/access.2022.3197769 · Cited by 96 · see citing works

Spatial–Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data

Liu, R., Huang, Z.-A., Hu, Y. F., Zhu, Z., Wong, K., & Tan, K. C. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3243000 · Cited by 92 · see citing works

Chatgpt: a meta-analysis after 2.5 months

Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100541 · Cited by 115 · see citing works

SELFormer: molecular representation learning via SELFIES language models

Yüksel, A., Ulusoy, E., Ünlü, A., & Doğan, T. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acdb30 · Cited by 92 · see citing works

Toward Green AI: A Methodological Survey of the Scientific Literature

Barbierato, E., & Gatti, A. (2024) · IEEE Access · doi:10.1109/access.2024.3360705 · Cited by 86 · see citing works

A general-purpose edge-feature guidance module to enhance vision transformers for plant disease identification

Chang, B.-F., Wang, Y., Zhao, X., Li, G., & Yuan, P.-Y. (2023) · Expert systems with applications · doi:10.1016/j.eswa.2023.121638 · Cited by 71 · see citing works

Exploring automated energy optimization with unstructured building data: A multi-agent based framework leveraging large language models

Xiao, T., & Xu, P. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114691 · Cited by 80 · see citing works

Relation DETR: Exploring Explicit Position Relation Prior for Object Detection

Hou, X., Liu, M., Zhang, S., Wei, P., Chen, B., & Lan, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_6 · Cited by 83 · see citing works

Applied Deep Learning-Based Crop Yield Prediction: A Systematic Analysis of Current Developments and Potential Challenges

Meghraouı, K., Sebari, I., Pilz, J., Kadi, K. A. E., & Bensiali, S. (2024) · Technologies · doi:10.3390/technologies12040043 · Cited by 80 · see citing works

On Efficient Training of Large-Scale Deep Learning Models

Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 126 · see citing works

EV battery fault diagnostics and prognostics using deep learning: Review, challenges & opportunities

Machlev, R. (2024) · Journal of Energy Storage · doi:10.1016/j.est.2024.110614 · Cited by 80 · see citing works

Applications of Artificial Intelligence to Obesity Research: Scoping Review of Methodologies

An, R., Shen, J., & Xiao, Y. (2022) · Journal of Medical Internet Research · doi:10.2196/40589 · Cited by 86 · see citing works

The Future of Material Scientists in an Age of Artificial Intelligence

Maqsood, A., Chen, C., & Jacobsson, T. J. (2024) · Advanced Science · doi:10.1002/advs.202401401 · Cited by 80 · see citing works

A Comprehensive Review of Remote Sensing and Artificial Intelligence Integration: Advances, Applications, and Challenges

Kazanskiy, N. L., Khabibullin, R. M., Nikonorov, A. V., & Khonina, S. N. (2025) · Sensors · doi:10.3390/s25195965 · Cited by 78 · see citing works

Multimodal Alignment and Fusion: A Survey

Li, S., & Tang, H. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02667-1 · Cited by 191 · see citing works

OmniVec: Learning robust representations with cross modal sharing

Srivastava, S., & Sharma, G. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00127 · Cited by 80 · see citing works

A review of transformer models in drug discovery and beyond

Jiang, J., Chen, L., Ke, L., Dou, B., Zhang, C., Feng, H., Zhu, Y., Qiu, H., Zhang, B., & Wei, G. (2024) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2024.101081 · Cited by 82 · see citing works

Improved Swin Transformer-Based Semantic Segmentation of Postearthquake Dense Buildings in Urban Areas Using Remote Sensing Images

Cui, L., Jing, X., Wang, Y., Huan, Y., Xu, Y., & Zhang, Q. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3225150 · Cited by 76 · see citing works

Transformers enable accurate prediction of acute and chronic chemical toxicity in aquatic organisms

Gustavsson, M., Käll, S., Svedberg, P., Inda-Diaz, J. S., Molander, S., Coria, J., Backhaus, T., & Kristiansson, E. (2024) · Science Advances · doi:10.1126/sciadv.adk6669 · Cited by 82 · see citing works

Improving long-term streamflow prediction in a poorly gauged basin using geo-spatiotemporal mesoscale data and attention-based deep learning: A comparative study

Ghobadi, F., & Kang, D. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128608 · Cited by 75 · see citing works

Evaluating the Effectiveness of Time Series Transformers for Demand Forecasting in Retail

Oliveira, J. M., & Ramos, P. (2024) · Mathematics · doi:10.3390/math12172728 · Cited by 74 · see citing works

Recent advances in deep learning models: a systematic literature review

Malhotra, R., & Singh, P. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15295-z · Cited by 77 · see citing works

Transformer-Gated Recurrent Unit Method for Predicting Stock Price Based on News Sentiments and Technical Indicators

Haryono, A. T., Sarno, R., & Sungkono, K. R. (2023) · IEEE Access · doi:10.1109/access.2023.3298445 · Cited by 80 · see citing works

Large language models for code completion: A systematic literature review

Husein, R. A., Aburajouh, H., & Catal, C. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103917 · Cited by 72 · see citing works

Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art

Rekavandi, A. M., Rashidi, S., Boussaïd, F., Hoefs, S., Akbaş, E., & Bennamoun, M. (2025) · ACM Computing Surveys · doi:10.1145/3758090 · Cited by 72 · see citing works

Large language models for life cycle assessments: Opportunities, challenges, and risks

Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works

From intuition to AI: evolution of small molecule representations in drug discovery

McGibbon, M., Shave, S., Dong, J. N., Gao, Y., Houston, D. R., Xie, J., Yang, Y., Schwaller, P., & Blay, V. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad422 · Cited by 74 · see citing works

MFGTN: A multi-modal fast gated transformer for identifying single trawl marine fishing vessel

Gu, Y., Hu, Z., Zhao, Y., Liao, J., & Zhang, W. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.117711 · Cited by 70 · see citing works

Performance of 4 Pre-Trained Sentence Transformer Models in the Semantic Query of a Systematic Review Dataset on Peri-Implantitis

Galli, C., Donos, N., & Calciolari, E. (2024) · Information · doi:10.3390/info15020068 · Cited by 69 · see citing works

E 2 VPT: An Effective and Efficient Approach for Visual Prompt Tuning

Han, C., Wang, Q., Cui, Y., Cao, Z., Wang, W., Qi, S., & Liu, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01604 · Cited by 87 · see citing works

Large language models for medicine: a survey

Zheng, Y., Gan, W., Chen, Z., Qi, Z., Liang, Q., & Yu, P. S. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02318-w · Cited by 81 · see citing works

When Search Engine Services Meet Large Language Models: Visions and Challenges

Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., & Helal, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3451185 · Cited by 82 · see citing works

Intrusion Detection Based on Federated Learning: A Systematic Review

Hernández-Ramos, J. L., Καρόπουλος, Γ., Chatzoglou, E., Kouliaridis, V., Campos, E. M., González-Vidal, A., & Kambourakis, G. (2025) · ACM Computing Surveys · doi:10.1145/3731596 · Cited by 108 · see citing works

From Turing to Transformers: A Comprehensive Review and Tutorial on the Evolution and Applications of Generative Transformer Models

Zhang, E. Y., Cheok, A. D., Pan, Z., Cai, J., & Yan, Y. (2023) · Sci · doi:10.3390/sci5040046 · Cited by 67 · see citing works

Next generation of computer vision for plant disease monitoring in precision agriculture: A contemporary survey, taxonomy, experiments, and future direction

Ding, W., Abdel‐Basset, M., Alrashdi, I., & Hawash, H. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120338 · Cited by 73 · see citing works

Exploring the synergies of hybrid convolutional neural network and Vision Transformer architectures for computer vision: A survey

Haruna, Y., Qin, S., Chukkol, A. H. A., Yusuf, A. A., Bello, I., & Lawan, A. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110057 · Cited by 95 · see citing works

AVLTrack: Dynamic Sparse Learning for Aerial Vision-Language Tracking

Xue, Y., Zhong, B., Jin, G., Shen, T., Tan, L., Li, N., & Zheng, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3549953 · Cited by 68 · see citing works

AIGC-empowered smart manufacturing: Prospects and challenges

Leng, J., Zheng, K., Li, R. C., Chen, C., Wang, B., Liu, Q., Chen, X., & Shen, W. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103076 · Cited by 70 · see citing works

A CNN-Attention Network for Continuous Estimation of Finger Kinematics from Surface Electromyography

Geng, Y., Yu, Z., Long, Y., Qin, L., Chen, Z., Li, Y., Guo, X., & Li, G. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3169448 · Cited by 66 · see citing works

Transformer technology in molecular science

Jian, J., Ke, L., Chen, L., Dou, B., Zhu, Y., Liu, J., Zhang, B., Zhou, T., & Wei, G. (2024) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.1725 · Cited by 68 · see citing works

Introduction to Artificial Intelligence

Papajorgji, P. J., & Moskowitz, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-78078-3_5 · Cited by 65 · see citing works

Efficient transformers: A survey

Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works

Survey: Transformer-based Models in Data Modality ConversionPreprint

Rashno, E., Eskandari, A., Anand, A., & Zulkernine, F. H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.04723 · Cited by 4 · see citing works

A comprehensive survey on applications of transformers for deep learning tasks

Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 522 · see citing works

Transformer Models and their Cross-Domain Applications: A Systematic Review

Pathania, K., Singh, S., & Sharma, A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10742-0

A Review on Transformer Models: Applications, Taxonomies, Open Issues and Challenges

Passi, N., Raj, M., & Shelke, N. A. (2024) · Asian Conference on Innovation in Technology (ASIANCON) · doi:10.1109/asiancon62057.2024.10838047 · Cited by 8 · see citing works