The DOI matches its Crossref record; 181 of the 183 references it cites were found in a registry or an index.
A survey of transformers
Open accessAbstract
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In addition to the above-mentioned DNNs, namely, AEs, RNNs, CNNs, GANs, Deep OCSVM, and DevNet, other DNNs such as Graph Neural Networks [39] and Transformers [119] have become prevalent in anomaly detection.”
From A Survey on Explainable Anomaly Detection (2024) · cites this paper as [119]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17325 · Cited by 7,752
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 1×
“, image classification [14, 33, 88], object detection [13, 88, 168, 172], image generation [61, 94] and video processing [3, 115].”
BackgroundWaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Self-citationDynamic Routing Between Capsules
Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
DARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 3,174
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
Random Features for Large-Scale Kernel Machines
Rahimi, A., & Recht, B. (2007) · Cited by 2,783
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Searching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,916
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,916
Self-citationLanguage Modeling with Gated Convolutional NetworksPreprint
Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08083 · Cited by 1,765
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
Self-citationReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394
Self-citationBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Meshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,255
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198
Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction
Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., & Lee, A. A. (2019) · ACS Central Science · doi:10.1021/acscentsci.9b00576 · Cited by 1,163
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
(2020) · International Conference on Machine Learning · Cited by 1,012
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933
Matrix capsules with EM routing
Hinton, G. E., Sabour, S., & Frosst, N. (2018) · International Conference on Learning Representations · Cited by 918
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 784
(2017) · International Conference on Machine Learning · Cited by 758
Neural Speech Synthesis with Transformer Network
Li, N., Liu, S., Liu, Y., Zhao, S., & Liu, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016706 · Cited by 754
Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear UnitsPreprint
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · Cited by 753
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 9×
“This is adopted by several studies [18, 19, 62].”
MethodInfluentialSet Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., & Teh, Y. W. (2018) · arXiv · doi:10.48550/arxiv.1810.00825 · Cited by 691
Learning Deep Transformer Models for Machine Translation
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673
On Layer Normalization in the Transformer ArchitecturePreprint
Xiong, R., Yang, Y., He, D., Zheng, K. W., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04745 · Cited by 658
Revealing the Dark Secrets of BERT
Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1445 · Cited by 650
Learning multiple visual domains with residual adapters
Rebuffi, S.-A., Bilen, H., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1705.08045 · Cited by 605
Music TransformerPreprint
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04281 · Cited by 595
Pay Less Attention with Lightweight and Dynamic ConvolutionsPreprint
Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10430 · Cited by 585
Efficient Content-Based Sparse Attention with Routing Transformers
Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509
FLAT: Chinese NER Using Flat-Lattice Transformer
Li, X., Yan, H. R., Qiu, X., & Huang, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.611 · Cited by 492
Self-citationConditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 4×
“, not able to handle sequences longer than sequences seen in the training time[20, 85].”
BackgroundInfluentialDeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Xin, J., Tang, R., Lee, J., Yu, Y., & Lin, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.204 · Cited by 476
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv · doi:10.48550/arxiv.1807.03819 · Cited by 467
CogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450
Zhang, X., Wei, F., & Zhou, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1499 · Cited by 428
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale UpPreprint
Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.07074 · Cited by 419
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2006.16668 · Cited by 412
Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376
Hierarchical Transformers for Multi-Document Summarization
Liu, Y., & Lapata, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1500 · Cited by 371
ETC: Encoding Long and Structured Inputs in Transformers
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S. K., Wang, Q., & Li, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.19 · Cited by 371
Axial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370
TENER: Adapting Transformer Encoder for Named Entity RecognitionPreprint
Yan, H. R., Deng, B., Li, X., & Qiu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04474 · Cited by 349
Self-citationAdaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335
Lite Transformer with Long-Short Range AttentionPreprint
Wu, Z., Liu, Z., Lin, J., Lin, Y., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11886 · Cited by 317
GLU Variants Improve Transformer
Shazeer, N. (2020) · arXiv · doi:10.48550/arxiv.2002.05202 · Cited by 303
Do Transformers Really Perform Bad for Graph Representation?Preprint
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05234 · Cited by 300
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, É., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1032 · Cited by 296
Understanding the Difficulty of Training Transformers
Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.463 · Cited by 290
Self-citationUnderstanding and Improving Layer NormalizationPreprint
Xu, J., Sun, X., Zhang, Z., Zhao, G., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07013 · Cited by 280
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276
Rethinking Positional Encoding in Language Pre-trainingPreprint
Ke, G., He, D., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15595 · Cited by 269
Document-Level Neural Machine Translation with Hierarchical Attention Networks
Miculicich, L., Ram, D., Pappas, N., & Henderson, J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1325 · Cited by 258
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255
How Much Position Information Do Convolutional Neural Networks Encode?
Islam, M. A., Jia, S., & Bruce, N. D. B. (2020) · arXiv · doi:10.48550/arxiv.2001.08248 · Cited by 253
Synthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251
Random Feature AttentionPreprint
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., & Kong, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02143 · Cited by 247
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA
Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01001 · Cited by 234
BERT Loses Patience: Fast and Robust Inference with Early ExitPreprint
Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., & Wei, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04152 · Cited by 227
ReZero is All You Need: Fast Convergence at Large DepthPreprint
Bachlechner, T. C., Majumder, B. P., Mao, H. H., Cottrell, G. W., & McAuley, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04887 · Cited by 226
Multi-Head Attention with Disagreement Regularization
Li, J., Tu, Z., Yang, B., Lyu, M. R.-T., & Zhang, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1317 · Cited by 214
A Survey on Visual TransformerPreprint
Han, K., Wang, Y., Chen, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12556 · Cited by 214
Self-citationUnderstanding and Improving Transformer From a Multi-Particle Dynamic System Point of ViewPreprint
Lu, Y., Li, Z., He, D., Sun, Z., Dong, B., Qin, T., Wang, L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02762 · Cited by 209
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205
Modeling Localness for Self-Attention Networks
Yang, B., Tu, Z., Wong, D. F., Meng, F., Chao, L. S., & Zhang, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1475 · Cited by 200
T-GSA: Transformer with Gaussian-Weighted Self-Attention for Speech Enhancement
Kim, J., El‐Khamy, M., & Lee, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053591 · Cited by 198
Very Deep Self-Attention Networks for End-to-End Speech Recognition
Pham, N.-Q., Nguyen, T., Niehues, J., Müller, M., & Waibel, A. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2702 · Cited by 196
Transformers without Tears: Improving the Normalization of Self-AttentionPreprint
Nguyen, T. Q., & Salazar, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05895 · Cited by 195
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Dong, Y., Cordonnier, J.-B., & Loukas, A. (2021) · arXiv · doi:10.48550/arxiv.2103.03404 · Cited by 195
End-to-End Object Detection with Adaptive Clustering TransformerPreprint
Zheng, M., Gao, P., Zhang, R., Li, K., Wang, X., Li, H., & Dong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09315 · Cited by 192
Developing Real-Time Streaming Transformer Transducer for Speech Recognition on Large-Scale Dataset
Chen, X., Wu, Y., Wang, Z., Liu, S., & Li, J. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413535 · Cited by 189
Sparse Sinkhorn AttentionPreprint
Tay, Y., Bahri, D., Liu, Y., Metzler, D., & Juan, D.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11296 · Cited by 185
Language Modeling with Deep Transformers
Irie, K., Zeyer, A., Schlüter, R., & Ney, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2225 · Cited by 181
Multi-Head Attention: Collaborate Instead of ConcatenatePreprint
Cordonnier, J.-B., Loukas, A., & Jaggi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16362 · Cited by 180
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel
Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1443 · Cited by 173
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingPreprint
Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03236 · Cited by 159
TransGAN: Two Transformers Can Make One Strong GANPreprint
Jiang, Y., Chang, S., & Wang, Z. (2021) · arXiv (Cornell University) · Cited by 147
Training Deeper Neural Machine Translation Models with Transparent Attention
Bapna, A., Chen, M., Fırat, O., Cao, Y. J., & Wu, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1338 · Cited by 144
RealFormer: Transformer Likes Residual Attention
He, R., Ravula, A., Kanagal, B., & Ainslie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.81 · Cited by 133
Talking-Heads AttentionPreprint
Shazeer, N., Lan, Z., Cheng, Y., Ding, N., & Hou, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02436 · Cited by 132
Gaussian Transformer: A Lightweight Approach for Natural Language Inference
Guo, M., Zhang, Y., & Liu, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016489 · Cited by 130
M6: A Chinese Multimodal PretrainerPreprint
Lin, J., Men, R., An, Y., Zhou, C., Ding, M., Zhang, Y., Wang, P., Wang, A., Jiang, L., Jia, X., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00823 · Cited by 126
R-Transformer: Recurrent Neural Network Enhanced TransformerPreprint
Wang, Z., Ma, Y., Liu, Z., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05572 · Cited by 124
Poolingformer: Long Document Modeling with Pooling AttentionPreprint
Zhang, H., Gong, Y., Shen, Y.-L., Li, W., Lv, J., Duan, N., & Chen, W.-Z. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2105.04371 · Cited by 123
Accelerating Neural Transformer via an Average Attention Network
Zhang, B., Xiong, D., & Su, J. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1166 · Cited by 116
DeLighT: Deep and Light-weight TransformerPreprint
Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.00623 · Cited by 113
Masked Language Modeling for Proteins via Linearly Scalable Long-Context TransformersPreprint
Choromański, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J. Q., Belanger, D., Colwell, L. J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03555 · Cited by 108
Learning to Encode Position for Transformer with Continuous Dynamical ModelPreprint
Liu, X., Yu, H., Dhillon, I. S., & Hsieh, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09229 · Cited by 104
Self-citationFast Transformer Decoding: One Write-Head is All You Need
Shazeer, N. (2019) · arXiv · doi:10.48550/arxiv.1911.02150 · Cited by 101
Encoding word order in complex embeddingsPreprint
Wang, B., Zhao, D., Lioma, C., Li, Q., Zhang, P., & Simonsen, J. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12333 · Cited by 101
Do We Really Need Explicit Position Encodings for Vision TransformersPreprint
Chu, X., Zhang, B., Tian, Z., Wei, X. K., & Xia, H. (2021) · arXiv (Cornell University) · Cited by 89
Insertion-based Decoding with Automatically Inferred Generation Order
Gu, J., Liu, Q., & Cho, K. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00292 · Cited by 88
Fast Transformers with Clustered AttentionPreprint
Vyas, A., Katharopoulos, A., & Fleuret, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04825 · Cited by 86
Mask Attention Networks: Rethinking and Strengthen Transformer
Fan, Z., Gong, Y., Liu, D., Wei, Z., Wang, S., Jiao, J., Duan, N., Zhang, R., & Huang, X. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.135 · Cited by 86
BP-Transformer: Modelling Long-Range Context via Binary PartitioningPreprint
Ye, Z., Guo, Q., Gan, Q., Qiu, X., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04070 · Cited by 79
Self-citationMulti-Scale Self-Attention for Text Classification
Guo, Q., Qiu, X., Liu, P., Xue, X., & Zhang, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6290 · Cited by 78
Self-citationOn Position Embeddings in BERT
Wang, B., Shang, L., Lioma, C., Jiang, X., Yang, H., Liu, Q., & Simonsen, J. G. (2021) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 77
Temporal Context Aggregation for Video Retrieval with Contrastive Learning
Shao, J., Wen, X., Zhao, B., & Xue, X. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00331 · Cited by 76
Augmenting Self-attention with Persistent Memory
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75
Pilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 75
PowerNorm: Rethinking Batch Normalization in TransformersPreprint
Shen, S., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07845 · Cited by 74
Large Memory Layers with Product Keys
Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H. (2019) · arXiv · doi:10.48550/arxiv.1907.05242 · Cited by 72
Hash Layers For Large Sparse Models
Roller, S., Sukhbaatar, S., Szlam, A., & Weston, J. (2021) · arXiv · doi:10.48550/arxiv.2106.04426 · Cited by 66
Hard-Coded Gaussian Attention for Neural Machine Translation
You, W., Sun, S., & Iyyer, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.687 · Cited by 65
Improving Transformer Models by Reordering their Sublayers
Press, O., Smith, N. A., & Levy, O. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.270 · Cited by 64
M6-T: Exploring Sparse Expert Models and BeyondPreprint
An, Y., Lin, J., Men, R., Zhou, C., Jiang, L., Jia, X., Wang, A., Zhang, J., Wang, J., Li, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15082 · Cited by 60
Linear Transformers Are Secretly Fast Weight Programmers
Schlag, I., Irie, K., & Schmidhuber, J. (2021) · arXiv · doi:10.48550/arxiv.2102.11174 · Cited by 60
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56
SETransformer: Speech Enhancement Transformer
Yu, W., Zhou, J., Wang, H.-B., & Tao, L. (2021) · Cognitive Computation · doi:10.1007/s12559-020-09817-2 · Cited by 55
Information Aggregation for Multi-Head Attention with Routing-by-Agreement
Li, J., Yang, B., Dou, Z.-Y., Wang, X., Lyu, M. R.-T., & Tu, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1359 · Cited by 53
Low-Rank and Locality Constrained Self-Attention for Sequence Modeling
Guo, Q., Qiu, X., Xue, X., & Zhang, Z. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2944078 · Cited by 47
Self-citationLow-Rank Bottleneck in Multi-head Attention Models
Bhojanapalli, S., Yun, C., Rawat, A. S., Reddi, S. J., & Kumar, S. (2020) · arXiv · doi:10.48550/arxiv.2002.07028 · Cited by 46
Controlling Computation versus Quality for Neural Sequence ModelsPreprint
Bapna, A., Arivazhagan, N., & Fırat, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07106 · Cited by 33
Improving End-to-End Speech Synthesis with Local Recurrent Neural Network Enhanced Transformer
Zheng, Y., Li, X., Xie, F., & Lü, L. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054148 · Cited by 29
On the Sub-layer Functionalities of Transformer Decoder
Yang, Y., Wang, L., Shi, S., Tadepalli, P. V., Lee, S., & Tu, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.432 · Cited by 29
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive LearningPreprint
Li, W., Gao, C., Niu, G., Xiao, X., Líu, H., Liu, J., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15409 · Cited by 29
Early Exiting with Ensemble Internal ClassifiersPreprint
Sun, T., Zhou, Y., Liu, X., Zhang, X., Jiang, H., Cao, Z., Huang, X., & Qiu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13792 · Cited by 26
Self-citationLazyFormer: Self Attention with Lazy UpdatePreprint
Ying, C., Ke, G., He, D., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12702 · Cited by 24
DeLighT: Very Deep and Light-weight TransformerPreprint
Mehta, S., Ghazvininejad, M., Iyer, S. V., Zettlemoyer, L., & Hajishirzi, H. (2020) · arXiv (Cornell University) · Cited by 21
RealFormer: Transformer Likes Residual AttentionPreprint
He, R., Ravula, A., Kanagal, B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11747 · Cited by 20
Guiding Attention for Self-Supervised Learning with Transformers
Deshpande, A., & Narasimhan, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.419 · Cited by 19
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive ConnectionPreprint
Li, X., Meng, Y., Zhou, M., Han, Q., Wu, F. Q., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09833 · Cited by 17
Linear Transformers Are Secretly Fast Weight Memory SystemsPreprint
Schlag, I., Irie, K., & Schmidhuber, J. (2021) · arXiv (Cornell University) · Cited by 14
Improving Multi-head Attention with Capsule Networks
Gu, S., & Feng, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_25 · Cited by 12
Memformer: The Memory-Augmented TransformerPreprint
Wu, Q., Lan, Z., Gu, J., & Yu, Z. (2021) · arXiv (Cornell University) · Cited by 11
ERNIE-Doc: A Retrospective Long-Document Modeling TransformerPreprint
Ding, S., Shang, J., Wang, S., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15688 · Cited by 11
Reformer-TTS: Neural Speech Synthesis with Reformer Network
Ihm, H. R., Lee, J. Y., Choi, B. J., Cheon, S. J., & Kim, N. S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2189 · Cited by 10
Compressed Self-Attention for Deep Metric Learning with Low-Rank Approximation
Chen, Z., Gong, M., Ge, L., & Du, B. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/285 · Cited by 8
Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale DatasetPreprint
Chen, X., Wu, Y., Wang, Z., Liu, S., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11395 · Cited by 8
Learning Shared Semantic Space for Speech-to-Text TranslationPreprint
Han, C., Wang, M., Ji, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03095 · Cited by 7
Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context SizePreprint
Yoshida, D., Ettinger, A., & Gimpel, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07027 · Cited by 5
Accelerating BERT Inference for Sequence Labeling via Early-ExitPreprint
Li, X., Shao, Y., Sun, T., Yan, H. R., Qiu, X., & Huang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13878 · Cited by 3
Self-citationWu, C., Wu, F., Qi, T., & Huang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01040 · Cited by 3
Memory-Efficient Differentiable Transformer Architecture SearchPreprint
Zhao, Y., Li, D., Shen, Y., Zhang, Z., Wei, F., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14669 · Cited by 2
Luna: Linear Unified Nested Attention
Ma, X., Kong, X., Wang, S., Zhou, C., May, J., Ma, H., & Zettlemoyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.01540
Flagged first, then by citations
2 of the 183 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
Self-citationTransfer learning enables predictions in network biology
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. (2023) · Nature · doi:10.1038/s41586-023-06139-9 · Cited by 1,193
Self-citationMultimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
A Review of the Role of Artificial Intelligence in Healthcare
Al Kuwaiti, A., Nazer, K., Al-Reedy, A., Al-Shehri, S., Al-Muhanna, A., Subbarayalu, A. V., Al Muhanna, D., & Al-Muhanna, F. A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13060951 · Cited by 915
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758
“Only 18% of users using the OpenAI API queried GPT-3 model with tasks familiar to typical NLP tasks, most of which are analytical.”
BackgroundInfluential
Metadata
- Authors
- Lin, T., Wang, Y., Liu, X., & Qiu, X.
- Affiliations
- Fudan University (CN)
- Corresponding author
- Qiu, Xipeng
- Venue
- AI Open
- Publisher
- Elsevier BV
- Published
- 2022
- Volume
- 3
- Pages
- 111-132
- ISSN
- 2666-6510 (ISSN-L)
- OpenAlex
- W3171649327
- Cited by
- 1,422
- References
- 183
- Citations by year
- 2026: 217 · 2025: 423 · 2024: 438 · 2023: 264 · 2022: 69
- FWCI
- 98.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- AI Open (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62022027)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
