Cited by 1,422Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 181 of the 183 references it cites were found in a registry or an index.

A survey of transformers

Open access

Lin, Tianyang, Wang, Yuxin, Liu, Xiangyang, Qiu, Xipeng·2022·AI Open

DOI 10.1016/j.aiopen.2022.10.001

Open this paper in CoChat — free account Open PDF

Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.aiopen.2022.10.001

Cites

181 of 181 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285

Flagged first, then by citations

Show all 183 references

2 of the 183 references couldn’t be matched to a known paper.

Sorted by citations

Items 101–200 of 1,001

Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysis
Bahr, L., Wehner, C., Wewerka, J., Bittencourt, J., Schmid, U., & Daub, R. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100807 · Cited by 92
Comparative Analysis of Deep Natural Networks and Large Language Models for Aspect-Based Sentiment Analysis
Mughal, N., Mujtaba, G., Shaikh, S., Kumar, A., & Daudpota, S. M. (2024) · IEEE Access · doi:10.1109/access.2024.3386969 · Cited by 85
Integrated Multi-Head Self-Attention Transformer model for electricity demand prediction incorporating local climate variables
Ghimire, S., Nguyen‐Huy, T., AL‐Musaylh, M. S., Deo, R. C., Casillas-Pérez, D., & Salcedo‐Sanz, S. (2023) · Energy and AI · doi:10.1016/j.egyai.2023.100302 · Cited by 68
From explainable to interpretable deep learning for natural language processing in healthcare: How far from reality?
Huang, G., Li, Y., Jameel, S., Long, Y., & Papanastasiou, G. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.05.004 · Cited by 81
A Review of Large Language Models: Fundamental Architectures, Key Technological Evolutions, Interdisciplinary Technologies Integration, Optimization and Compression Techniques, Applications, and Challenges
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
KNN-Contrastive Learning for Out-of-Domain Intent Classification
Zhou, Y., Liu, P., & Qiu, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.352 · Cited by 61
Self-citation
Reliability of electric vehicle charging infrastructure: A cross-lingual deep learning approach
Liu, Y., Francis, A., Hollauer, C., Lawson, M. C., Shaikh, O., Cotsman, A., Bhardwaj, K., Banboukian, A., Li, M., Webb, A., et al. (2023) · Communications in Transportation Research · doi:10.1016/j.commtr.2023.100095 · Cited by 67
Exploring transformer models for sentiment classification: A comparison of BERT , RoBERTa , ALBERT , DistilBERT , and XLNet
Areshey, A., & Mathkour, H. I. (2024) · Expert Systems · doi:10.1111/exsy.13701 · Cited by 78
What is generative in generative artificial intelligence? A design-based perspective
Bordas, A., Masson, P. L., Thomas, M., & Weil, B. (2024) · Research in Engineering Design · doi:10.1007/s00163-024-00441-x · Cited by 86
Automatic detection for bioacoustic research: a practical guide from and for biologists and computer scientists
Kershenbaum, A., Akçay, Ç., Saheer, L. B., Barnhill, A., Best, P., Cauzinille, J., Clink, D. J., Dassow, A., Dufourq, E., Growcott, J., et al. (2024) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.13155 · Cited by 72
A Fault Diagnosis Method for Rolling Bearing Based on 1D-ViT Model
Xu, P., & Zhang, L. (2023) · IEEE Access · doi:10.1109/access.2023.3268534 · Cited by 59
The sudden disruptive rise of generative artificial intelligence? An evaluation of their impact on higher education and the global workplace
Wong, W. K. O. (2024) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2024.100278 · Cited by 73
Opportunities and challenges in transformer neural networks for battery state estimation: Charge, health, lifetime, and safety
Zhao, J., Han, X., Wu, Y., Wang, Z., & Burke, A. F. (2024) · Journal of Energy Chemistry · doi:10.1016/j.jechem.2024.11.011 · Cited by 66
CNNs, RNNs and Transformers in human action recognition: a survey and a hybrid model
Alomar, K., Aysel, H. I., & Cai, X. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11388-3 · Cited by 104
Video deepfake detection using a hybrid CNN-LSTM-Transformer model for identity verification
Πετμεζάς, Γ., Vanian, V., Konstantoudakis, K., Almaloglou, E. E. I., & Zarpalas, D. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20548-6 · Cited by 73
AI-powered integration of multimodal imaging in precision medicine for neuropsychiatric disorders
Huang, W., & Shu, N. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102132 · Cited by 69
Object detection using convolutional neural networks and transformer-based models: a review
Shah, S., & Tembhurne, J. V. (2023) · Journal of Electrical Systems and Information Technology · doi:10.1186/s43067-023-00123-z · Cited by 72
Multiobjective multihydropower reservoir operation optimization with transformer-based deep reinforcement learning
Wu, R., Wang, R., Hao, J., Wu, Q., & Wang, P. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.130904 · Cited by 56
Vision transformers in domain adaptation and domain generalization: a study of robustness
Alijani, S., Fayyad, J., & Najjaran, H. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10353-5 · Cited by 63
Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 101
LRCN: Layer-residual Co-Attention Networks for visual question answering
Han, D., Shi, J., Zhao, J., Wu, H., Zhou, Y., Li, L.-H., Khan, M. K., & Li, K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125658 · Cited by 60
Advancing drug discovery with deep attention neural networks
Lavecchia, A. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104067 · Cited by 57
Transformers in biosignal analysis: A review
Anwar, A., Khalifa, Y., Coyle, J. L., & Sejdić, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102697 · Cited by 60
A survey of transformer networks for time series forecasting
Zhao, J., Chu, F., Xie, L., Che, Y., Wu, Y., & Burke, A. F. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100883 · Cited by 71
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Świątczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 59
Deep Learning for State of Health Estimation of Lithium-Ion Batteries in Electric Vehicles: A Systematic Review
Liu, C., Li, H., Li, K., Wu, Y., & Lv, B. (2025) · Energies · doi:10.3390/en18061463 · Cited by 67
JDMR-Net: Joint Detection and Modulation Recognition Networks for LPI Radar Signals
Zhang, Z., Zhu, M., Li, Y., & Wang, S. (2023) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3293074 · Cited by 53
Sparsity in transformers: A systematic literature review
Farina, M., Ahmad, U., Taha, A. T. M., Younes, H., Mesbah, Y., Yu, X., & Witold Pedrycz, C. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127468 · Cited by 59
Single-cell foundation models: bringing artificial intelligence into cell biology
Baek, S., Song, K., & Lee, I. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01547-5 · Cited by 46
A Survey on Data-Driven Runoff Forecasting Models Based on Neural Networks
Sheng, Z., Wen, S., Feng, Z.-K., Gong, J., Shi, K., Guo, Z., Yang, Y., & Huang, T. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3259434 · Cited by 53
CNN and Transformer Fusion for Remote Sensing Image Semantic Segmentation
Chen, X., Li, D.-F., Liu, M.-Y., & Jia, J. (2023) · Remote Sensing · doi:10.3390/rs15184455 · Cited by 45
Estimating SoC, SoH, or RuL of Rechargeable Batteries via IoT: A Review
Bokstaller, J., Schneider, J., & Brocke, J. V. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3342367 · Cited by 52
Hallucinations in Large Language Models (LLMs)
Reddy, G. P., Kumar, Y. V. P., & Prakash, K. P. (2024) · IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream) · doi:10.1109/estream61684.2024.10542617 · Cited by 68
Mask-guided BERT for few-shot text classification
Liao, W., Liu, Z., Dai, H., Wu, Z., Zhang, Y., Huang, X., Chen, Y., Jiang, X., Liu, D., Zhu, D., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128576 · Cited by 73
Transformer for Nonintrusive Load Monitoring: Complexity Reduction and Transferability
Wang, L., Mao, S., & Nelms, R. M. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3163347 · Cited by 51
Transformer-Based Device-Type Identification in Heterogeneous IoT Traffic
Luo, Y., Chen, X., Ge, N., Feng, W., & Lü, J. (2022) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3221967 · Cited by 53
AI-Enabled Portable E-Nose Regression Predicting Harmful Molecules in a Gas Mixture
Yang, J., Hu, X., Feng, L., Liu, Z., Murtazt, A., Qin, W., Zhou, M. Q., Liu, J., Bi, Y., Qian, J., et al. (2024) · ACS Sensors · doi:10.1021/acssensors.4c00050 · Cited by 50
DMGAN: Dynamic Multi-Hop Graph Attention Network for Traffic Forecasting
Li, R., Zhang, F., Li, T., Zhang, N., & Zhang, T. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3221316 · Cited by 49
LoT: A Transformer-Based Approach Based on Channel State Information for Indoor Localization
Li, W., Meng, X., Zhao, Z., Liu, Z., Chen, C., & Wang, H. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3318835 · Cited by 55
Deep learning for precipitation nowcasting: A survey from the perspective of time series forecasting
An, S., Oh, T.-J., Sohn, E.-H., & Kim, D. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126301 · Cited by 60
Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting
Nazi, Z. A., Hossain, M. R., & Mamun, F. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100124 · Cited by 53
Effect of tokenization on transformers for biological sequences
Dotan, E., Jaschek, G., Pupko, T., & Belinkov, Y. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae196 · Cited by 49
Knowledge-enhanced semantic communication system with OFDM transmissions
Xu, X., Xiong, H., Wang, Y., Che, Y., Han, S., Wang, B., & Zhang, P. (2023) · Science China Information Sciences · doi:10.1007/s11432-022-3624-4 · Cited by 48
Self-citation
Deep Learning-Based Ship Speed Prediction for Intelligent Maritime Traffic Management
Mekkaoui, S. E., Benabbou, L., Caron, S., & Berrado, A. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11010191 · Cited by 51
GCAM: lightweight image inpainting via group convolution and attention mechanism
Chen, Y., Xia, R., Yang, K., & Zou, K. (2023) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-023-01999-z · Cited by 46
Empowering precision medicine: AI-driven schizophrenia diagnosis via EEG signals: A comprehensive review from 2002–2023
Jafari, M., Sadeghi, D., Shoeibi, A., Alinejad‐Rokny, H., Beheshti, A., López-García, D., Chen, Z., Acharya, U. R., & Górriz, J. M. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05155-6 · Cited by 49
Point Cloud Completion: A Survey
Tesema, K. W., Hill, L., Jones, M. W., Ahmad, M., & Tam, G. K. L. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3344935 · Cited by 46
A Systematic Review and Identification of the Challenges of Deep Learning Techniques for Undersampled Magnetic Resonance Image Reconstruction
Hossain, M. B., Shinde, R. K., Oh, S., Kwon, K., & Kim, N. (2024) · Sensors · doi:10.3390/s24030753 · Cited by 46
PETformer: Long-Term Time Series Forecasting via Placeholder-Enhanced Transformer
Lin, S., Lin, W., Wu, W., Wang, S., & Wang, Y. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3502437 · Cited by 66
Self-citation
Deep learning-based regression of food quality attributes using near-infrared spectroscopy and hyperspectral imaging: A review
Xiao, Y., Zhou, L., Zhao, Y., Qi, H., Pu, Y., & Zhang, C. (2025) · Food Chemistry · doi:10.1016/j.foodchem.2025.145932 · Cited by 46
Fully Simulated-Data-Driven Transfer-Learning Method for Rolling-Bearing-Fault Diagnosis
Ai, T., Liu, Z., Zhang, J., Liu, H., Jin, Y., & Zuo, M. J. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3301901 · Cited by 45
ICFormer: A Deep Learning model for informed lithium-ion battery diagnosis and early knee detection
Costa, N., Anseán, D., Dubarry, M., & Sánchez, L. (2023) · Journal of Power Sources · doi:10.1016/j.jpowsour.2023.233910 · Cited by 45
A Dual-Scale Transformer-Based Remaining Useful Life Prediction Model in Industrial Internet of Things
Li, J., Wang, K., Hou, X., Lan, D., Yunwen, W., Wang, H., Liu, L., & Mumtaz, S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3376706 · Cited by 45
Artificial intelligence and machine learning in cell-free-DNA-based diagnosticsPreprint
Tsui, W., Ding, S. C., Jiang, P., & Lo, Y. M. D. (2025) · Genome Research · doi:10.1101/gr.278413.123 · Cited by 51
A trained Mask R-CNN model over PlanetScope imagery for very-high resolution surface water mapping in boreal forest-tundra
Freitas, P., Vieira, G. T., Canário, J., Vincent, W. F., Pina, P., & Mora, C. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114047 · Cited by 46
Transformer-Based Channel Prediction for Rate-Splitting Multiple Access-Enabled Vehicle-to-Everything Communication
Zhang, S., Zhang, S., Mao, Y., Yeung, K. L., Clerckx, B., & Quek, T. Q. S. (2024) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3395670 · Cited by 52
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 255
Self-citation
Self-Supervised RGB-NIR Fusion Video Vision Transformer Framework for rPPG Estimation
Park, S., Kim, B.-K., & Dong, S.-Y. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3217867 · Cited by 43
Optimizing Vision Transformers for Histopathology: Pretraining and Normalization in Breast Cancer Classification
Baroni, G. L., Rasotto, L., Roitero, K., Tulisso, A., Loreto, C. D., & Mea, V. D. (2024) · Journal of Imaging · doi:10.3390/jimaging10050108 · Cited by 43
Optimising TinyML with quantization and distillation of transformer and mamba models for indoor localisation on edge devices
Suwannaphong, T., Jovan, F., Craddock, I. J., & McConville, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-94205-9 · Cited by 59
Bi-hemisphere asymmetric attention network: recognizing emotion from EEG signals based on the transformer
Zhong, X., Gu, Y., Luo, Y., Zeng, X., & Liu, G. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04228-2 · Cited by 43
TransFoxMol: predicting molecular property with focused attention
Gao, J., Shen, Z., Xie, Y., Lu, J., Lu, Y., Chen, S., Bian, Q., Guo, Y., Shen, L., Wu, J. F., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad306 · Cited by 41
Intelligent predicting and monitoring of ultra-high-performance fiber reinforced concrete composites − A review
Fan, D., Chen, Z., Cao, Y., Liu, K., Yin, T., Lv, X., Lu, J., Zhou, A., Poon, C. S., & Yu, R. (2024) · Composites Part A Applied Science and Manufacturing · doi:10.1016/j.compositesa.2024.108555 · Cited by 42
Dynamic Temporal Dependency Model for Multiple Steps Ahead Short-Term Load Forecasting of Power System
Jiang, B., Yang, H., Wang, Y., Liu, Y., Geng, H., Zeng, H., & Ding, J. (2024) · IEEE Transactions on Industry Applications · doi:10.1109/tia.2024.3375802 · Cited by 41
Self-citation
Recent Advances in Remote Sensing and Artificial Intelligence for River Water Quality Forecasting: A Review
Pan, D., Deng, Y., Yang, S. X., & Gharabaghi, B. (2025) · Environments · doi:10.3390/environments12050158 · Cited by 42
Artificial intelligence perspectives: A systematic literature review on modeling, control, and optimization of fluid catalytic cracking
Khaldi, M. K., Al‐Dhaifallah, M., & Taha, O. (2023) · Alexandria Engineering Journal · doi:10.1016/j.aej.2023.08.066 · Cited by 43
Understanding the brain with attention: A survey of transformers in brain sciences
Chen, C., Wang, H., Chen, Y., Yin, Z., Yang, X., Ning, H., Zhang, Q., Li, W., Xiao, R., & Zhao, J. (2023) · Brain‐X · doi:10.1002/brx2.29 · Cited by 44
Applications and perspectives of Generative Artificial Intelligence in agriculture
Pallottino, F., Violino, S., Figorilli, S., Pane, C., Aguzzi, J., Colle, G., Nemmi, E. N., Montaghi, A., Chatzievangelou, D., Antonucci, F., et al. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.109919 · Cited by 44
A Comprehensive Survey on Deep Learning-based Predictive Maintenance
Khan, U., Cheng, D. S., Setti, F., Fummi, F., Cristani, M., & Capogrosso, L. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3732287 · Cited by 49
LW-CMDANet: A Novel Attention Network for SAR Automatic Target Recognition
Lang, P., Fu, X., Cheng, F., Dong, J., Qin, R., & Martorella, M. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3195074 · Cited by 43
All You Need Is Feedback: Communication With Block Attention Feedback Codes
Özfatura, E., Shao, Y., Perotti, A., Popović, B. M., & Gündüz, D. (2022) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2022.3223901 · Cited by 41
CAT: Center Attention Transformer With Stratified Spatial–Spectral Token for Hyperspectral Image Classification
Feng, J., Wang, Q., Zhang, G., Jia, X., & Yin, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3374954 · Cited by 39
Towards a comprehensive understanding of the impact of large language models on natural language processing: challenges, opportunities and future directions^†
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 42
Real-time prediction of TBM penetration rates using a transformer-based ensemble deep learning model
Zhang, M., Ji, A., Zhou, C., Ding, Y., & Wang, L. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105793 · Cited by 38
Advances in artificial intelligence for olfaction and gustation: a comprehensive review
Hao, Z., Li, H., Guo, J., & Xu, Y. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11309-4 · Cited by 41
Transformer based Kalman Filter with EM algorithm for time series prediction and anomaly detection of complex systems
Ma, M., Fu, L., Zhai, Z., & Sun, R.-B. (2024) · Measurement · doi:10.1016/j.measurement.2024.114378 · Cited by 40
Novel approaches for fake news detection based on attention-based deep multiple-instance learning using contextualized neural language models
Karaoğlan, K. M. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128263 · Cited by 40
Precision DNA methylation typing via hierarchical clustering of Nanopore current signals and attention-based neural network
Dai, Q., Chen, H., Yi, W., Zhao, J., Zhang, W., He, P., Liu, X., Zheng, Y., & Shi, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae596 · Cited by 37
Self-citation
When Geoscience Meets Foundation Models: Toward a general geoscience artificial intelligence system
Zhang, H., Xu, J., Cui, H., Li, L., Yang, Y., Tang, C., & Boers, N. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3496478 · Cited by 59
AI augmented edge and fog computing for Internet of Health Things (IoHT)
Rajagopal, D., & Subramanian, P. K. T. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2431 · Cited by 41
Electrical Power Generation Forecasting from Renewable Energy Systems Using Artificial Intelligence Techniques
Baseer, M. A., Almunif, A., Alsaduni, I., & Tazeen, N. (2023) · Energies · doi:10.3390/en16186414 · Cited by 36
Large Language Model Empowered by Domain-Specific Knowledge Base for Industrial Equipment Operation and Maintenance
Wang, H., & Li, Y. (2023) · International Conference on System Reliability and Safety Engineering (SRSE) · doi:10.1109/srse59585.2023.10336112 · Cited by 41
Natural language processing with transformers: a review
Țucudean, G., Bucos, M., Drăgulescu, B., & Căleanu, C. D. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2222 · Cited by 57
Motor Imagery EEG signals classification using a Transformer-GCN approach
Hamidi, A., & Kiani, K. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112686 · Cited by 36
Predicting Harmful Algal Blooms Using Explainable Deep Learning Models: A Comparative Study
Demiray, B. Z., Mermer, O., Baydaroğlu, Ö., & Demir, İ. (2025) · Water · doi:10.3390/w17050676 · Cited by 42
Solvent Screening for Separation Processes Using Machine Learning and High-Throughput Technologies
Edaugal, J. P., Zhang, D., Liu, D., Glezakou, V., & Sun, N. (2025) · Chem & Bio Engineering · doi:10.1021/cbe.4c00170 · Cited by 39
Object Detection and X-Ray Security Imaging: A Survey
Wu, J., Xu, X., & Yang, J. (2023) · IEEE Access · doi:10.1109/access.2023.3273736 · Cited by 40
Improved Transformer-Based Privacy-Preserving Architecture for Intrusion Detection in Secure V2X Communications
Lai, Q., Chen, X., Chen, J., Wang, W. W., Chen, J., Gadekallu, T. R., Cai, M., & Hu, X. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3324081 · Cited by 40
Transformers in health: a systematic review on architectures for longitudinal data analysis
Siebra, C., Kurpicz-Briki, M., & Wac, K. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10677-z · Cited by 44
CD-CTFM: A Lightweight CNN-Transformer Network for Remote Sensing Cloud Detection Fusing Multiscale Features
Ge, W., Yang, X., Jiang, R., Shao, W., & Zhang, L. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3361933 · Cited by 38
Multilevel Feedback Joint Representation Learning Network Based on Adaptive Area Elimination for Cross-View Geo-Localization
Ge, F., Zhang, Y., Wang, L., Liu, W., Liu, Y., Coleman, S., & Kerr, D. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3396330 · Cited by 35
A deep cross-modal neural cognitive diagnosis framework for modeling student performance
Song, L., He, M., Shang, X., Yang, C., Liu, J., Yu, M., & Lu, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120675 · Cited by 34
InParformer: Evolutionary Decomposition Transformers with Interactive Parallel Attention for Long-Term Time Series Forecasting
Cao, H., Huang, Z., Yao, T., Wang, J., He, H., & Wang, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25845 · Cited by 37
Self-citation
Enhancing hydrological modeling with transformers: a case study for 24-h streamflow prediction
Demiray, B. Z., Sit, M. A., Mermer, O., & Demir, İ. (2024) · Water Science & Technology · doi:10.2166/wst.2024.110 · Cited by 45
A novel approach to forecast water table rise in arid regions using stacked ensemble machine learning and deep artificial intelligence models
Elzain, H. E., Abdalla, O. A. E., Al-Maktoumi, A. K., Rashidovich, K. A., & Eltayeb, M. M. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131668 · Cited by 35
The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
Pashangpour, S., & Nejat, G. (2024) · Robotics · doi:10.3390/robotics13080112 · Cited by 42
Self-Administered Interventions Based on Natural Language Processing Models for Reducing Depressive and Anxiety Symptoms: Systematic Review and Meta-Analysis
Villarreal‐Zegarra, D., Reátegui-Rivera, C. M., García-Serna, J., Quispe-Callo, G., Lázaro-Cruz, G., Centeno-Terrazas, G., Gálvez-Arévalo, R. A., Escobar-Agreda, S., Rodríguez, A. D., & Finkelstein, J. (2024) · JMIR Mental Health · doi:10.2196/59560 · Cited by 39
CGAformer: Multi-scale feature Transformer with MLP architecture for short-term photovoltaic power forecasting
Chen, R., Liu, G., Cao, Y., Xiao, G., & Tang, J. (2024) · Energy · doi:10.1016/j.energy.2024.133495 · Cited by 35
Transformer-based image and video inpainting: current challenges and future directions
Elharrouss, O., Damseh, R., Belkacem, A. N., Badidi, E., & Lakas, A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11075-9 · Cited by 33
Analog in-memory computing attention mechanism for fast and energy-efficient large language models
Leroux, N., Manea, P.-P., Sudarshan, C., Finkbeiner, J., Siegel, S., Strachan, J. P., & Neftci, E. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00854-1 · Cited by 44

Authors: Lin, T., Wang, Y., Liu, X., & Qiu, X.
Affiliations: Fudan University (CN)
Corresponding author: Qiu, Xipeng
ORCID: Lin, Tianyang
Wang, Yuxin
Liu, Xiangyang
Qiu, Xipeng
Venue: AI Open
Publisher: Elsevier BV
Published: 2022
Volume: 3
Pages: 111-132
ISSN: 2666-6510 (ISSN-L)
DOI: 10.1016/j.aiopen.2022.10.001
OpenAlex: W3171649327
Cited by: 1,422
References: 183
Citations by year: 2026: 217 · 2025: 423 · 2024: 438 · 2023: 264 · 2022: 69
FWCI: 98.23
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
creativecommons.org/licenses/by/4.0/ (vor, 284 d)
Locations: AI Open (publishedVersion, cc-by)
arXiv (Cornell University) (submittedVersion)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
arXiv (Cornell University)
Funders: National Natural Science Foundation of China (62022027)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords: transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A survey of transformers

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks