Cited by 1,422Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 181 of the 183 references it cites were found in a registry or an index.

A survey of transformers

Open access

Lin, Tianyang, Wang, Yuxin, Liu, Xiangyang, Qiu, Xipeng·2022·AI Open

DOI 10.1016/j.aiopen.2022.10.001

Open this paper in CoChat — free account Open PDF

Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.aiopen.2022.10.001

Cites

181 of 181 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285

Flagged first, then by citations

Show all 183 references

2 of the 183 references couldn’t be matched to a known paper.

Sorted by citations

Items 201–300 of 1,001

A survey on deep learning fundamentals
Tian, C., Cheng, T., Peng, Z., Zuo, W., Tian, Y., Zhang, Q., Wang, F., & Zhang, D. D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11368-7 · Cited by 46
Knowledge Inheritance for Pre-trained Language Models
Qin, Y., Lin, Y., Jing, Y., Zhang, J., Han, X., Zhang, Z., Su, Y., Liu, Z., Li, P., Sun, M., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.288 · Cited by 37
A comparative assessment of deep learning models for day-ahead load forecasting: Investigating key accuracy drivers
Pelekis, S., Seisopoulos, I.-K., Spiliotis, E., Pountridis, T., Karakolis, E., Mouzakitis, S., & Askounis, D. (2023) · Sustainable Energy Grids and Networks · doi:10.1016/j.segan.2023.101171 · Cited by 42
Position-Aware Relational Transformer for Knowledge Graph Embedding
Li, G., Sun, Z., Hu, W. P., Cheng, G., & Qu, Y. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3262937 · Cited by 33
An Overview of Vision Transformers for Image Processing: A Survey
Kameswari, C. S., Kavitha, J., Reddy, T. S., Chinthaguntla, B., Jagatheesaperumal, S. K., Gaftandzhieva, S. N., & Doneva, R. Z. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140830 · Cited by 38
Multi-hop Question Answering
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 119
Navigating the frontier of drug-like chemical space with cutting-edge generative AI models
Lavecchia, A. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104133 · Cited by 40
A Lightweight and Efficient Distracted Driver Detection Model Fusing Convolutional Neural Network and Vision Transformer
Li, Z., Zhao, X., Wu, F., Chen, D., & Wang, C. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3447041 · Cited by 32
CAF-ViT: A cross-attention based Transformer network for underwater acoustic target recognition
Dong, W., Fu, J. W., Zou, N., Zhao, C., Miao, Y., & Shen, Z. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.120049 · Cited by 32
Explainable artificial intelligence based intelligent fault diagnosis: A systematic review from applications to insights
Li, T., Chen, J., Liu, T., Sun, C., Zhao, Z., Chen, X., & Yan, R. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.111935 · Cited by 44
'My agent understands me better': Integrating Dynamic Human-like Memory Recall and Consolidation in LLM-Based Agents
Hou, Y., Tamoto, H., & Miyashita, H. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650839 · Cited by 31
Semantic Segmentation of Land Cover in Urban Areas by Fusing Multisource Satellite Image Time Series
Yan, J., Liu, J., Liang, D., Wang, Y., Li, J., & Wang, L. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3329709 · Cited by 31
Self-citation
ISAC Receiver Design: A Learning-Based Two-Stage Joint Data-and-Target Parameter Estimation
Hu, J., Valiulahi, I., & Masouros, C. (2024) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2024.3402398 · Cited by 31
Citation prediction by leveraging transformers and natural language processing heuristics
Buscaldi, D., Dessı̀, D., Motta, E., Murgia, M., Osborne, F., & Recupero, D. R. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103583 · Cited by 34
ProkBERT family: genomic language models for microbiome applications
Ligeti, B., Szepesi-Nagy, I., Bodnár, B., Ligeti-Nagy, N., & Juhász, J. (2024) · Frontiers in Microbiology · doi:10.3389/fmicb.2023.1331233 · Cited by 30
MaskCAE: Masked Convolutional AutoEncoder via Sensor Data Reconstruction for Self-Supervised Human Activity Recognition
Cheng, D., Zhang, L., Qin, L., Wang, S., Wu, H., & Song, A. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3373019 · Cited by 31
DeePathNet: A Transformer-Based Deep Learning Model Integrating Multiomic Data with Cancer Pathways
Cai, Z., Poulos, R. C., Aref, A., Robinson, P. J., Reddel, R. R., & Zhong, Q. (2024) · Cancer Research Communications · doi:10.1158/2767-9764.crc-24-0285 · Cited by 36
Deep Representation Learning: Fundamentals, Technologies, Applications, and Open Challenges
Payandeh, A., Baghaei, K. T., Fayyazsanavi, P., Ramezani, S. B., Chen, Z., & Rahimi, S. (2023) · IEEE Access · doi:10.1109/access.2023.3335196 · Cited by 49
A time series anomaly detection method based on series-parallel transformers with spatial and temporal association discrepancies
Fu, S., Gao, X., Zhai, F., Li, B., Xue, B., Yu, J., Meng, Z., & Zhang, G. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119978 · Cited by 32
Diving into the technology: a systematic literature review on strategic use of chatbots in hospitality service encounters
Sam, S. J. I., & Jasim, K. M. (2023) · Management Review Quarterly · doi:10.1007/s11301-023-00388-8 · Cited by 37
Harnessing Generative Pre-Trained Transformers for Construction Accident Prediction with Saliency Visualization
Yoo, B., Kim, J., Park, S., Ahn, C. R., & Oh, T. K. (2024) · Applied Sciences · doi:10.3390/app14020664 · Cited by 37
A Comprehensive Survey of Recent Transformers in Image, Video and Diffusion Models
Le, D. P. C., Wang, D., & Le, V.-T. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.050790 · Cited by 36
Deep Learning in Heart Sound Analysis: From Techniques to Clinical Applications
Zhao, Q., Geng, S., Wang, B., Sun, Y., Nie, W., Bai, B., Yu, C., Zhang, F., Tang, G., Zhang, D., et al. (2024) · Health Data Science · doi:10.34133/hds.0182 · Cited by 43
A Stethoscope for Drones: Transformers-Based Methods for UAVs Acoustic Anomaly Detection
Anidjar, O. H., Barak, A., Moshe, B. B., Hagai, E., & Tuvyahu, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262702 · Cited by 34
Art authentication with vision transformers
Schaerf, L., Postma, E. O., & Popovici, C. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08864-8 · Cited by 35
Data Augmentation Using Transformers and Similarity Measures for Improving Arabic Text Classification
Refai, D., Abu-Soud, S. M., & Abdel‐Rahman, M. J. (2023) · IEEE Access · doi:10.1109/access.2023.3336311 · Cited by 29
An ensemble deep learning approach for air quality estimation in Delhi, India
Mohan, A. S., & Abraham, L. (2024) · Earth Science Informatics · doi:10.1007/s12145-023-01210-5 · Cited by 31
Fine-Grained Multilingual Hate Speech Detection Using Explainable AI and Transformers
Siddiqui, J. A., Yuhaniz, S. S., Mujtaba, G., Soomro, S. A., & Mahar, Z. A. (2024) · IEEE Access · doi:10.1109/access.2024.3470901 · Cited by 33
Recent Advances in Deep Learning for Channel Coding: A Survey
Matsumine, T., & Ochiai, H. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3472094 · Cited by 39
Intelligent Cinematography: a review of AI research for cinematographic production
Azzarelli, A., Anantrasirichai, N., & Bull, D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11089-3 · Cited by 48
LegalBert-pt: A Pretrained Language Model for the Brazilian Portuguese Legal Domain
Silveira, R., Ponte, C., Almeida, V. M. S. S. D., Pinheiro, V. C. M., & Furtado, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45392-2_18 · Cited by 29
A Very Deep Graph Convolutional Network for13C NMR Chemical Shift Calculations with Density Functional Theory Level Performance for Structure Assignment
Ai, W.-J., Li, J., Cao, D., Liu, S., Yuan, Y.-Y., Li, Y., Tan, G., Xu, K.-P., Yu, X., Kang, F., et al. (2024) · Journal of Natural Products · doi:10.1021/acs.jnatprod.3c00862 · Cited by 28
Hybrid fragment-SMILES tokenization for ADMET prediction in drug discovery
Aksamit, N., Tchagang, A., Li, Y., & Ombuki-Berman, B. M. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05861-z · Cited by 31
Prototype-Optimized unsupervised domain adaptation via dynamic Transformer encoder for sensor drift compensation in electronic nose systems
Sun, J., Zheng, H., Diao, W., Sun, Z., Qi, Z., & Wang, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125444 · Cited by 28
Efficient LLMs Training and Inference: An Introduction
Li, R., Fu, D., Shi, C., Huang, Z., & Lu, G. (2024) · IEEE Access · doi:10.1109/access.2024.3501358 · Cited by 36
Transformers and audio detection tasks: An overview
Zaman, K., Li, K., Şah, M., Direkoğlu, C., Okada, S., & Unoki, M. (2024) · Digital Signal Processing · doi:10.1016/j.dsp.2024.104956 · Cited by 36
Fine-grained crop pest classification based on multi-scale feature fusion and mixed attention mechanisms
Qian, Y., Xiao, Z., & Deng, Z. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1500571 · Cited by 31
Transforming Surgical Training With AI Techniques for Training, Assessment, and Evaluation: Scoping Review
Escobar-Castillejos, D., Barrera-Animas, A. Y., Noguez, J., Magana, A. J., & Beneš, B. (2025) · Journal of Medical Internet Research · doi:10.2196/58966 · Cited by 31
Soft Sensing Transformer: Hundreds of Sensors are Worth a Single Word
Zhang, C., Yella, J., Huang, Y., Qian, X., Петров, С. А., Rzhetsky, A., & Bom, S. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671925 · Cited by 28
Bard, ChatGPT and 3DGPT: a scientometric analysis of generative AI tools and assessment of implications for mechanical engineering education
Mustapha, K. B., Mustapha, K. B., Yap, E. H., & Abakr, Y. A. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-10-2023-0198 · Cited by 33
Comparing Fine-Tuning, Zero and Few-Shot Strategies with Large Language Models in Hate Speech Detection in English
Pan, R., García-Díaz, J. A., & Valencia‐García, R. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.049631 · Cited by 34
Can transformers transform financial forecasting?
Souto, H. G., & Moradi, A. (2024) · China Finance Review International · doi:10.1108/cfri-01-2024-0032 · Cited by 30
Emission analysis based on mixed traffic flow and license plate recognition model
Wu, S., Sun, D., & Qiu, G. (2024) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2024.104331 · Cited by 27
ERG-AI: enhancing occupational ergonomics with uncertainty-aware ML and LLM feedback
Sen, S., Gonzalez-Sanchez, V. E., Husom, E. J., Tverdal, S., Tokas, S., & Tjøsvoll, S. O. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05796-1 · Cited by 32
Transformer-Based hyperspectral image analysis for phenotyping drought tolerance in blueberries
Rahman, M. H., Busby, S., Ru, S., Hanif, S., Sanz‐Sáez, Á., Zheng, J., & Rehman, T. U. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109684 · Cited by 28
Evaluating Machine Learning-Based Soft Sensors for Effluent Quality Prediction in Wastewater Treatment Under Variable Weather Conditions
Voipan, D., Voipan, A. E., & Barbu, M. (2025) · Sensors · doi:10.3390/s25061692 · Cited by 27
radarODE: An ODE-Embedded Deep Learning Model for Contactless ECG Reconstruction From Millimeter-Wave Radar
Zhang, Y., Guan, R., Li, L., Yang, R., Yue, Y., & Lim, E. G. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3563945 · Cited by 27
Object Detection Based on CNN and Vision‐Transformer: A Survey
Cao, J., Peng, B., Gao, M., Hao, H., Li, X., & Mou, H. (2025) · IET Computer Vision · doi:10.1049/cvi2.70028 · Cited by 37
E-Mamba: Efficient Mamba network for hyperspectral and LiDAR joint classification
Zhang, Y., Gao, H., Chen, Z., Zhang, C., Ghamisi, P., & Zhang, B. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103649 · Cited by 27
Beyond Word Embeddings: Heterogeneous Prior Knowledge Driven Multi-Label Image Classification
Deng, X., Feng, S., Lyu, G., Wang, T., & Lang, C. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3171095 · Cited by 27
STT-Net: Simplified Temporal Transformer for Emotion Recognition
Khan, M., Saddik, A. E., Deriche, M. A., & Gueaieb, W. (2024) · IEEE Access · doi:10.1109/access.2024.3413136 · Cited by 29
State of Health Estimation of Lithium-Ion Batteries Using Data Augmentation and Feature Mapping
Yao, W., Lai, R., Tian, Y., Li, X., & Tian, J. (2024) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2024.3471867 · Cited by 26
A survey of text classification based on pre-trained language model
Wu, Y., & Wan, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128921 · Cited by 43
FlightScope: An Experimental Comparative Review of Aircraft Detection Algorithms in Satellite Imagery
Ghazouali, S. E., Gucciardi, A., Venturini, F., Venturi, N., Rueegsegger, M., & Michelucci, U. (2024) · Remote Sensing · doi:10.3390/rs16244715 · Cited by 33
Human–robot interaction through joint robot planning with large language models
Asuzu, K., Singh, H., & Idrissi, M. (2025) · Intelligent Service Robotics · doi:10.1007/s11370-024-00570-1 · Cited by 33
Large Language Model-Augmented Learning for Auto-Delineation of Treatment Targets in Head-and-Neck Cancer Radiotherapy
Rajendran, P., Yang, Y., Niedermayr, T. R., Gensheimer, M. F., Beadle, B. M., Le, Q., Xing, L., & Dai, X. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2025.110740 · Cited by 31
STMHCpan, an accurate Star-Transformer-based extensible framework for predicting MHC I allele binding peptides
Ye, Z., Li, S., Mi, X., Shao, B., Dai, Z., Ding, B., Feng, S.-W., Sun, B., Shen, Y., & Xiao, Z.-D. (2023) · Briefings Bioinform. · doi:10.1093/bib/bbad164 · Cited by 23
Nucleic Transformer: Classifying DNA Sequences with Self-Attention and Convolutions
He, S., Gao, B., Sabnis, R., & Sun, Q. (2023) · ACS Synthetic Biology · doi:10.1021/acssynbio.3c00154 · Cited by 25
What Formal Languages Can Transformers Express? A Survey
Strobl, L., Merrill, W., Weiss, G., Chiang, D., & Angluin, D. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00663 · Cited by 91
MOSS: An Open Conversational Large Language Model
Sun, T., Zhang, X., He, Z., Li, P., Cheng, Q., Liu, X., Yan, H., Shao, Y., Tang, Q., Zhang, S., et al. (2024) · Machine Intelligence Research · doi:10.1007/s11633-024-1502-8 · Cited by 48
Self-citation
Artificial general intelligence for the upstream geoenergy industry: A review
Li, J. X., Zhang, T., Zhu, Y., & Chen, Z. (2024) · Gas Science and Engineering · doi:10.1016/j.jgsce.2024.205469 · Cited by 38
YOLO-DroneMS: Multi-Scale Object Detection Network for Unmanned Aerial Vehicle (UAV) Images
Zhao, X., & Chen, Y. (2024) · Drones · doi:10.3390/drones8110609 · Cited by 25
Accurate Detection and Evaluation of the Airgap Asymmetry Fault in DS-PMSLM Based on OSVT and ECA-ENet
Song, J., Cao, J., Zhao, J., Wang, L., Wu, X., Wang, X., & Lu, S. (2024) · IEEE Transactions on Transportation Electrification · doi:10.1109/tte.2024.3495980 · Cited by 25
Spiking-PhysFormer: Camera-based remote photoplethysmography with parallel spike-driven transformer
Liu, M., Tang, J., Chen, Y., Li, H., Li, H., Li, S., Wang, K., Gan, J., Wang, Y., & Chen, H. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107128 · Cited by 25
Self-citation
A time series forecasting method for oil production based on Informer optimized by Bayesian optimization and the hyperband algorithm (BOHB)
Deng, W., Xin, X., Song, R., Yang, X., Wang, W., & Yu, G. (2025) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2025.109068 · Cited by 26
VM-UNet++ research on crack image segmentation based on improved VM-UNet
Tang, W., Wu, Z., Wang, W., Pan, Y., & Gan, W. (2025) · Scientific Reports · doi:10.1038/s41598-025-92994-7 · Cited by 25
SciDeBERTa: Learning DeBERTa for Science Technology Documents and Fine-Tuning Information Extraction Tasks
Jeong, Y., & Kim, E. (2022) · IEEE Access · doi:10.1109/access.2022.3180830 · Cited by 31
Memristor-Based Light-Weight Transformer Circuit Implementation for Speech Recognizing
Xiao, H., Zhou, Y., Gao, T., Duan, S., Chen, G., & Hu, X. (2023) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2023.3237582 · Cited by 25
Evolution of Deep Learning-Based Sequential Recommender Systems: From Current Trends to New Perspectives
Yoon, J. H., & Jang, B. (2023) · IEEE Access · doi:10.1109/access.2023.3281981 · Cited by 30
Emerging artificial intelligence applications: metaverse, IoT, cybersecurity, healthcare - an overview
Sharma, N. V., & Jindal, N. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17890-6 · Cited by 34
Advancing Additive Manufacturing Through Machine Learning Techniques: A State-of-the-Art Review
Xiao, S., Li, J., Wang, Z., Chen, Y., & Tofighi, S. (2024) · Future Internet · doi:10.3390/fi16110419 · Cited by 31
DHCT-GAN: Improving EEG Signal Quality with a Dual-Branch Hybrid CNN–Transformer Network
Cai, Y., Zhao, M., & Huang, D. (2025) · Sensors · doi:10.3390/s25010231 · Cited by 31
PCTNet: depth estimation from single structured light image with a parallel CNN-transformer network
Zhu, X., Han, Z., Zhang, Z., Song, L., Wang, H., & Guo, Q. (2023) · Measurement Science and Technology · doi:10.1088/1361-6501/acd136 · Cited by 24
Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Models
Li, W., Hu, A., Xu, N., & He, G. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3350661 · Cited by 30
LEViT- Leaf Disease identification and classification using an enhanced Vision transformers(ViT) model
Prashanthi, B., Krishna, A. V. P., & Rao, C. M. (2024) · Multimedia tools and applications · doi:10.1007/s11042-024-19866-6 · Cited by 29
Multi-category sorting of plastic waste using Swin Transformer: A vision-based approach
Wang, Z., Ye, L., Chen, F., Zhou, T., & Zhao, Y. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.122742 · Cited by 28
Deep models for multi-view 3D object recognition: a review
Alzahrani, M. S., Usman, M., Jarraya, S. K., Anwar, S., & Helmy, T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10941-w · Cited by 25
A Review and Comparative Study of Explainable Deep Learning Models Applied on Action Recognition in Real Time
Mahmoudi, S. A., Amel, O., Stassin, S., Liagre, M., Benkedadra, M., & Mancaş, M. (2023) · Electronics · doi:10.3390/electronics12092027 · Cited by 24
SymTC: A symbiotic Transformer-CNN net for instance segmentation of lumbar spine MRI
Chen, J., Qian, L., Ma, L., Urakov, T. M., Gu, W., & Liang, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108795 · Cited by 28
DTformer: An Efficient Digital Twin Model for Loss Measurement in UHVDC Transmission Systems
Sun, Y., Shi, Y., Hu, Q., Xie, C., & Su, T. (2023) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2023.3278300 · Cited by 23
Early Diagnosis of Schizophrenia in EEG Signals Using One Dimensional Transformer Model
Shoeibi, A., Jafari, M., Sadeghi, D., Alizadehsani, R., Alinejad‐Rokny, H., Beheshti, A., & Górriz, J. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61140-7_14 · Cited by 22
Generative Self-Supervised Learning With Spectral-Spatial Masking for Hyperspectral Target Detection
Chen, X., Zhang, Y., Dong, Y., & Du, B. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3423781 · Cited by 23
A transformer-based deep reinforcement learning approach for dynamic parallel machine scheduling problem with family setups
Li, F., Lang, S., Tian, Y., Hong, B., Rolf, B., Noortwyck, R., Schulz, R., & Reggelin, T. (2024) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-024-02470-8 · Cited by 25
Retrieve and rerank for automated ICD coding via Contrastive Learning
Niu, K., Wu, Y., Li, Y., & Li, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104396 · Cited by 22
Episode-level prediction of freezing of gait based on wearable inertial signals using a deep neural network model
Huang, D., Wu, C., Wang, Y., Zhang, Z., Chen, C., Li, L., Zhang, W., Zhang, Z., Li, J., Guo, Y., et al. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105613 · Cited by 21
Self-citation
Spam detection for Youtube video comments using machine learning approaches
Xiao, A. S., & Liang, Q. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100550 · Cited by 22
A Multi-Scale Deep Learning Algorithm for Enhanced Forest Fire Danger Prediction Using Remote Sensing Images
Yang, J., Jiang, H., Wang, S., & Ma, X. (2024) · Forests · doi:10.3390/f15091581 · Cited by 26
ELAKT: Enhancing Locality for Attentive Knowledge Tracing
Pu, Y., Liu, F., Shi, R., Yuan, H., Chen, R., Peng, T., & Wu, W. (2024) · ACM Transactions on Information Systems · doi:10.1145/3652601 · Cited by 27
Self-supervised transformers for turbulent flow time series
Drikakis, D., Kokkinakis, I. W., Fung, D., & Spottswood, S. M. (2024) · Physics of Fluids · doi:10.1063/5.0214468 · Cited by 21
Boosting grape bunch detection in RGB-D images using zero-shot annotation with Segment Anything and GroundingDINO
Devanna, R. P., Reina, G., Cheein, F. A., & Milella, A. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109611 · Cited by 21
YOLO-SDW: A method for detecting infection in corn leaves
Yang, H., Sheng, S., Jiang, F., Zhang, T., Wang, S., Xiao, J., Zhang, H., Peng, C., & Wang, Q. (2024) · Energy Reports · doi:10.1016/j.egyr.2024.11.072 · Cited by 21
A Comparative Study of Fine-Tuning Deep Learning Models for Leaf Disease Identification and Classification
Prashanthi, B., Krishna, A. V. P., & Rao, C. M. (2025) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.9017 · Cited by 21
Advances in Remote Sensing and Deep Learning in Coastal Boundary Extraction for Erosion Monitoring
Blais, M.-A., & Akhloufi, M. A. (2025) · Geomatics · doi:10.3390/geomatics5010009 · Cited by 23
Advances in Skeleton-Based Fall Detection in RGB Videos: From Handcrafted to Deep Learning Approaches
Hoang, V.-H., Lee, J. W., Piran, M. J., & Park, C. (2023) · IEEE Access · doi:10.1109/access.2023.3307138 · Cited by 22
A modified transformer and adapter-based transfer learning for fault detection and diagnosis in HVAC systems
Wang, Z., Li, D., Cao, Z.-W., Gao, F., & Li, M.-J. (2024) · Energy Storage and Saving · doi:10.1016/j.enss.2024.02.004 · Cited by 21
Comparison of Transformers with LSTM for classification of the behavioural time budget in horses based on video data
Martin-Cirera, A., Nowak, M., Norton, T., Auer, U., & Oczak, M. (2024) · Biosystems Engineering · doi:10.1016/j.biosystemseng.2024.04.014 · Cited by 22
Large Language Models in Genomics—A Perspective on Personalized Medicine
Ali, S. M., Qadri, Y. A., Ahmad, K., Lin, Z., Leung, M.-F., Kim, S. W., Vasilakos, A. V., & Zhou, T. (2025) · Bioengineering · doi:10.3390/bioengineering12050440 · Cited by 26
Applying machine learning to primate bioacoustics: Review and perspectives
Cauzinille, J., Favre, B., Marxer, R., & Rey, A. (2024) · American Journal of Primatology · doi:10.1002/ajp.23666 · Cited by 29
A Review of the Applications, Benefits, and Challenges of Generative AI for Sustainable Toxicology
Alam, F., Alnazzawi, T. S. M., Mehmood, R., & Almaghthawi, A. (2025) · Current Research in Toxicology · doi:10.1016/j.crtox.2025.100232 · Cited by 25
An overview of transformers for video anomaly detection
Dilek, E., & Dener, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11218-1 · Cited by 28

Authors: Lin, T., Wang, Y., Liu, X., & Qiu, X.
Affiliations: Fudan University (CN)
Corresponding author: Qiu, Xipeng
ORCID: Lin, Tianyang
Wang, Yuxin
Liu, Xiangyang
Qiu, Xipeng
Venue: AI Open
Publisher: Elsevier BV
Published: 2022
Volume: 3
Pages: 111-132
ISSN: 2666-6510 (ISSN-L)
DOI: 10.1016/j.aiopen.2022.10.001
OpenAlex: W3171649327
Cited by: 1,422
References: 183
Citations by year: 2026: 217 · 2025: 423 · 2024: 438 · 2023: 264 · 2022: 69
FWCI: 98.23
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
creativecommons.org/licenses/by/4.0/ (vor, 284 d)
Locations: AI Open (publishedVersion, cc-by)
arXiv (Cornell University) (submittedVersion)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
arXiv (Cornell University)
Funders: National Natural Science Foundation of China (62022027)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords: transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A survey of transformers

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks