Cited by 1,422Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 181 of the 183 references it cites were found in a registry or an index.

A survey of transformers

Open access

Lin, Tianyang, Wang, Yuxin, Liu, Xiangyang, Qiu, Xipeng·2022·AI Open

DOI 10.1016/j.aiopen.2022.10.001

Open this paper in CoChat — free account Open PDF

Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1016/j.aiopen.2022.10.001

Cites

181 of 181 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285

Flagged first, then by citations

Show all 183 references

2 of the 183 references couldn’t be matched to a known paper.

Sorted by citations

Items 601–700 of 1,001

Toward Explainable AI in Spacecraft Health Monitoring: Comparative Benchmarking of Anomaly Detection Models and Open-Source Datasets
Tahir, H., Jabbarpour, M. R., Vo, Q. B., Kowalczyk, R., Barr, J., & Bessell, T. (2025) · IEEE Access · doi:10.1109/access.2025.3639551 · Cited by 7
Toward AI foundation models for epidemics: Promise, challenges, and paths forward
Lau, M. S. Y., Metcalf, C. J. E., Liu, Z., Grenfell, B. T., & Jin, W. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2526192123 · Cited by 10
Artificial intelligence with metasurfaces: from intelligent design to intelligent computing
Ma, Q., Feng, Z. R., Hou, J., Liu, C., Ning, Y. M., Li, R. S., Wu, Q. W., Huang, Y. S., Zhang, Y., Gu, Z., et al. (2026) · PhotoniX · doi:10.1186/s43074-026-00239-1 · Cited by 8
TCF-Trans: Temporal Context Fusion Transformer for Anomaly Detection in Time Series
Peng, X., Li, H., Lin, Y., Chen, Y., Fan, P., & Lin, Z. (2023) · Sensors · doi:10.3390/s23208508 · Cited by 7
LIME-Explained Small-Scale Tabular Transformer Used for Improving the Classification Performance of Multi-Category Causes of Death in Colorectal Cancer
Yu, N., Deng, F., Lin, Y., & Zhao, L. (2023) · International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) · doi:10.1109/iciibms60103.2023.10347787 · Cited by 6
RM-GPT: Enhance the comprehensive generative ability of molecular GPT model via LocalRNN and RealFormer
Fan, W., He, Y., & Zhu, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102827 · Cited by 7
Merging two cultures: Deep and statistical learning
Bhadra, A., Datta, J., Polson, N. G., Sokolov, V., & Xu, J. (2024) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1647 · Cited by 13
Anticipating Job Market Demands—A Deep Learning Approach to Determining the Future Readiness of Professional Skills
Weichselbraun, A., Süsstrunk, N., Waldvogel, R., Glatzl, A., Brașoveanu, A., & Scharl, A. (2024) · Future Internet · doi:10.3390/fi16050144 · Cited by 10
Transformer Encoder-Based Deep Learning Model for Time Series Prediction
Mandal, U. (2024) · Advances in environmental engineering and green technologies book series · doi:10.4018/979-8-3693-2351-9.ch013 · Cited by 6
Investigation of alternative attention modules in transformer models for remaining useful life predictions: addressing challenges in high-frequency time-series data
Boos, E., Zimmermann, J., Wiemer, H., & Ihlenfeldt, S. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.01.012 · Cited by 6
SW-YOLOX: A YOLOX-based real-time pedestrian detector with shift window-mixed attention mechanism
Tsai, C., Wang, R.-Y., & Chiu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128357 · Cited by 7
AI-driven transcriptomic encoders: From explainable models to accurate, sample-independent cancer diagnostics
Croce, D., Smirnov, A., Tiburzi, L., Travaglini, S., Costa, R. M. D. S., Calabrese, A., Basili, R., Ghiron, N. L., & Melino, G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125126 · Cited by 6
Energy Efficient FPGA-Based Accelerator for Dynamic Sparse Transformer
Li, Z., Lai, Y., & Zhang, H. (2024) · International Conference on Communications, Circuits and Systems · doi:10.1109/icccas62034.2024.10652850 · Cited by 7
Deep Learning for Generating Phase-Conditioned Infrared Spectra
Na, G. S. (2024) · Analytical Chemistry · doi:10.1021/acs.analchem.4c04786 · Cited by 6
Fusing Attention and Convolution: A Hybrid Model for Brain Stroke Prediction
Bhuvanya, R., Kujani, T., & Sivakumar, K. (2024) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7022 · Cited by 6
Charge Diagnostics and State Estimation of Battery Energy Storage Systems Through Transformer Models
Zequera, R. A. G., Rassõlkin, A., Vaimann, T., & Kallaste, A. (2025) · IEEE Access · doi:10.1109/access.2025.3532858 · Cited by 6
Neural Optimisation of Fixed Beamformers With Flexible Geometric Constraints
Yan, L., Huang, W., Abhayapala, T. D., Feng, J., & Kleijn, W. B. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533372 · Cited by 6
Integration of FBG Reflection and Transmission Spectra for Sensing Capacity Enhancement With Transformer CNN Demodulation
Arockiyadoss, M. A., Yao, C.-K., Kumar, P., Chung, Y.-J., Dehnaw, A. M., & Peng, P. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3571065 · Cited by 6
A Mechanical Fault Diagnosis Method for On-Load Tap Changers Based on GOA-Optimized FMD and Transformer
Wei, R., Chen, Z., Wang, Q., Yongsheng, D., Wang, H., Jiang, F., Liu, D., & Wang, X. (2025) · Energies · doi:10.3390/en18143848 · Cited by 6
Generating Rational Commonsense Knowledge-Aware Dialogue Responses With Channel-Aware Knowledge Fusing Network
Wu, S., Li, Y., Zhang, D., & Wu, Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3199649 · Cited by 6
TC-Fuse: A Transformers Fusing CNNs Network for Medical Image Segmentation
Geng, P., Lu, J., Zhang, Y., Ma, S., Tang, Z., & Liu, J. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.027127 · Cited by 8
Dynamic Spatio-Temporal Graph Fusion Convolutional Network for Urban Traffic Prediction
Ma, H., Qin, X., Jia, Y., & Zhou, J. (2023) · Applied Sciences · doi:10.3390/app13169304 · Cited by 6
Broader and Deeper: A Multi-Features with Latent Relations BERT Knowledge Tracing Model
Li, Z., Jacobsen, M., Shi, L., Zhou, Y., & Wang, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42682-7_13 · Cited by 7
Change Detection Needs Neighborhood Interaction in Transformer
Ma, H., Zhao, L., Li, B., Niu, R., & Wang, Y. (2023) · Remote Sensing · doi:10.3390/rs15235459 · Cited by 6
Self-citation
Generative AI: Impactful Considerations to Responsible Data Practices in Business Execution
Osmëni, T., & Ali, M. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384863 · Cited by 8
Generative AI for the Maritime Environments
Reddy, G. P., Sinha, S., & Park, S. (2024) · International Conference on Ubiquitous and Future Networks · doi:10.1109/icufn61752.2024.10625100 · Cited by 8
Multi-Layer Attention-Based Explainability via Transformers for Tabular Data
Gavito, A. T., Klabjan, D., & Utke, J. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00010 · Cited by 6
Automated Line Labelling: Dataset for Contour Detection and 3D Reconstruction
Santhanam, H., Doiphode, N., & Shi, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00315 · Cited by 6
Cicero: An AI-Based Writing Assistant for Legal Users
Luzi, F. D., Macrì, M., Mecella, M., & Mencattini, T. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-34674-3_13 · Cited by 6
Personality Detection on Reddit Using DistilBERT
Julianda, A. R., & Maharani, W. (2023) · Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi) · doi:10.29207/resti.v7i5.5236 · Cited by 6
Deep Learning for Precipitation Nowcasting: A Survey from the Perspective of Time Series ForecastingPreprint
An, S., Oh, T.-J., Sohn, E.-H., & Kim, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4888852 · Cited by 6
HpT: Hybrid Acceleration of Spatio-Temporal Attention Model Training on Heterogeneous Manycore Architectures
Dahal, S., Dhingra, P., Thapa, K. K., Pande, P. P., & Kalyanaraman, A. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3522781 · Cited by 7
LLMs and Their Applications in Medical Artificial Intelligence
Mao, W., Qiu, X., & Abbasi, A. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3711837 · Cited by 5
Self-citation
Mai: A Transformer Based Domain Specific Chatbot for Menstrual Health
Mughal, S. F., Aamir, S., Samad, A., Zehra, U., & Syed, A. A. (2025) · ACM Journal on Responsible Computing · doi:10.1145/3711711 · Cited by 10
SpaceFusion: Advanced Deep Learning Operator Fusion via Space-Mapping Graph
Zhu, L., Yao, J., & Guan, H. (2025) · European Conference on Computer Systems · doi:10.1145/3689031.3696087 · Cited by 5
Explainable transformer-based deep survival analysis in childhood acute lymphoblastic leukemia
Cui, Y., Dong, W., Li, Y., Janitz, A. E., Pokala, H. R., & Zhu, R. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110118 · Cited by 5
Corporate AI play and short term skill-biased AI change
Bughin, J. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.102904 · Cited by 7
Multi-domain fusion network: A novel approach to OCTA image segmentation in diabetic retinopathy
Jia, G., Ma, F., Li, S., Zhang, Z., Liu, H., Guo, Y., & Meng, J. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107945 · Cited by 5
Transformer-based AI for Sentiment Analysis in Marketing
Petratos, P. N., & Giannoula, M. (2024) · Asia Conference on Machine Learning and Computing (ACMLC) · doi:10.1145/3690771.3690795 · Cited by 5
Empowering cardiovascular diagnostics with SET-MobileNet: A lightweight and accurate deep learning based classification approach
Safdar, Z., Sheng, J., Saeed, M. U., Ramzan, M., & Al‐Zubaidi, A. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105684 · Cited by 5
Fringe projection profilometry with lightweight transformer-CNN network for high-dynamic-range scenes
Cao, J., Wang, X., Zhang, X., & Tu, D. (2025) · Measurement Science and Technology · doi:10.1088/1361-6501/ae00e3 · Cited by 5
Premenstrual dysphoric disorder in online peer support communities: a Reddit case study
Evkoski, B., Letina, S., Novak, P. K., & Riddell, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-19220-2 · Cited by 5
Transformer-based detection of abnormal rice growth using drone-based multispectral imaging
Li, Y., Dang, L. M., Wang, H., Fayaz, M., Danish, S., Shang, J., Song, H., & Moon, H. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111055 · Cited by 5
Decomposition Based Interference Management Framework for Local 6G Networks
Gunarathne, S., Sivalingam, T., Mahmood, N. H., Rajatheva, N., & Latva‐aho, M. (2023) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshps58843.2023.10464770 · Cited by 5
Combining traditional and spiking neural networks for energy-efficient detection of Eimeria parasites
Vázquez, I., Ayasi, B., Şeker, H., Luengo, J., Sedano, J., & García-Vico, Á. M. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111681 · Cited by 6
Enhancing Type 2 Diabetes Treatment Decisions With Interpretable Machine Learning Models for Predicting Hemoglobin A1c Changes: Machine Learning Model Development
Kurasawa, H., Waki, K., Seki, T., Chiba, A., Fujino, A., Hayashi, K., Nakahara, E., Haga, T., Noguchi, T., & Ohe, K. (2024) · JMIR AI · doi:10.2196/56700 · Cited by 5
Towards human-leveled vision systems
Ding, J., & Huang, T. (2024) · Science China Technological Sciences · doi:10.1007/s11431-024-2762-5 · Cited by 5
Drone image recognition and intelligent power distribution network equipment fault detection based on the transformer model and transfer learning
Zhong, J., Chen, Y., Gao, J., & Lv, X. (2024) · Frontiers in Energy Research · doi:10.3389/fenrg.2024.1364445 · Cited by 5
An efficient joint training model for monaural noisy-reverberant speech recognition
Lian, X., Xia, N., Dai, G., & Yang, H. (2024) · Applied Acoustics · doi:10.1016/j.apacoust.2024.110322 · Cited by 5
Data fusion of sparse, heterogeneous, and mobile sensor devices using adaptive distance attention
Lepioufle, J.-M., Schneider, P., Hamer, P., Ødegård, R. Å., Vallejo, I., Cao, T. V., Taherkordi, A., & Wojcikowski, M. (2024) · Environmental Data Science · doi:10.1017/eds.2024.18 · Cited by 8
Protecting Vehicle Location Privacy with Contextually-Driven Synthetic Location Generation
Yadav, S., Yu, C., Xie, X., Huang, Y., & Qiu, C. (2024) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3678717.3691211 · Cited by 5
Weird-Net: Weighted Relative Distance Attention for Efficient and Robust Sequence Processing
Hu, L., Zhan, F., Huang, W., Gan, W., Hu, H., He, H., & Han, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3513912 · Cited by 5
A Hybrid Model of State-Space Model and Attention for Hyperspectral Image Denoising
Shao, M., Tan, X., Shang, K., Liu, T., & Cao, X. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3556024 · Cited by 5
Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition
Aloysius, N., Geetha, M., & Nedungadi, P. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3564828 · Cited by 6
Transformers and Long Short-Term Memory Transfer Learning for GenIV Reactor Temperature Time Series Forecasting
Pantopoulou, S., Cilliers, A. C., Tsoukalas, L. H., & Heifetz, A. (2025) · Energies · doi:10.3390/en18092286 · Cited by 5
Sign Language Recognition—Dataset Cleaning for Robust Word Classification in a Landmark-Based Approach
Antonowicz, P., Kasperek, D., & Podpora, M. (2025) · IEEE Access · doi:10.1109/access.2025.3566338 · Cited by 5
Forecasting of exchange rate time series based on event-aware transformer mode
Zhang, S., Che, T., Zhu, Z., Luo, G., & Feng, P. F. (2025) · Soft Computing · doi:10.1007/s00500-025-10558-3 · Cited by 6
Settlement early warning method for high speed railway subgrades based on TD Transformer
Kebing, W., & Qinghuai, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-05067-0 · Cited by 5
A comparative study of deep learning approaches for stock price prediction
Vuong, P. H., Phu, L. H., Nguyen, T. H. V., Duy, L. N., Bao, P. T., & Trinh, T. D. (2025) · Digital Finance · doi:10.1007/s42521-025-00149-0 · Cited by 5
FullTransNet: Full Transformer With Local-Global Attention for Video Summarization
Lan, L., Jiang, L., Yu, T., Liu, X., & He, Z. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3634718 · Cited by 5
Self-citation
Deep Learning Waveform Channel Modeling for Wideband Optical Fiber Transmission: Model Comparisons, Challenges and Potential Solutions
Shi, M., Yang, H., Niu, Z., Zeng, C., Zhang, Y., Xiao, J., Chen, M., Hu, W., & Yi, L. (2026) · Journal of Lightwave Technology · doi:10.1109/jlt.2026.3659990 · Cited by 5
Transformers in Machine Learning: Literature Review
Thoyyibah, T., Haryono, W., Zailani, A. U., Djaksana, Y. M., Rosmawarni, N., & Arianti, N. D. (2023) · Jurnal Penelitian Pendidikan IPA · doi:10.29303/jppipa.v9i9.5040 · Cited by 5
NBGuru: Generating Explorable Data Science Flowcharts to Facilitate Asynchronous Communication in Interdisciplinary Data Science Teams
Keelawat, P. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3607020 · Cited by 7
A review of in-memory computing for machine learning: architectures, options
Snåšel, V., Dang, T. K., Kueng, J., & Kong, L. (2023) · International Journal of Web Information Systems · doi:10.1108/ijwis-08-2023-0131 · Cited by 6
MLLCD: A Meta Learning-based Method for Lung Cancer Diagnosis Using Histopathology Images
Hu, X., Wang, S., Li, H., & Zhang, Q. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385566 · Cited by 5
Innovative approaches to high school physics competitions: Harnessing the power of AI and open science
Borovský, D., Hanč, J., & Hančová, M. (2024) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2715/1/012011 · Cited by 5
A Hybrid ViT-CNN Model Premeditated for Rice Leaf Disease Identification
Pavithra, P., & Baranidharan, B. (2024) · International Journal of Computational Methods and Experimental Measurements · doi:10.18280/ijcmem.120104 · Cited by 5
What Is Generative in Generative Artificial Intelligence? A Design-Based Perspective
Bordas, A., Masson, P. L., Thomas, M., & Weil, B. (2024) · Springer eBooks · doi:10.1007/978-3-031-71922-6_8 · Cited by 6
A Rapid Detection Method for Coal Ash Content in Tailings Suspension Based on Absorption Spectra and Deep Feature Extraction
Zhu, W., Zhang, X., Zhu, Z., Fu, W., Liu, N., & Zhang, Z. (2024) · Mathematics · doi:10.3390/math12111685 · Cited by 5
Predictability and Causality in Spanish and English Natural Language Generation
Busto-Castiñeira, A., González-Castaño, F. J., García-Méndez, S., & de Arriba-Pérez, F. (2024) · IEEE Access · doi:10.1109/access.2024.3420710 · Cited by 7
Assessment of few-hits machine learning classification algorithms for low-energy physics in liquid argon detectors
Moretti, R., Rossi, M., Biassoni, M., Giachero, A., Grossi, M., Guffanti, D., Labranca, D., Terranova, F., & Vallecorsa, S. (2024) · The European Physical Journal Plus · doi:10.1140/epjp/s13360-024-05287-9 · Cited by 5
Vision Transformers Inference Acceleration Based on Adaptive Layer Normalization
Keddous, F. E., Llanza, A., Shvai, N., & Nakib, A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2024.128524 · Cited by 9
Neural Machine Translation for Low-Resource Languages: Experiments on Makassar-Indonesian
Afra, D. I. N., Uliniansyah, M. T., Latief, A. D., Yuyun, Y., Nurfadhilah, E., Gunarso, G., & Sampurno, T. (2024) · International Conference on Computer, Control, Informatics and its Applications · doi:10.1109/ic3ina64086.2024.10732202 · Cited by 6
Understanding Big Data in Neurosurgery
Beheshti, A., Alinejad-Rokny, H., Molina, E. S., & Ieva, A. D. (2024) · Advances in experimental medicine and biology · doi:10.1007/978-3-031-64892-2_10 · Cited by 5
Dual-Path Beat Tracking: Combining Temporal Convolutional Networks and Transformers in Parallel
Thapa, N., & Lee, J. (2024) · Applied Sciences · doi:10.3390/app142411777 · Cited by 5
A novel residual block: replace Conv1× 1 with Conv3×3 and stack more convolutions
Hu, X., Sheng, G., Zhang, D., & Li, L. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1302 · Cited by 5
TI-former: A Time-Interval Prediction Transformer for Timestamped Sequences
Ryu, H., Yu, S., & Lee, K. Y. (2023) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera57763.2023.10197830 · Cited by 5
Lightweight Scheme to Capture Stock Market Sentiment on Social Media Using Sparse Attention Mechanism: A Case Study on Twitter
Wu, S., & Gu, F. (2023) · Journal of risk and financial management · doi:10.3390/jrfm16100440 · Cited by 7
Evolving masked low-rank transformer for long text understanding
Liu, C., Chen, X., Lin, J., Hu, P., Wang, J., & Geng, X. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.111207 · Cited by 5
MS-Transformer: Introduce multiple structural priors into a unified transformer for encoding sentences
Le, Q., Zhang, Y., Yin, Q., & Liu, T. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101304 · Cited by 5
NLP as a Lens for Causal Analysis and Perception Mining to Infer Mental Health on Social MediaPreprint
Garg, M., Saxena, C., Naseem, U., Sohn, S., & Dorr, B. J. (2023) · doi:10.36227/techrxiv.21972974.v1 · Cited by 5
Survey on Multimodal Transformers for RobotsPreprint
Miyazawa, K., & Nagai, T. (2023) · doi:10.36227/techrxiv.21993317 · Cited by 5
Short-term vital parameter forecasting in the intensive care unit: A benchmark study leveraging data from patients after cardiothoracic surgery
Hinrichs, N., Roeschl, T., Lanmüeller, P., Balzer, F., Eickhoff, C., O’Brien, B., Falk, V., & Meyer, A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000598 · Cited by 4
Pre-trained Language Models for Decoding Protein Language: a Survey
Amjad, M., Munir, A., Zia, U., & Paracha, R. Z. (2024) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt262145.2024.10740205 · Cited by 4
On the Use of Embedding Techniques for Modeling User Navigational Behavior in Intelligent Prefetching Strategies
Büyüktanır, T., & Aktaş, M. S. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8356 · Cited by 4
Transformer-Based Power Optimization for Max-Min Fairness in Cell-Free Massive MIMO
Chafaa, I., Bacci, G., & Sanguinetti, L. (2025) · IEEE Wireless Communications Letters · doi:10.1109/lwc.2025.3568336 · Cited by 4
High-Performance Prompting for LLM Extraction of Compression Fracture Findings from Radiology Reports
Kanani, M. M., Monawer, A. H., Brown, L., King, W. E., Miller, Z. D., Venugopal, N., Heagerty, P. J., Jarvik, J. G., Cohen, T., & Cross, N. M. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01530-6 · Cited by 4
Deep Learning-Driven Models for Analyzing Numerical Control Power Signals and Surface Characteristics in Sustainable Milling of Additively Manufactured Al-Si Alloys
Gupta, M. K., Ross, N. S., Shibi, C. S., Chudy, R., Niesłony, P., Korkmaz, M. E., Królczyk, J. B., & Królczyk, G. (2025) · International Journal of Precision Engineering and Manufacturing-Green Technology · doi:10.1007/s40684-025-00759-4 · Cited by 4
An effective measure for evaluating the environmental impact of AI tasks
Masciari, E., & Napolitano, E. V. (2025) · Computing · doi:10.1007/s00607-025-01512-0 · Cited by 5
SFT-GAN: Sparse Fast Transformer Fusion Method Based on GAN for Remote Sensing Spatiotemporal Fusion
Ma, Z., Bao, W., Feng, W., Zhang, X., Ma, X., & Qu, K. (2025) · Remote Sensing · doi:10.3390/rs17132315 · Cited by 4
The physical-encoded photovoltaic forecasting method combined with continuous learning and multi-digital twins mechanisms
Liu, S., Tian, J., Dai, Y., Ji, Z., & Banerjee, A. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126390 · Cited by 4
Tourist Flow Prediction Based on GA-ACO-BP Neural Network Model
Yang, X., Cheng, Y., Dong, M., & Xie, X. (2025) · Informatics · doi:10.3390/informatics12030089 · Cited by 4
Behind the Magic, MERLIM: Multi-Modal Evaluation Benchmark for Large Image-Language Models
Villa, A., Alcázar, J. L., Soto, Á., & Ghanem, B. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00054 · Cited by 5
Magnetic resonance image processing transformer for general accelerated image restoration
Shen, G., Li, M., Anderson, S. W., Farris, C. W., & Zhang, X. (2025) · Scientific Reports · doi:10.1038/s41598-025-23851-w · Cited by 5
Deep learning and digital twin integration for indoor environmental conditions (IEC): A state-of-the-art review
Haghighat, S., & Zhang, R. (2025) · Indoor Environments · doi:10.1016/j.indenv.2025.100137 · Cited by 5
Revolutionizing predictive maintenance: Remaining useful life forecasting based on cognitive artificial intelligence using knowledge growing system
Syamsiana, I. N., Febriani, N. A., Sumari, A. D. W., & Sutjipto, R. (2025) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2025.103880 · Cited by 4
DTG: Dual transformers-based generative adversarial networks for retinal 2D/3D OCT image classification
Hammou, B. A., Duval, R., Boucher, M. C., & Farida Cheriet, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103915 · Cited by 4
Symbolic distillation of a fully tokenized transformer for settlement prediction in pre-bored grouted planted nodular piles in stratified soils
La, H., & Nguyen, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.115186 · Cited by 4
Mapping the technological evolution of generative AI: a patent network analysis
Mohammadi, N., Dahooie, J. H., Bengari, A. A., & Rahimi, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-26810-7 · Cited by 5
Time-series InSAR monitoring and evolution predictive analysis of filling-area subsidence in Yan’an New District
Chen, K., Zhang, B., Jiang, H., Lv, J., Liu, A. K., Jiang, Y., & Zhang, R. (2026) · Advances in Space Research · doi:10.1016/j.asr.2026.01.069 · Cited by 4

Authors: Lin, T., Wang, Y., Liu, X., & Qiu, X.
Affiliations: Fudan University (CN)
Corresponding author: Qiu, Xipeng
ORCID: Lin, Tianyang
Wang, Yuxin
Liu, Xiangyang
Qiu, Xipeng
Venue: AI Open
Publisher: Elsevier BV
Published: 2022
Volume: 3
Pages: 111-132
ISSN: 2666-6510 (ISSN-L)
DOI: 10.1016/j.aiopen.2022.10.001
OpenAlex: W3171649327
Cited by: 1,422
References: 183
Citations by year: 2026: 217 · 2025: 423 · 2024: 438 · 2023: 264 · 2022: 69
FWCI: 98.23
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
www.elsevier.com/legal/tdmrep-license (tdm)
creativecommons.org/licenses/by/4.0/ (vor, 284 d)
Locations: AI Open (publishedVersion, cc-by)
arXiv (Cornell University) (submittedVersion)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
arXiv (Cornell University)
Funders: National Natural Science Foundation of China (62022027)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords: transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A survey of transformers

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks