DeepNet: Scaling Transformers to 1,000 Layers
Open accessThe DOI matches its Crossref record; 66 of the 68 references it cites were found in a registry or an index.
Abstract
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, makingDeepNorma preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate thatDeepNethas superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction. Our code is available athttps://aka.ms/torchscale.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, they add a scaling factor to the residual stream or adjust the placement of normalization within the residual [12,122,123].”
From A Survey of Large Language Models (2026) · cites this paper as [123]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Motivated by the above observation, we introduce a new normalization function (D EEP N ORM ) at residual connections [22], which has theoretical justification of bounding the model update by a constant.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“R ECENT years have witnessed a trend towards large-scale Transformer [1] models.”
BackgroundMethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Our analysis is based on SGD update, and we empirically verify it works well for Adam optimizer [29].”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The capacity has substantially increased from millions of parameters [2], [3] to billions [4], [5], [6], [7], [8], [9], [10], [11], and even trillions [12], [13].”
BackgroundMethodHighly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 4×
“Transformers have achieved success across many fields, including machine translation [1], [61], language modelling [2], [4], [5], [45], speech recognition [62], vision pre-training [31], [53], [54] and vision-language pre-training [63].”
BackgroundMethodInfluential(2010) · Cited by 14,134
Research works printed but not linked: 2
- Wang, Magneto: A foundation transformer, 36077
- Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
Cited by
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
“Compared with LN, DeepNorm [157] has shown a better capability to ensure the stability in training, which has been adopted by GLM-130B with post normalization.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214
“Transformers have grown larger [5] and deeper [87], but equipping them with longer context remains difficult [83], since the self-attention module at their heart has time and memory complexity quadratic in sequence length.”
BackgroundLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
Self-citationScaling neural machine translation to 200 languages
NLLB Team, Costa‐jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K. S., Kalbassi, E., Lam, J., Licht, D. J., et al. (2024) · Nature · doi:10.1038/s41586-024-07335-x · Cited by 495
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Chen, B., Cheng, X., Li, P., Geng, Y.-A., Gong, J., Li, S., Bei, Z., Tan, X., Wang, B., Zeng, X., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.05.547496 · Cited by 107
“We adopt the Post-LN initialized with the DeepNorm [59].”
MethodE-Branchformer: Branchformer with Enhanced Merging for Speech Recognition
Kim, K., Wu, F., Peng, Y., Pan, J. S., Sridhar, P., Han, K. J., & Watanabe, S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022656 · Cited by 219
“Transformers and their variants have been further studied to reduce computation cost [21, 22, 23, 24, 25] or to train deep models stably [26].”
Transformer in reinforcement learning for decision-making: a survey
Yuan, W., Chen, J., Chen, S., Lu, L., Hu, Z., Li, P., Feng, D., Liu, F., & Chen, J. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300548 · Cited by 127
CellFM: a large-scale foundation model pre-trained on transcriptomics of 100 million human cells
Zeng, Y., Xie, J., Shangguan, N., Wei, Z., Li, W., Su, Y., Yang, S., Zhang, C., Zhang, J., Fang, N., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59926-5 · Cited by 156
“To address this gap, CellFM employs the new post-norm normalization method DeepNorm 56 .”
InfluentialShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., & Chen, W. (2024) · arXiv · doi:10.48550/arxiv.2403.03853 · Cited by 60
Chen, B., Cheng, X., Li, P., Geng, Y., Gong, J., Li, S., Bei, Z., Tan, X., Wang, B., Zeng, X., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02636-z · Cited by 240
“We adopt the Post-LN initialized with the DeepNorm 48 .”
Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages
Heffernan, K. S., Çelebi, O., & Schwenk, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.154 · Cited by 158
“Training NMT models for a large number of languages faces many challenges and a large variety of architectures have been explored (Ma et al., 2021; Wang et al., 2022; Escolano et al., 2021).”
BackgroundNie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 118
Migratable urban street scene sensing method based on vision language pre-trained model
Zhang, Y., Zhang, F., & Chen, N. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102989 · Cited by 55
Learning From Partially Labeled Data for Multi-Organ and Tumor Segmentation
Xie, Y., Zhang, J., Xia, Y., & Shen, C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3312587 · Cited by 56
“Specifically, wefollowthedeepnormstrategy[48]tostabilizethedeep Transformers.”
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
“Finally, normalization strategies, including pre-layer normalization (Baevski & Auli, 2019) and DeepNorm (Wang et al., 2024a), optimize network architecture by stabilizing training.”
Mixture of Attention Heads: Selecting Attention Heads Per Token
Zhang, X., Shen, Y., Huang, Z., Zhou, J., Rong, W., & Zhang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.278 · Cited by 101
Xue, Q., Song, Y., Wu, H., Cheng, Y., & Pan, H. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1309594 · Cited by 29
Training Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 32
“Modifications were made to the standard Transformer architecture [82], adopting a DeepNorm [85] strategy and layer normalization [5].”
Adaptive Cross Transformer With Contrastive Learning for Surface Defect Detection
Huang, X., Li, Y., Bao, Y., & Zheng, W. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3470998 · Cited by 32
“Generally, increasing the parameter scale or adding deep structures in the Transformer framework enhances the generalization capability [16].”
BackgroundTransDetector: A Transformer-Based Detector for Underwater Acoustic Differential OFDM Communications
Li, Y., Wang, S., Liu, D., Zhou, C., & Gui, Z. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3367179 · Cited by 21
“Wang et al. [29] proposed a DeepNorm that combines the well performance of Post-LN and the stable training of Pre-LN.”
BackgroundMethodJoint Holistic and Masked Face Recognition
Zhu, Y., Ren, M., Jing, H., Dai, L., Sun, Z., & Li, P. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3280717 · Cited by 26
“Based on the position of normalization layers, a Trans-former block has three structural alternatives as shown in Fig.1, namely PreNorm [19], Res-PostNorm [72], and Deep-Norm [73].”
MethodPan, J.-S., Ting, Y.-S., & Yu, J. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae068 · Cited by 28
Khotimah, W. N., Bennamoun, M., Boussaïd, F., Xu, L., Edwards, D., & Sohel, F. A. (2023) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2023.103286 · Cited by 18
Transformers with Learnable Activation Functions
Fang, H., Lee, J.-U., Moosavi, N. S., & Gurevych, I. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.181 · Cited by 27
Neural Machine Translation for Low-Resource Languages from a Chinese-centric Perspective: A Survey
Zhang, J., Su, K., Li, H., Mao, J., Tian, Y., Wen, F., Guo, C., & Matsumoto, T. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3665244 · Cited by 29
Koishekenov, Y., Berard, A., & Nikoulina, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.198 · Cited by 72
MultiTrans: Multi-branch transformer network for medical image segmentation
Zhang, Y., Balestra, G., Zhang, K., Wang, J., Rosati, S., & Giannini, V. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108280 · Cited by 18
MindLLM: Lightweight large language model pre-training, evaluation and domain application
Yang, Y., Sun, H., Li, J., Liu, R., Li, Y., Liu, Y., Gao, Y., & Huang, H. (2024) · AI Open · doi:10.1016/j.aiopen.2024.08.001 · Cited by 18
PrCRS: a prediction model of severe CRS in CAR-T therapy based on transfer learning
Wei, Z., Zhao, C., Zhang, M., Xu, J., Xu, N., Wu, S., Xin, X., Yu, L., & Feng, W. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05804-8 · Cited by 12
“Moreover, Deep-Net [29] recently proposed incorporating untrained rule-based scaling into the skip connection to stabilize preLN in Transformers.”
BackgroundTian, Z., Cao, X., Zeng, B., & Guan, X. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.125840 · Cited by 11
Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation
Zhang, S., Liang, Y., Wang, S., Chen, Y., Han, W., Liu, J., & Xu, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.448 · Cited by 16
StageInteractor: Query-based Object Detector with Cross-stage Interaction
Teng, Y., Liu, H., Guo, S., & Wang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00605 · Cited by 17
“In this paper, we improve query-based object detector from a new perspective of the architecture scalability [66, 46, 55]: we reuse dynamic operators among decoder layers to capture”
MethodT-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
Duquenne, P.-A., Gong, H., Sagot, B., & Schwenk, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.391 · Cited by 33
Underwater Dam Crack Image Classification Algorithm Based on Improved VanillaNet
Zhu, S., Li, X., Wan, G., Wang, H., Shao, S., & Shi, P. (2024) · Symmetry · doi:10.3390/sym16070845 · Cited by 14
Self-citationA multi-modal deep language model for contaminant removal from metagenome-assembled genomes
Zou, B., Wang, J., Ding, Y., Zhang, Z., Huang, Y., Fang, X., Cheung, K. C., See, S., & Zhang, L. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00908-5 · Cited by 9
One‐Fit‐All Transformer for Multimodal Geophysical Inversion: Method and Application
Jiang, Y., Ma, J., Jie-Yuan, N. I. N. G., Li, J., Wu, H., & Bao, T. (2025) · Journal of Geophysical Research Machine Learning and Computation · doi:10.1029/2024jh000432 · Cited by 14
MASTER: A Multi-modal Foundation Model for Human Activity Recognition
Zhu, G., Zhao, D. Y., Li, C., Zhao, M., Zhang, Z., Quan, H., & Ma, H. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749511 · Cited by 16
“To make training more stable, we adopt the post-norm and deep-norm architecture [42], as the pre-norm architecture encounters convergence issues during training when the number of transformer layers increases.”
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism
Zhang, J., Tan, M., Dai, P., & Zhu, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.43 · Cited by 23
“With the rapid development and wide industrial applications, researchers have devoted great effect in manually designing neural netoweks (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; He et al., 2016; Huang et al., 2017; Wang et al., 2022).”
BackgroundYuan, H. (2025) · iRadiology · doi:10.1002/ird3.115 · Cited by 13
“Second, its architecture exhibits exceptional scalability, supporting models with over 100 billion parameters to capture intricate linguistic relationships in human language [5].”
Integration AI Techniques in Low-Resource Language: The Case of Kazakh Language
Asmaganbetova, K., Auyeskhan, U., Zarina, N., Alen, S., & Yeskazina, M. (2024) · IEEE AITU: Digital Generation · doi:10.1109/ieeeconf61558.2024.10585350 · Cited by 13
“Following that, evaluations are conducted with several other cutting-edge models such as Deepnet [18] and M2M-100 [20], which provide performance metrics for a subset of 87 languages included in Flores-10.”
MethodMultivariate Time-Series Missing Data Imputation with Convolutional Transformer Model
Wang, Y., He, D., & Li, H. (2025) · Symmetry · doi:10.3390/sym17050686 · Cited by 8
B2T Connection: Serving Stability and Performance in Deep Transformers
Takase, S., Kiyono, S., Kobayashi, S., & Suzuki, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.192 · Cited by 42
“However, Wang et al. (2022) demonstrated that the training of previous methods except DeepNet failed in a much deeper configuration than normally used, i.e., 100L-100L.”
MethodInfluentialExploring All-In-One Knowledge Distillation Framework for Neural Machine Translation
Miao, Z., Zhang, W., Su, J., Li, X., Luan, J., Chen, Y., Wang, B., & Zhang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.178 · Cited by 13
“To further improve model performance, there have been many efforts in exploring wider or deeper Transformers (Wang et al., 2019; Xu et al., 2021; Wang et al., 2022).”
EFT6D: An efficient fusion transformer network for 6D object pose estimation
Zhang, M., Xie, T., Zang, X., Zang, X., Zhao, J., & Zhang, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129035 · Cited by 6
How is language intelligence evolving? A multi-dimensional survey of large language models
Chen, X., Zhou, L., Chen, J., Wang, G., & Li, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130637 · Cited by 11
BiG2S: A dual task graph-to-sequence model for the end-to-end template-free reaction prediction
Hu, H., Jiang, Y., Yang, Y. Z., & Chen, J. X. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05048-8 · Cited by 7
“BiG2S introduces a special initialization method with the normalization of each layer in Transformer which are based on DeepNet [30].”
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Lu, P., Kobyzev, I., Rezagholizadeh, M., Rashid, A., Ghodsi, A., & Langlais, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.363 · Cited by 8
Awad, M. A., Kadhum, A. M., Waheeb, A. S., Kyhoiesh, H. A. K., Elsalam, H. E. A., & Azab, I. H. E. (2025) · RSC Advances · doi:10.1039/d5ra06776f · Cited by 6
Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation
Maillard, J., Gao, C., Kalbassi, E., Sadagopan, K. R., Goswami, V., Koehn, P., Fan, A., & Guzmán, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.154 · Cited by 53
“State of the art machine translation models are able to cover hundreds of languages (Ma et al., 2021; Wang et al., 2022; Siddhant et al., 2022; NLLB Team et al., 2022) by relying on large amounts of annotated (Skadin , š et al., 2014; Lison and Tiede-mann, 2016; Agi´c and Vuli´c, 2019) and…”
BackgroundCollaborative local–global context modeling for session-based recommendation
Li, W., Chen, B., Gao, M. Y., An, J., Dong, H., Chen, C., Fan, W., & Zhu, Z. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104196 · Cited by 6
RePaIR: Repaired pruning at initialization resilience
Zhao, H., Guan, R., Man, K. L., Yu, L., & Yue, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107086 · Cited by 6
Deep Sparse Conformer for Speech Recognition
Wu, X. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10384 · Cited by 4
“(11) In order to ensure the model updating in DeepNet to be constrained by bounds for stable training, α is assigned with various values alike (2N) 1 4 for encoder-only framework (e.g., BERT [25]), and (2M) 1 4 for decoder-only framework (e.g., GPT [22]) where N and M respectively stand for the number of f(·), such as…”
MethodInfluentialThe Xiaomi Text-to-Text Simultaneous Speech Translation System for IWSLT 2022
Guo, B., Liu, M., Zhang, W., Chen, H., Mu, C., Li, X., Cui, J., Wang, B., & Guo, Y. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.17 · Cited by 4
Cong, A., Jin, Y., Yi, H., Jiang, Y., Li, J., Wang, S., & Hou, L. (2026) · Nonlinear Dynamics · doi:10.1007/s11071-025-12204-x · Cited by 4
Adaptive gated meta graph retention network: A model for urban traffic flow prediction
Li, X., & Bao, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129703 · Cited by 5
MUNI-NLP Systems for Low-resource Indic Machine Translation
Signoroni, E., & Rychlý, P. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.91 · Cited by 5
“Apart from the baseline, we tested bigger and deeper models, inspired by work such as (Narang et al., 2021; Wei et al., 2022; Wang et al., 2022), on mni-eng , which we considered as the "easiest" direction for the models.”
MethodRevisiting Pre-training of Embedding Layers in Transformer-based Neural Machine Translation
Neishi, M., & Yoshinaga, N. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.534 · Cited by 2
Coformer for session-based recommendation with dual positional information
Li, W., Zhu, Z., Dong, H., Chen, C., Gao, M. Y., & Fan, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127732 · Cited by 2
Jeong, S. I., Jeong, M. S., Jeon, E. S., & Park, K. R. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3585188 · Cited by 2
“In general, stacking more layers and enlarging parameters improve performance in deep learning [74], [75].”
A multimodal deep learning network for precise detection of estrus and pseudo-estrus in sows
Duan, Y., Yang, Y., Cao, Y., Liu, J., Li, L., Cao, R., Hu, G., & Liu, Z. (2025) · Smart Agricultural Technology · doi:10.1016/j.atech.2025.101279 · Cited by 2
A Modular Perspective on the Evolution of Deep Learning: Paradigm Shifts and Contributions to AI
Wei, Y., Wang, Y., & Watada, J. (2025) · Applied Sciences · doi:10.3390/app151910539 · Cited by 3
MobileNMT: Enabling Translation in 15MB and 30ms
Lin, Y., Wang, X., Zhang, Z., Wang, M., Xiao, T., & Zhu, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.36 · Cited by 9
Scaling Law for Document Neural Machine Translation
Zhuocheng, Z., Gu, S., Zhang, M., & Feng, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.556 · Cited by 10
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing
Liu, P., Gao, Z., Chen, Y., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.920 · Cited by 11
IOL Research Machine Translation Systems for WMT23 General Machine Translation Shared Task
Zhang, W. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.19 · Cited by 2
“We chose Transformer(Vaswani et al., 2017) as our base translation model and used both pre-norm and deep-norm(Wang et al., 2022) variants to help us train deeper models.”
MethodMUNI-NLP Submission for Czech-Ukrainian Translation Task at WMT23
Rychlý, P., & Teslia, Y. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.14 · Cited by 2
“We use the DeepNorm (Wang et al., 2022) modi-fication of the transformer (Vaswani et al., 2017) architecture in the TorchScale library (Ma et al., 2022).”
MethodBinVPR: Binary Neural Networks towards Real-Valued for Visual Place Recognition
Wang, J., Han, J., Dong, R., & Kan, J. (2024) · Sensors · doi:10.3390/s24134130 · Cited by 1
Fusing AI: Multimodal Language Models Inference Across Diverse Inputs
Jovanovic, M., & Campbell, M. (2024) · Computer · doi:10.1109/mc.2024.3445515 · Cited by 1
Danayi, A., & Hamid Soltanian-zadeh, A. (2024) · International Symposium on Telecommunications · doi:10.1109/ist64061.2024.10843655 · Cited by 1
Toward Effective Knowledge Distillation: Navigating Beyond Small-Data Pitfall
Hao, Z., Guo, J., Han, K., Hu, H., Xu, C., & Wang, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3607982 · Cited by 4
“Scaling up datasets [28], [41], model sizes [11], [42], [43], and training iterations [38] has consistently resulted in performance improvements across various tasks.”
Cong, A., Wang, S., Jin, Y., Yi, H., Chang, Z., Jiang, Y., Hou, L., & Saeed, N. A. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133915 · Cited by 1
Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
Duquenne, P.-A., Schwenk, H., & Sagot, B. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2484 · Cited by 2
“multilingual supervised models, M2M100 [19] and Deepnet [20].”
MethodPepper Leaf Diagnosis Using Deep-Net with Low-Dimensional Image Classification
Rababa, L., Ali, N., Al-essa, R., & Alzu’bi, A. (2023) · International Conference on Information and Communication Systems (ICICS) · doi:10.1109/icics60529.2023.10330492 · Cited by 1
“In this pepper, we improved the efficiency of Pepper leaf image detection using fine-tuned deep learning architecture, DeepNet [5], with a PCA dimensionality-reduction approach.”
MethodDual-stream perception cross-flattening transformer for few-shot surface defect detection
LI, Y., Wang, S., Jing, Z., Zheng, J., Han, X. B., Zheng, X., & Sun, F. (2025) · Information Sciences · doi:10.1016/j.ins.2025.123017 · Cited by 2
USST’s System for AutoSimTrans 2022
Hui, Z., & Jun, Y. L. (2022) · doi:10.18653/v1/2022.autosimtrans-1.7 · Cited by 1
“In our system, we train a Transformer (Vaswani et al., 2017) with a deep encoder (Meng et al., 2020) as baseline for abtaining rich source representations, besides we initialize the model with the method mentioned in DeepNet (Wang et al., 2022) in order to stabilize the training of the deeper model.”
MethodIntegration of protein and coding sequences enables mutual augmentation of the language modelPreprint
Zhao, H.-R., Cheng, M., Jinhua, Z., Wang, H., Yang, X.-R., Wang, B., Sun, Y., Fang, M.-H., Chen, E., Li, H., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.24.620004 · Cited by 1
Self-citationResidual Matrix Transformers: Scaling the Size of the Residual Stream
Mak, B., & Flanigan, J. (2025) · arXiv · doi:10.48550/arxiv.2506.22696 · Cited by 1
“…that modifications that involve residual stream scaling, normalization, and weight initialization to improve moment propagation (Kedia et al., 2024, Wang et al., 2024, Shleifer et al., 2021, Huang et al., 2020, Bachlechner et al., 2020) are orthogonal to our approach because they could be extended…”
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“Beyond merely adjusting the position of LayerNorm, researchers have developed DeepNorm [23], which combines a tailored parameter initialization strategy with modified residual connections to stabilize training.”
TbsNet: the importance of thin-branch structures in CNNs
Hu, X., Sheng, G., Shi, P., & Ding, Y. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1429
Peer Review #3 of "TbsNet: the importance of thin-branch structures in CNNs (v0.2)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.2/reviews/3
Peer Review #3 of "TbsNet: the importance of thin-branch structures in CNNs (v0.3)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.3/reviews/3
Peer Review #1 of "TbsNet: the importance of thin-branch structures in CNNs (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.1/reviews/1
Peer Review #1 of "TbsNet: the importance of thin-branch structures in CNNs (v0.2)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.2/reviews/1
Peer Review #3 of "TbsNet: the importance of thin-branch structures in CNNs (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.1/reviews/3
Peer Review #2 of "TbsNet: the importance of thin-branch structures in CNNs (v0.1)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.1/reviews/2
Peer Review #2 of "TbsNet: the importance of thin-branch structures in CNNs (v0.3)"
(2023) · PeerJ Computer Science · doi:10.7287/peerj-cs.1429v0.3/reviews/2
Multi-Path Transformer is Better: A Case Study on Neural Machine Translation
Lin, Y., Zhou, S., Li, Y., Ma, A., Xiao, T., & Zhu, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.414 · Cited by 2
Implicit regularization of deep residual networks towards neural ODEsPreprint
Marion, P., Wu, Y., Sander, M. E., & Gérard Biau, A. (2023) · HAL · doi:10.48550/arxiv.2309.01213 · Cited by 14
Enhancing In-vehicle Multi-turn Dialogue Text Rewriting with Lightweight Text FilteringPreprint
Xue, P., Tan, L., Lin, L., & She, K. (2023) · Research Square · doi:10.21203/rs.3.rs-3714877/v1
Huang, S., Feng, C., Shi, G., Li, Z., Zhao, X., Li, X., & Wang, X. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3661305 · Cited by 1
“Drawing inspiration from research on MoE [27] and deep Transformer architectures [17, 38], we have observed that especially as the number of domains increases, the combination of different transformer encoder and decoder sub-layers plays a crucial role in adapting different domains within the…”
MethodSelf-citationLightweight Neural Data Sequence Modeling by Scale Causal Blocks
Akutsu, H., & Arai, K. (2024) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7074
“Also, research on very deep Transformer methods is progressing, and it is now possible to construct large-scale models [24].”
Foundations of Generative Information Retrieval
Ai, Q., Zhan, J., & Liu, Y. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_2 · Cited by 5
Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
Xu, S., Dong, H., Zhang, C., & Wang, C. (2025) · Applied Sciences · doi:10.3390/app15084155 · Cited by 1
Passban, P., Rezagholizadeh, M., & Way, A. (2025) · Machine translation · doi:10.1007/978-3-031-85747-8_1
Chen, W., Liu, Z., He, B., & Wang, R. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105490 · Cited by 1
A Multi-Scale Perception Feature Fusion Method for Self-Supervised Monocular Depth Estimation
Meng, C., & Yu, C. (2025) · doi:10.1109/icnlp65360.2025.11108431
“A multiscale monocular depth estimate based on a graph convolutional network, comprising two DepthNet[5], was designed by Masoumian et al[6].”
Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
Zhang, Z., Lin, P., Wang, Z., Zhang, Y., & Xu, Z. J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3646483 · Cited by 18
“Recent studies also investigate the impact of initialization on the training process of LLMs [66], [67], [68], [69], [70], [71].”
Kang, Q., Guo, Y., Hu, B., Li, X., Yang, Q., Yang, Y., Lu, D., Sun, H., Xu, R., Xu, Z., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.28.696777 · Cited by 1
“To further stabilize deep training, we combine Layer Normalization with DeepNorm-style scaling [54], which helps control gradient magnitudes and mitigates vanishing or exploding gradients in deep Transformer stacks.”
Self-citationDuoTransFormer for nonlinear seismic response prediction: Cover large, focus local, and enforce law
Hu, X., Xiao, C., Zhang, Z., & Gao, J. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104375
Metadata
- Authors
- Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F.
- Affiliations
- Chinese Academy of Sciences (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-10
- Volume
- 46
- Issue
- 10
- Pages
- 6761-6774
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4394698525
- Cited by
- 113
- References
- 68
- Citations by year
- 2026: 21 · 2025: 27 · 2024: 23
- FWCI
- 10.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- layer normalization, initialization, training stability, model scaling, multilingual machine translation, BEiT, language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.