DeepNet: Scaling Transformers to 1,000 Layers
Open accessThe DOI matches its Crossref record; 66 of the 68 references it cites were found in a registry or an index.
Abstract
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, makingDeepNorma preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate thatDeepNethas superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction. Our code is available athttps://aka.ms/torchscale.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, they add a scaling factor to the residual stream or adjust the placement of normalization within the residual [12,122,123].”
From A Survey of Large Language Models (2026) · cites this paper as [123]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“Motivated by the above observation, we introduce a new normalization function (D EEP N ORM ) at residual connections [22], which has theoretical justification of bounding the model update by a constant.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“R ECENT years have witnessed a trend towards large-scale Transformer [1] models.”
BackgroundMethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Our analysis is based on SGD update, and we empirically verify it works well for Adam optimizer [29].”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 3×
“The capacity has substantially increased from millions of parameters [2], [3] to billions [4], [5], [6], [7], [8], [9], [10], [11], and even trillions [12], [13].”
BackgroundMethodHighly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 4×
“Transformers have achieved success across many fields, including machine translation [1], [61], language modelling [2], [4], [5], [45], speech recognition [62], vision pre-training [31], [53], [54] and vision-language pre-training [63].”
BackgroundMethodInfluential(2010) · Cited by 14,134
Research works printed but not linked: 2
- Wang, Magneto: A foundation transformer, 36077
- Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
Cited by
Items 101–119 of 119
Entropy-Guided Condensing for Vision Transformer
Lin, S., Lyu, P., Liu, D., Li, Z., Wang, W., Chang, X., & Zheng, Y. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02753-y
“Model scaling (Hou et al., 2025; Shi et al., 2025; Wang et al., 2025, 2024; Yin et al., 2025) for ViTs is a promising way toward the foundation model but remains challenging.”
PLMTox: Low-Rank Adaptation on Protein Language Models for Peptide Toxicity Prediction
Wang, X., Liu, C., Du, W., Song, L., Ren, Z., & Shang, X. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356266
“It uses GeGLU [10] and Rotary Position Embeddings (RoPE) [11] to encode the order of amino acid sequences and DeepNorm [12] for Post-LN initialization.”
Ali, H., Xie, J., & Qiang, T. (2026) · Digital Signal Processing · doi:10.1016/j.dsp.2026.105989
Mahmoudi, M., & Safari, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6344774
Cross-modal progressive modeling for neuro-visual representation learning
Sun, Y., Pu, J., Sun, K., Fu, Z., Duan, H., & Long, Y. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133450
Revisiting Layer Normalization for Point Cloud Test Time Adaptation
Yazdanpanah, M., Bahri, A., Noori, M., Dastani, S., Barbeau, S., Osowiechi, D., Hakim, G. A. V., Ayed, I. B., & Desrosiers, C. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00060 · Cited by 2
“Building on this LN-aware perspective, we develop TTA mechanisms tailored to Transformer backbones on point clouds: methods that are compatible with token-centric inference, avoid dependence on source data, and eliminate backpropagation at test time, while remaining efficient and robust under the…”
FlattenGPT: Depth Compression for Transformer with Layer FlatteningPreprint
Xu, R., Guo, Q., Zhu, Y., Ji, X., Yang, M., & Zhang, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6890079
Learnable foveated sampling and dual-prior reconstruction for single-pixel imaging
Huang, X., Yan, Q., Song, D., Xiong, Z., Fang, Z., Yin, J., Dai, X., & Luo, X. (2026) · Optics Express · doi:10.1364/oe.601620
MonMamba: A Hybrid State Space Model for Mongolian G2P with Linguistic Knowledge Guidance
Dong, W., & Sa, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2856-0_13
Ren, Q.-D.-E.-J., Wang, H., Li, Y., Ji, Y., & Cha, G. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_6
Self-citationLi, W., Gao, M. Y., An, J., Chen, B., Tang, J., Wang, X., & Gong, Y. Y. (2026) · Decision Support Systems · doi:10.1016/j.dss.2026.114740
Tong, J., Chen, G., Huang, X., Huang, J., & Zheng, J. (2026) · Structural Health Monitoring · doi:10.1177/14759217261458991
Spectral Decoupling in Attention Heads: Tracking the Onset of Memorization in Vision Transformers
Jena, D., Roy, S. D., & Abrol, V. (2026) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom71105.2026.11622982
“While a growing body of literature investigates signal propagation and scaling limits in Transformers at extreme depths [12, 13, 14], the specific application of RMT to understand signal routing, particularly the generalization-memorization trade-off remains underexplored [15].”
Liu, Y., Sun, L. C., Zeng, R., Huang, M. C. Y., & Wu, H. X. (2026) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1145/3816440.3818580
Constella: A Novel Framework for Cost-Efficient Distributed AI Inference in LEO Space Data Centers
Stanisic, A., Gravara, M., Herrera, J. L., & Nastić, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35251-4_24
“Modern LEO constellations often comprise thousands of satellites [4], and state-of-the-art DNNs are becoming more complex [24], which further amplifies the combinatorial growth of the prob-Constella lem.”
Deng, X., & Wu, D. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00431
“Probabilistic methods like TACTiS[13] address distributional shifts but remain decoupled from event context, while advances in scaling depth[14], efficiency[15], position embeddings[17], sparse attention[18], higher-order representations[19], and incremental learning[20] improve temporal modeling…”
Multi-Head Normalization for Wide Vision Transformers
Wei, G., Wang, F., Yuille, A., & Chellappa, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37550-6_28
Bounded Autonomy and Verifiable Safety for Agentic AI Enabled Automation
Ramaswamy, S., & Nayak, D. (2026) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-026-02467-w · Cited by 2
MIFARec: Mixed-Interest-Based Fourier Analysis Network for Time-Specific Recommendation
Wang, J., Zeng, J., Zhang, X., Liu, W., Xue, T., & Tang, S. (2026) · Applied Sciences · doi:10.3390/app16199827
Metadata
- Authors
- Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F.
- Affiliations
- Chinese Academy of Sciences (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-10
- Volume
- 46
- Issue
- 10
- Pages
- 6761-6774
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4394698525
- Cited by
- 113
- References
- 68
- Citations by year
- 2026: 21 · 2025: 27 · 2024: 23
- FWCI
- 10.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- layer normalization, initialization, training stability, model scaling, multilingual machine translation, BEiT, language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.