The DOI matches its Crossref record.
Rethinking the Value of Transformer Components
Open accessAbstract
Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures.In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different perspectives.Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others.We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models.Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These heuristics fail to account for the varying functional roles of transformer layers in information processing.
“these heuristics fail to account for two critical aspects: (1) the varying functional roles of transformer layers in information processing (Wang and Tu, 2020; Zhang et al., 2024; Skean et al., 2025)”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Wang and Tu, 2020
The Introduction and Conclusion report that Transformer components contribute differently to information flow and model performance, including differences between attention and feed-forward layers and between layer positions. The paper does not establish that the citing document’s heuristics fail to account for those differences. (LLM, conf=0.82)
Experimental — accuracy not yet measuredDifferent layers of LLMs vary in their importance and process information at different levels of granularity.
“Previous studies (Wang and Tu, 2020; Zhang et al., 2024; Skean et al., 2025) have shown that different layers of LLMs vary in their importance and process information at different levels of granularity.”
From EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025) · cites this paper as Wang and Tu, 2020
The paper’s ‘Several observations on component importance’ section reports that Transformer components at different layers contribute differently to translation performance. It studies machine-translation Transformers, not LLMs, and does not establish that layers process information at different levels of granularity. Page numbers are not available in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 34
- Niladri Chatterji, Behnam Neyshabur, and Hanie Sedghi. 2020. The intriguing role of module criticality in the generalization of deep networks. In ICLR.
- Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. In ACL.
- Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014. Exploiting linear structure within convolutional networks for efficient evaluation. In NeurIPS.
- Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine- Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv.
- Tobias Domhan. 2018. How much attention do you need? a granular analysis of neural machine translation architectures. In ACL.
- Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing Transformer Depth on Demand with Structured Dropout. ICLR.
- Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In ICLR.
- Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In ICML.
- Xinwei Geng, Longyue Wang, Xing Wang, Bing Qin, Ting Liu, and Zhaopeng Tu. 2020. How Does Selective Mechanism Improve Self-Attention Networks? In ACL.
- Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In NeurIPS.
- Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In ICLR.
- Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. In arXiv.
- Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2018. CondenseNet: An Efficient DenseNet using Learned Group Convolutions. In CVPR.
- Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In NAACL.
- Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. 2020. A signal propagation perspective for pruning neural networks at initialization. ArXiv, abs/1906.06307. · arXiv
- Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-head attention with dis- agreement regularization. In EMNLP.
- Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In ACL.
- Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. 2017. Runtime neural pruning. In NeurIPS.
- Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. 2017. The expressive power of neural networks: A view from the width. In NeurIPS.
- Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better than One? In NeurIPS.
- Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In NeurIPS.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
- Siyuan Qiao, Zhe Lin, Jianming Zhang, and Alan Yuille. 2019. Neural rejuvenation: Improving deep network training by enhancing computational resource utilization. In CVPR.
- Alessandro Raganato and Jörg Tiedemann. 2018. An Analysis of Encoder Representations in Transformer-Based Machine Translation. In EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
- Alex Renda, Jonathan Frankle, and Michael Carbin. 2020. Comparing rewinding and fine-tuning in neural network pruning. In ICLR.
- Andrew Michael Saxe, James L McClelland, and Surya Ganguli. 2014. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL.
- Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019a. Encoders help you disambiguate word senses in neural machine translation. In EMNLP.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
- Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL.
- Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019. Exploiting sentential context for neural machine translation. In ACL.
- Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In ACL.
- Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard Baraniuk. 2020. Drawing early-bird tickets: Towards more efficient training of deep networks. ICLR.
- Chiyuan Zhang, Samy Bengio, and Yoram Singer. 2019. Are all layers created equal? ICML Workshop.
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
EvolKV: Evolutionary KV Cache Compression for LLM Inference
Yu, B., & Chai, Y. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.88
Metadata
- Authors
- Wang, W., & Tu, Z.
- Affiliations
- Chinese University of Hong Kong (HK)
- Venue
- Proceedings of the 28th International Conference on Computational Linguistics
- Published
- 2020
- Pages
- 6019-6029
- ISSN
- 1525-2477 (ISSN-L)
- OpenAlex
- W3113709932
- Cited by
- 39
- References
- 0
- Citations by year
- 2025: 7 · 2024: 9 · 2023: 11 · 2022: 7 · 2021: 3 · 2020: 2
- FWCI
- 2.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- transformer models, machine translation, training strategies, model architecture design, model interpretability, training efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
