Roles and Utilization of Attention Heads in Transformer-based Neural Language Models
Open accessAbstract
Sentence encoders based on the transformer architecture have shown promising results on various natural language tasks.The main impetus lies in the pre-trained neural language models that capture long-range dependencies among words, owing to multi-head attention that is unique in the architecture.However, little is known for how linguistic properties are processed, represented, and utilized for downstream tasks among hundreds of attention heads inside the pre-trained transformerbased model.For the initial goal of examining the roles of attention heads in handling a set of linguistic features, we conducted a set of experiments with ten probing tasks and three downstream tasks on four pre-trained transformer families (GPT, GPT2, BERT, and ELECTRA).Meaningful insights are shown through the lens of heat map visualization and utilized to propose a relatively simple sentence representation method that takes advantage of most influential attention heads, resulting in additional performance improvements on the downstream tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep layers encode semantic information and distal relationships within sentences.
“Previous research (Vig & Belinkov, 2019;Jo & Myaeng, 2020;Geva et al., 2021;Wan et al., 2022), which has shown that deep layers encode semantic information and distal relationships within sentences.”
From From Imitation to Introspection: Probing Self-Consciousness in Language Models (2025) · cites this paper as Jo & Myaeng, 2020
Section 4.3 reports that attention heads in upper layers are mostly activated for two tasks the authors describe as semantic. Section 6.2 qualifies this pattern: heads and layers handle both syntactic and semantic information, and the highest layers can contribute less in large models. The cited paper does not establish the claim about distal relationships within sentences. (LLM, conf=0.86)
Experimental — accuracy not yet measuredSemantic information tends to activate deeper layers in transformer models.
“As highlighted by Jo & Myaeng (2020), semantic information tends to activate deeper layers in transformer models.”
From From Imitation to Introspection: Probing Self-Consciousness in Language Models (2025) · cites this paper as Jo & Myaeng (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Jain, S., & Wallace, B. C. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1357 · Cited by 2,287 · see citing works
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,514 · see citing works
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1198 · Cited by 572 · see citing works
Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06316 · Cited by 480 · see citing works
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2gscircifffd · Cited by 477 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Dissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · see citing works
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 374 · see citing works
What do Neural Machine Translation Models Learn about Morphology?
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., & Glass, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1080 · Cited by 307 · see citing works
Does String-Based Neural MT Learn Source Syntax?
Shi, X., Padhi, I., & Knight, K. K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1159 · Cited by 302 · see citing works
An Analysis of Encoder Representations in Transformer-Based Machine Translation
Raganato, A., & Tiedemann, J. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5431 · Cited by 277 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y. (2024) · arXiv (Cornell University) · doi:10.57702/tcr06vve · Cited by 228 · see citing works
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., & Glass, J. R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016309 · Cited by 127 · see citing works
On Identifiability in TransformersPreprint
Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., & Wattenhofer, R. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04211 · Cited by 122 · see citing works
Investigating Language Universal and Specific Properties in Word Embeddings
Qian, P., Qiu, X., & Huang, X. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1140 · Cited by 42 · see citing works
Analyzing Linguistic Knowledge in Sequential Model of Sentence
Qian, P., Qiu, X., & Huang, X. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1079 · Cited by 40 · see citing works
On the Validity of Self-Attention as Explanation in Transformer ModelsPreprint
Brunner, G., Liu, Y., Pascual, D., Richter, O., & Wattenhofer, R. P. (2019) · arXiv (Cornell University) · Cited by 13 · see citing works
Flagged first, then by citations
4 of the 35 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Hamidreza Ghader and Christof Monz. 2017. What does attention in neural machine translation pay at- tention to? In Proceedings of the Eighth Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 30-39, Taipei, Taiwan. Asian Federation of Natural Lan- guage Processing.
- Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected con- volutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
Cited by
Sorted by citations
Zhang, T., Hasib, M. M., Chiu, Y., Han, Z.-F., Jin, Y.-F., Flores, M. A., Chen, Y., & Huang, Y. (2022) · Cancers · doi:10.3390/cancers14194763 · Cited by 55 · see citing works
FIND: Human-in-the-Loop Debugging Deep Text Classifiers
Lertvittayakumjorn, P., Specia, L., & Toni, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.24 · Cited by 35 · see citing works
Artificial Text Detection via Examining the Topology of Attention Maps
Kushnareva, L., Cherniavskii, D., Mikhailov, V., Artemova, E., Barannikov, S., Bernstein, A., Piontkovskaya, I., Piontkovski, D., & Burnaev, E. V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.50 · Cited by 35 · see citing works
Deep Learning-Based Knowledge Graph Generation for COVID-19
Kim, T., Yun, Y. I., & Kim, N. (2021) · Sustainability · doi:10.3390/su13042276 · Cited by 34 · see citing works
Bonfigli, A., Bacco, L., Merone, M., & Dell’Orletta⋄, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103003 · Cited by 27 · see citing works
Tikhonov, A., & Ryabinin, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.310 · Cited by 24 · see citing works
Li, C., Knopman, D. S., Xu, W., Cohen, T., & Pakhomov, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.131 · Cited by 22 · see citing works
Koishekenov, Y., Berard, A., & Nikoulina, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.198 · Cited by 21 · see citing works
Acceptability Judgements via Examining the Topology of Attention Maps
Cherniavskii, D., Tulchinskii, E., Mikhailov, V., Proskurina, I., Kushnareva, L., Artemova, E., Barannikov, S., Piontkovskaya, I., Piontkovski, D., & Burnaev, E. V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.7 · Cited by 18 · see citing works
GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability
Jo, S., Jang, G., & Park, H. (2025) · Proceedings - International Conference on Image Processing · doi:10.1109/icip55913.2025.11084559 · Cited by 17 · see citing works
Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval
Jang, K.-R., Kang, J., Hong, G., Myaeng, S.-H., Park, J., Yoon, T., & Seo, H.-C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.78 · Cited by 17 · see citing works
Self-citationA vision attention driven Language framework for medical report generation
Arısoy, M. V., Arısoy, A., & Uysal, İ. (2025) · Scientific Reports · doi:10.1038/s41598-025-95666-8 · Cited by 15 · see citing works
SAMPLE-BASED DYNAMIC HIERARCHICAL TRANSFORMER WITH LAYER AND HEAD FLEXIBILITY VIA CONTEXTUAL BANDIT
Meng, F., Zhang, L., Chen, Y. A., & Wang, Y. (2024) · Proceedings on Engineering Sciences · doi:10.24874/pes06.02.001 · Cited by 10 · see citing works
Attention as a Guide for Simultaneous Speech Translation
Papi, S., Negri, M., & Turchi, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.745 · Cited by 9 · see citing works
Deciphering Stereotypes in Pre-Trained Language Models
Ma, W., Scheible, H., Wang, B., Veeramachaneni, G., Chowdhary, P., Sun, A., Koulogeorge, A., Wang, L., Yang, D., & Vosoughi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.697 · Cited by 8 · see citing works
Li, C., Wang, S., Zhang, Y., Zhang, J., & Zong, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1026 · Cited by 8 · see citing works
Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
Taktasheva, E., Mikhailov, V., & Artemova, E. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.17 · Cited by 7 · see citing works
Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?
Kim, Z. M., Besacier, L., Nikoulina, V., & Schwab, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.250 · Cited by 6 · see citing works
From Imitation to Introspection: Probing Self-Consciousness in Language Models
Chen, S., Yu, S., Zhao, S., & Lu, C. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.392 · Cited by 4 · see citing works
UHD-BERT: Bucketed Ultra-High Dimensional Sparse Representations for Full Ranking.Preprint
Jang, K.-R., Kang, J., Hong, G., Myaeng, S.-H., Park, J., Yoon, T., & Seo, H.-C. (2021) · arXiv (Cornell University) · Cited by 4 · see citing works
Self-citationCan BERT eat RuCoLA? Topological Data Analysis to Explain
Proskurina, I., Artemova, E., & Piontkovskaya, I. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.15 · Cited by 3 · see citing works
Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering
Kim, J.-H., Kang, J., Kim, K.-M., Hong, G., & Myaeng, S.-H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.138 · Cited by 3 · see citing works
Self-citationHafner, F. S., Hafner, L., & Corizzo, R. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00382-y · Cited by 2 · see citing works
Neural Approaches to Historical Words Reconstruction
Fourrier, C. (2022) · theses.fr (ABES) · doi:10.70675/647b8a58zeea0z48cdz8229zf1d7ee8b0742 · Cited by 2 · see citing works
Graph-Induced Transformers for Efficient Multi-Hop Question Answering
Hong, G., Kim, J.-H., Kang, J., & Myaeng, S.-H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.702 · Cited by 1 · see citing works
Self-citationDisentangling the Linguistic Competence of Privacy-Preserving BERT
Arnold, S., Kemmerzell, N., & Schreiner, A. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.5
Ryu, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.27
AERO: Softmax-Only LLMs for Efficient Private InferencePreprint
Jha, N. K., & Reagen, B. (2024) · Qeios · doi:10.32388/wwlt24
AERO: Softmax-Only LLMs for Efficient Private InferencePreprint
Jha, N. K., & Reagen, B. (2024) · Qeios · doi:10.32388/wwlt24.2
Interpreting Attention Mechanisms of NMT with Linguistic Features
Guanghui, C. A. I., & Junguo, Z. H. U. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024edp7292
Are LLMs and RAG Trustworthy Enough for Your Business?
Emekci, H. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8332-2.ch004
Enhancing Masked Language Modeling in BERT Models Using Pretrained Static Embeddings
Mištera, A., & Král, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02551-7_19
Dhar, S., Ilham, M. H. M., Joy, M. J. H., & Azhar, T. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491381
Optimizing Transformers: Metaheuristics for Attention Head Pruning
Ferreira, P., Antunes, C. H., Batista, J. P., & Pinto, T. M. P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-23604-3_19
- Authors
- Jo, J.-Y., & Myaeng, S.-H.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Corresponding author
- Myaeng, Sung-Hyon
- ORCID
- Jo, Jae-young
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 3404-3417
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034256339
- Cited by
- 33
- References
- 35
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 4 · 2023: 7 · 2022: 6 · 2021: 7 · 2020: 1
- FWCI
- 1.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- Korea Advanced Institute of Science and Technology
- Ministry of Science and ICT, South Korea (2013‐0‐00131)
- Institute for Information and Communications Technology Promotion (2013-0-00131)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- attention heads, multi-head attention, transformer language models, probing tasks, sentence representation, pre-trained language models, linguistic features, GPT, BERT, Electra, GPT-2, heat map visualization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
