SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Code summarization aims to generate natural language descriptions of source code, facilitating programmers to understand and maintain it rapidly. While previous code summarization efforts have predominantly focused on method-level, this paper studies file-level code summarization, which can assist programmers in understanding and maintaining large source code projects. Unlike method-level code summarization, file-level code summarization typically involves long source code within a single file, which makes it challenging for Transformer-based models to understand the code semantics for the maximum input length of these models is difficult to set to a large number that can handle long code input well, due to the quadratic scaling of computational complexity with the input sequence length. To address this challenge, we propose SparseCoder, an identifier-aware sparse transformer for effectively handling long code sequences. Specifically, the SparseCoder employs a sliding window mechanism for self-attention to model short-term dependencies and leverages the structure message of code to capture long-term dependencies among source code identifiers by introducing two types of sparse attention patterns named global and identifier attention. To evaluate the performance of SparseCoder, we construct a new dataset FILE-CS for file-level code summarization in Python. Experimental results show that our SparseCoder model achieves state-of-the-art performance compared with other pre-trained models, including full self-attention and sparse models. Additionally, our model has low memory overhead and achieves comparable performance with models using full self-attention mechanism. Furthermore, we verify the generality of SparseCoder on other code understanding tasks, i.e., code clone detection and code search, and results show that our model outperforms baseline models in both tasks, demonstrating that our model can generate better code representations for various downstream tasks. Our source code and experimental data are anonymously available at: https://github.com/DeepSoftwareAnalytics/SparseCoder.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Code-2 BERT [10] and CuBERT [29] are based on BERT [8] and trained on natural languages and code using masked language modeling.”
MethodLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · cited 4×
“With the great success of pre-training in code intelligence [2, 10, 15, 16, 46], pre-trained code models have been used to improve the performance of code summarization.”
BackgroundMethodInfluentialSelf-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Self-citationGenerating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673
Self-citationUnified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
Towards automatically generating summary comments for Java methods
Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., & Vijay-Shanker, K. (2010) · Proceedings of the IEEE/ACM international conference on Automated software engineering · doi:10.1145/1858996.1859006 · Cited by 514 · cited 2×
“1) Rule-based and IR-based approaches: Early work in this area largely employed rule-based and IR-based approaches [9, 19, 20, 35, 36, 38, 41].”
BackgroundMethod
Research works printed but not linked: 36
- 10.18653/v1/2020.acl-main.449 · DOI · Paper page
- Beltagy, 2020, Longformer: The long -document transformer, arXiv preprint
- 10.1007/978-3-031-33374-3_35 · DOI
- 10.18653/v1/P19-1285 · DOI · Paper page
- Feng, 2020, CodeBERT: A Pre-Trained Model for Programming and Natural Languages, arXiv Preprint
- 10.1016/j.jss.2022.111495 · DOI
- 10.1145/3524610.3527907 · DOI · Paper page
- 10.1109/SANER53432.2022.00013 · DOI
- Guo, 2021, Graphcodebert: Pre-training code representations with data flow, ICLR
- Guo, Modeling Hierarchical Syntax Structure with Triplet Position for Source Code Summa-rization
- 10.1109/WCRE.2010.13 · DOI · Paper page
- Haije, 2016, Automatic comment generation using a neural translation model
- 10.1145/3196321.3196334 · DOI · Paper page
- Hu, 2019, Deep code comment generation with hybrid lexical and syntactical information, Empirical Software Engineering
- 10.1007/s10664-019-09730-9 · DOI · Paper page
- Hu, 2018, Summarizing Source Code with Trans-ferred API Knowledge, IJCAI
- 10.18653/v1/P16-1195 · DOI · Paper page
- 10.18653/v1/p16-1195 · DOI · Paper page
- Kanade, Learning and evaluating contextual embedding of source code, 5110
- 10.1109/ICSE.2019.00087 · DOI · Paper page
- 10.1145/3387904.3389268 · DOI · Paper page
- 10.1145/3477495.3531811 · DOI
- 10.1109/ASE.2019.00026 · DOI · Paper page
- 10.1145/2597008.2597149 · DOI · Paper page
- Movshovitz-Attias, 2013, Nat-ural Language Models for Predicting Programming Comments, ACL
- 10.18653/v1/2020.findings-emnlp.232 · DOI · Paper page
- 10.18653/v1/2021.emnlp-main.332 · DOI
- Son, 2022, Boosting Code Summarization by Embedding Code Structures, COLING
- 10.1109/ICSME.2014.77 · DOI · Paper page
- 10.48550/ARXIV.1706.03762 · DOI · Paper page
- 10.1145/3238147.3238206 · DOI · Paper page
- 10.1145/3524610.3527903 · DOI
- 10.18653/v1/2021.acl-short.107 · DOI
- 10.18653/v1/2020.findings-emnlp.350 · DOI
- Zaheer, 2020, Big bird: Transformers for longer sequences, Advances in Neural Information Pro-cessing Systems, 33, 17283
- Zhang, 2020, Retrieval-based Neural Source Code Summarization, ICSE
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41
“Publication date: July 2024.”
Hu, X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 15
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11
Self-citation
Metadata
- Authors
- Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z.
- Venue
- 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)
- Publisher
- IEEE
- Published
- 2024-03-12
- Pages
- 614-625
- OpenAlex
- W4400681637
- Cited by
- 13
- References
- 17
- FWCI
- 5.11
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization. https://app.cochat.ai/papers/10.1109/saner60148.2024.00068
- The paper
- Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z. (2024). SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization. 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). https://doi.org/10.1109/saner60148.2024.00068
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.