CoChat

SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization

Open access
Wang, Yanlin, Huang, Yanxian, Guo, Daya, Zhang, Hongyu, Zheng, Zibin · 2024 · 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)
Cited by 13

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Code summarization aims to generate natural language descriptions of source code, facilitating programmers to understand and maintain it rapidly. While previous code summarization efforts have predominantly focused on method-level, this paper studies file-level code summarization, which can assist programmers in understanding and maintaining large source code projects. Unlike method-level code summarization, file-level code summarization typically involves long source code within a single file, which makes it challenging for Transformer-based models to understand the code semantics for the maximum input length of these models is difficult to set to a large number that can handle long code input well, due to the quadratic scaling of computational complexity with the input sequence length. To address this challenge, we propose SparseCoder, an identifier-aware sparse transformer for effectively handling long code sequences. Specifically, the SparseCoder employs a sliding window mechanism for self-attention to model short-term dependencies and leverages the structure message of code to capture long-term dependencies among source code identifiers by introducing two types of sparse attention patterns named global and identifier attention. To evaluate the performance of SparseCoder, we construct a new dataset FILE-CS for file-level code summarization in Python. Experimental results show that our SparseCoder model achieves state-of-the-art performance compared with other pre-trained models, including full self-attention and sparse models. Additionally, our model has low memory overhead and achieves comparable performance with models using full self-attention mechanism. Furthermore, we verify the generality of SparseCoder on other code understanding tasks, i.e., code clone detection and code search, and results show that our model outperforms baseline models in both tasks, demonstrating that our model can generate better code representations for various downstream tasks. Our source code and experimental data are anonymously available at: https://github.com/DeepSoftwareAnalytics/SparseCoder.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/saner60148.2024.00068

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 36

  1. 10.18653/v1/2020.acl-main.449 · DOI · Paper page
  2. Beltagy, 2020, Longformer: The long -document transformer, arXiv preprint
  3. 10.1007/978-3-031-33374-3_35 · DOI
  4. 10.18653/v1/P19-1285 · DOI · Paper page
  5. Feng, 2020, CodeBERT: A Pre-Trained Model for Programming and Natural Languages, arXiv Preprint
  6. 10.1016/j.jss.2022.111495 · DOI
  7. 10.1145/3524610.3527907 · DOI · Paper page
  8. 10.1109/SANER53432.2022.00013 · DOI
  9. Guo, 2021, Graphcodebert: Pre-training code representations with data flow, ICLR
  10. Guo, Modeling Hierarchical Syntax Structure with Triplet Position for Source Code Summa-rization
  11. 10.1109/WCRE.2010.13 · DOI · Paper page
  12. Haije, 2016, Automatic comment generation using a neural translation model
  13. 10.1145/3196321.3196334 · DOI · Paper page
  14. Hu, 2019, Deep code comment generation with hybrid lexical and syntactical information, Empirical Software Engineering
  15. 10.1007/s10664-019-09730-9 · DOI · Paper page
  16. Hu, 2018, Summarizing Source Code with Trans-ferred API Knowledge, IJCAI
  17. 10.18653/v1/P16-1195 · DOI · Paper page
  18. 10.18653/v1/p16-1195 · DOI · Paper page
  19. Kanade, Learning and evaluating contextual embedding of source code, 5110
  20. 10.1109/ICSE.2019.00087 · DOI · Paper page
  21. 10.1145/3387904.3389268 · DOI · Paper page
  22. 10.1145/3477495.3531811 · DOI
  23. 10.1109/ASE.2019.00026 · DOI · Paper page
  24. 10.1145/2597008.2597149 · DOI · Paper page
  25. Movshovitz-Attias, 2013, Nat-ural Language Models for Predicting Programming Comments, ACL
  26. 10.18653/v1/2020.findings-emnlp.232 · DOI · Paper page
  27. 10.18653/v1/2021.emnlp-main.332 · DOI
  28. Son, 2022, Boosting Code Summarization by Embedding Code Structures, COLING
  29. 10.1109/ICSME.2014.77 · DOI · Paper page
  30. 10.48550/ARXIV.1706.03762 · DOI · Paper page
  31. 10.1145/3238147.3238206 · DOI · Paper page
  32. 10.1145/3524610.3527903 · DOI
  33. 10.18653/v1/2021.acl-short.107 · DOI
  34. 10.18653/v1/2020.findings-emnlp.350 · DOI
  35. Zaheer, 2020, Big bird: Transformers for longer sequences, Advances in Neural Information Pro-cessing Systems, 33, 17283
  36. Zhang, 2020, Retrieval-based Neural Source Code Summarization, ICSE

Cited by

See all 13

Metadata

Authors
Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z.
Venue
2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)
Publisher
IEEE
Published
2024-03-12
Pages
614-625
OpenAlex
W4400681637
Cited by
13
References
17
FWCI
5.11
Type
conference paper
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization. https://app.cochat.ai/papers/10.1109/saner60148.2024.00068
The paper
Wang, Y., Huang, Y., Guo, D., Zhang, H., & Zheng, Z. (2024). SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization. 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). https://doi.org/10.1109/saner60148.2024.00068

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.