CoChat
Sign Up
Cited by 8Software Engineering ResearchTopic ModelingInformation Retrieval and Search Behavior

Natural Is the Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models

Open access
Wang, Yan, Li, Xiaoning, Nguyen, Tien N., Wang, Shaohua, Ni, Chao, Ding, Ling·2024·Proceedings of the ACM on Software Engineering
DOI 10.1145/3643753
Open this paper in CoChat — free account

Abstract

Pre-trained Large Language Models (LLM) have achieved remarkable successes in several domains. However, code-oriented LLMs are often heavy in computational complexity, and quadratically with the length of the input code sequence. Toward simplifying the input program of an LLM, the state-of-the-art approach has the strategies to filter the input code tokens based on the attention scores given by the LLM. The decision to simplify the input program should not rely on the attention patterns of an LLM, as these patterns are influenced by both the model architecture and the pre-training dataset. Since the model and dataset are part of the solution domain, not the problem domain where the input program belongs, the outcome may differ when the model is pre-trained on a different dataset. We propose S lim C ode , a model-agnostic code simplification solution for LLMs that depends on the nature of input code tokens. As an empirical study on the LLMs including CodeBERT, CodeT5, and GPT-4 for two main tasks: code search and summarization, we reported that 1) the removal ratio of code has a linear-like relation with the saving ratio on training time, 2) the impact of categorized tokens on code simplification can vary significantly, 3) the impact of categorized tokens on code simplification is task-specific but model-agnostic, and 4) the above findings hold for the paradigm–prompt engineering and interactive in-context learning. The empirical results showed that S lim C ode can improve the state-of-the-art technique by 9.46 % and 5.15 % in terms of MRR and BLEU score on code search and summarization, respectively. More importantly, S lim C ode is 133 times faster than the state-of-the-art approach. Additionally, S lim C ode can reduce the cost of invoking GPT-4 by up to 24 % per API query, while still producing comparable results to those with the original code. With this result, we call for a new direction on code-based, model-agnostic code simplification solutions to further empower LLMs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3643753

Cites

24 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional code compressors such as DietCode and SlimCode improve syntax and structure awareness but are generally limited to function-level pruning or short code examples, leaving compression of long context for code largely unaddressed.

    “Traditional code compressors such as DietCode [26] and SlimCode [27] improve syntax and structure awareness but are generally limited to function-level pruning or short code examples, leaving compression of long context for code largely unaddressed.”

    From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [27]

    The SlimCode Algorithm (p. 13) describes token removal within individual code snippets, and the Empirical Evaluation (p. 14) uses model input limits of 200 and 256 for code search and summarization. These passages support describing the evaluated setting as short code snippets, but they do not establish that SlimCode is inherently limited to function-level pruning or that long-context code compression is largely unaddressed. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 13

  1. [3][3] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to Represent Programs with Graphs. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net. https://openreview.net/forum?id=BJOFETxR- -
  2. [6][6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). - · arXiv
  3. [8][8] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850 (2022). - · arXiv
  4. [9][9] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net. https: //openreview.net/forum?id=jLoC4ez43PZ -
  5. [10][10] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019). - · arXiv
  6. [12][12] Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021. Treebert: A tree-based pre-trained model for programming language. In Uncertainty in Artificial Intelligence . PMLR, 54-63. -
  7. [13][13] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2019. Pre-trained contextual embedding of source code. arXiv preprint arXiv:2001.00059 (2019). - · arXiv
  8. [14][14] Rafael-Michael Karampatsis and Charles Sutton. 2020. Scelmo: Source code embeddings from language models. arXiv preprint arXiv:2004.13214 (2020). - · arXiv
  9. [18][18] Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. DLFix: context-based code transformation learning for automated program repair. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE '20) . Association for Computing Machinery, New York, NY, USA, 602-614. https://doi.org/10.1145/ 3377811.3380345 -
  10. [23][23] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. 2022. CodeReviewer: Pre-training for automating code review activities. arXiv e-prints (2022), arXiv-2203. -
  11. [24][24] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). - · arXiv
  12. [29][29] Chao Ni, Kaiwen Yang, Xin Xia, David Lo, Xiang Chen, and Xiaohu Yang. 2022. Defect Identification, Categorization, and Repair: Better Together. arXiv preprint arXiv:2204.04856 (2022). - · arXiv
  13. [30][30] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. arXiv preprint (2022). -

Cited by

Sorted by citations

Related

Authors
Wang, Y., Li, X., Nguyen, T. N., Wang, S., Ni, C., & Ding, L.
Affiliations
  • Central University of Finance and Economics (CN)
  • The University of Texas at Dallas (US)
  • Zhejiang University (CN)
ORCID
Venue
Proceedings of the ACM on Software Engineering
Publisher
Association for Computing Machinery (ACM)
Published
2024-07-12
Volume
1
Issue
FSE
Pages
586-608
ISSN
2994-970X (ISSN-L)
DOI
10.1145/3643753
OpenAlex
W4400582826
Cited by
6
References
23
Citations by year
2026: 1 · 2025: 3 · 2024: 2
FWCI
2.26
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (CNS-2120386)
  • National Security Agency (NCAE-C-002-2021)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
code simplification, large language models, code search, code summarization, training time reduction, prompt engineering, GPT-4, CodeBERT, CodeT5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review