CoChat
Sign Up
Cited by 0Software Engineering ResearchTeaching and Learning ProgrammingTopic Modeling

What Makes In-Context Examples Effective for Code Generation?

Open access
Li, Dongze, Chen, Songqiang, Cao, Jialun, Cheung, Shing-Chi·2026·Proceedings of the ACM on Software Engineering
DOI 10.1145/3832275

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

In-Context Learning (ICL) has emerged as a promising solution to enhance the code generation capabilities of Large Language Models (LLMs) by incorporating code examples inside the prompt to let LLMs learn from demonstrations. However, despite their effectiveness gains, it remains unclear which specific properties of ICL-provided code examples (e.g., solution insight, essential contextual information, identifier naming styles, code formatting) drive these gains. This paper systematically investigates the impact of different sources and internal features of code examples on ICL for code generation through controlled experiments on contest-style programming questions and repository-level tasks. Our results show that while LLMs struggle to extract generalizable problem-solving insights from provided solutions to similar questions or repository snippets, their retrieval-augmented ICL performance can significantly benefit from explicit contextual information, such as input/output demonstrations, required helper functions, and namespace information. Through targeted mutation operators, we further find that identifier naming is substantially more critical than code formatting or low-level implementation details, with the elimination of descriptive variable names causing performance drops of up to 30 percentage points. Finally, we demonstrate that LLMs significantly prefer semantically meaningful identifier names and that adherence to surface-level naming conventions is far less important than semantic clarity. These findings provide practical guidelines for constructing effective ICL code examples and highlight challenges in reflection-based learning for code generation.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper systematically studies which properties of in-context code examples drive LLM code-generation gains, using controlled retrieval and mutation experiments on LeetCode and DevEval. It finds explicit contextual information and semantically meaningful identifier naming are the most beneficial, while LLMs largely fail to exploit problem-solving insights from similar question-solution examples.

Key findings

  • Explicit contextual information helps: augmenting BM25-retrieved examples with unittests and required function/variable information raised Pass@1 for GPT-4o-mini from 19.62 to 27.65, Qwen-7B from 16.04 to 20.48, Qwen-32B from 22.18 to 28.67, and DS-Coder from 16.89 to 23.04; adding namespace information also improved all models (e.g., Qwen-32B on BM25 from 26.70 to 30.34, and on gist-large from 36.79 to 39.66). Similar question-solution retrieval on LeetCode did not significantly improve over Zero Shot or Random.
  • Identifier naming is more critical than formatting or implementation details: VOE and FVE caused the largest drops, with FVE reducing Qwen-7B on LeetCode-Python by 30 percentage points (86.46 to 55.52); on DevEval, FVE dropped GPT-4o-mini from 37.91 to 25.58, Qwen-7B from 24.67 to 17.66, and Qwen-32B from 36.79 to 26.91.
  • Naming style matters mainly through semantic informativeness: CN and VN generally outperformed VHR and VMR, which caused the largest degradation and were often worse than VOE; on LeetCode-Python CN generally beat VN, while on DevEval VN was consistently higher than CN.
  • LLMs struggle to extract generalizable problem-solving insights: CodeBLEU similarity between outputs and retrieved examples showed no significant difference with versus without ICL (p ≥ 0.99), and augmenting LeetCode BM25 content with required functions/variables did not improve performance (p > 0.1; p_HB > 0.6).

Methods

  • Controlled experiments on LeetCode questions (362 of 381 LiveCodeBench-v5 questions) and DevEval tasks (1427 of 1825 tasks), using Python and Java for LeetCode; four LLMs: GPT-4o-mini, Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and DeepSeek-Coder-V2-Lite-Instruct, with temperature=0, max_tokens=8000, and k=5 retrieved documents.
  • RQ1 built RAG-ICL systems with BM25, embedding models (gist-large, jina-v2-code, codesage-small), algorithm-tag retrieval, random retrieval, plus Ground Truth and Zero Shot baselines; RQ2 applied eight mutation operators across naming (FOE, VOE, FVE), format (INR, LSR, TOK), and implementation details (NL, LT); RQ3 synthesized naming styles (CN, VN, VHR, VMR) plus VOE.
  • Main metric was Pass@1; analyses used Wilcoxon signed-rank tests, paired t-tests, Linear Mixed Models, Holm-Bonferroni correction, Bootstrap confidence intervals, and CodeBLEU similarity for reflection analysis.

Limitations

  • Stated in the paper: mutation operators may not cover all code features; LeetCode and DevEval may not represent all real-world development scenarios; potential data contamination was addressed by partitioning questions around model knowledge cutoffs and finding high Spearman rank correlations across subsets.
  • Stated in the paper: statistical assumptions for paired t-tests and LMMs were frequently violated, so conclusions are grounded on Wilcoxon signed-rank tests with Bootstrap CIs, with t-tests and LMMs reported only as concordance checks.
  • Evident from the text: the study covers four LLMs, two benchmarks, Python/Java, k=5, and a limited set of retrievers and mutation operators, so generalization to other models, languages, retrieval settings, or code properties is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3832275

Cites

98 of 98 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

17 of 51 cited sources checked · 6 supported · 1 flagged · accuracy not yet measured

  • Study design and conductown data
    § 3.3 Selection of LLMs

    The default maximum context length for all LLMs is sufficient to accommodate all prompts synthesized in the study since the maximum token number is approximately 11K for LeetCode and 27K for DevEval under k=5.

    “We adopt the default maximum context length for all LLMs (128K for GPT-4o-mini [52] and Qwen models [56]; 32K for DS-Coder [17]), which is sufficient to accommodate all the prompts synthesized in our study since the maximum token number is approximately 11K for LeetCode and 27K for DevEval under 𝑘 = 5 configuration.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [52] [52] OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancingcost-efficient-intelligence/ -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [56] [56] Qwen:, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. doi:10.48550/arXiv.2412.15115 - source
    • cites [17] [17] DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. doi:10.48550/arXiv.2406.1193 source

      The citation is attached to the claimed 32K maximum context length for DS-Coder. Section 3.4, “Long Context Extension” (page unavailable), instead gives DeepSeek-Coder-V2 a 128K context length; 32K was a sequence length used during one training stage. The paper does not provide the citing study’s LeetCode or DevEval prompt counts. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation
    § 2.2 Motivation

    Existing studies primarily evaluate different knowledge sources for retrieval-augmented ICL, but do not systematically analyze which types of properties and information within retrieved content are most beneficial.

    “First, existing studies [77, 84] primarily evaluate different knowledge sources for retrieval-augmented ICL, but do not systematically analyze which types of properties and information within retrieved content (e.g., input/output demonstrations, required function/variable information, repository structure) are most beneficial.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [77] [77] Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2025. CodeRAG-Bench: Can Retrieval Augment Code Generation?. In Findings of the Association for Computational Linguistics: NAACL 2025 . Association for Computational Linguistics, Albuquerque, New Mexico, 3199-3214. doi:10.18653/v1/2025. findings-naacl.176 - source
    • cites [84] [84] Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, and Xin Xia. 2025. An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities. ACM Trans. Softw. Eng. Methodol. 34, 7, Article 188 (Aug 2025), 28 pages. doi:10.1145/3717061 - source
  • Premisecitation
    § 5.2 Threats to Validity

    LeetCode and DevEval are among the most widely adopted benchmarks for evaluating LLM code generation capabilities.

    “we argue that LeetCode and DevEval are among the most widely adopted benchmarks for evaluating LLM code generation capabilities [30, 43, 66]”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [30] [30] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. doi:10.48550/arXiv.2409.12186 - source

      Section 7.2 (page unavailable) mentions LeetCode as a source of problems for LiveCodeBench, not as a benchmark whose adoption is assessed. Searches for DevEval and for the claimed adoption of either benchmark found no passage addressing the claim. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [43] [43] Jia Li, Xianjie Shi, Kechi Zhang, Ge Li, Zhi Jin, Lei Li, Huangzhao Zhang, Jia Li, Fang Liu, Yuwei Zhang, Zhengwei Tao, Yihong Dong, Yuqi Zhu, and Chongyang Tao. 2025. GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation. doi:10.48550/arXiv.2504.10046 - source
    • cites [66] [66] ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong, Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang, Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Shen, Liang Xiang, and Yonghui Wu. 2025. Seed-Coder: Let the Code Model Curate Data for Itself. doi:10.48550/arXiv.2506.03524 - source
  • Interpretationown data
    § 4.2 RQ2: Effect of Different Code Features

    There is a correlation between LLMs' coding ability and context code leveraging with information elimination.

    “The pattern indicates a correlation between the LLMs' coding ability (demonstrated by the performance on code generation benchmarks [12, 90]) and context code leveraging with information elimination.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [12] [12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Such source
    • cites [90] [90] Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In The Thirteenth International source
  • Findingown data
    § What Makes In-Context Examples Effective for Code Generation?

    LLMs significantly prefer semantically meaningful identifier names.

    “Finally, we demonstrate that LLMs significantly prefer semantically meaningful identifier names”
    • Also stated in: another passage · plain
      “Moreover, our experiments reveal that LLMs prefer semantically meaningful identifier names”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 21
  • May combine two claims. 24
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2.2 Motivation

    Directly incorporating the correct solution code as ICL content substantially enhances LLM performance.

    “More recently, Wang et al. [77] demonstrate that directly incorporating the correct solution code as ICL content substantially enhances LLM performance”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al. [77] source

      Section 2.3 identifies the canonical documents for HumanEval and MBPP as programming-solution documents. Section 3.3 reports that providing canonical context substantially improves generation on basic programming problems, but not across all models or tasks; it notes that GPT models show no gains in some open-domain settings. The paper describes supplying documents as context, not testing correct solution code specifically as few-shot ICL content. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2.2 Motivation

    Retrieved examples from different knowledge sources (e.g., programming solutions, GitHub repositories) yield limited degrees of benefit.

    “retrieved examples from different knowledge sources (e.g., programming solutions, GitHub repositories) yield limited degrees of benefit”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Wang et al. [77] source

      Section 4, “RACG with Open Retrieval” (page unavailable), reports modest gains from some sources and no improvement from others on HumanEval; on RepoEval, open sources add little beyond local code. But it also reports that programming solutions yield the largest gains on ODEX and that mixed sources perform as well as gold documents on HumanEval. Calling the benefit uniformly limited overstates these task-dependent results. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Experimental Design

    Recent studies have demonstrated remarkable successes of ICL in code generation tasks.

    “Recent studies have demonstrated remarkable successes of ICL in code generation tasks [24, 44, 71, 89].”
    Quote in the paperNot a substantial claim1 of 4 sources checked
    • cites [24] [24] Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R. Lyu. 2023. What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? . In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, Los Alamitos, CA, USA, 761-773. doi:10.1109/ASE56229.2023.00109 - source
    • cites [44] [44] Jia Li, Chongyang Tao, Jia Li, Ge Li, Zhi Jin, Huangzhao Zhang, Zheng Fang, and Fang Liu. 2025. Large Language Model-Aware In-Context Learning for Code Generation. ACM Trans. Softw. Eng. Methodol. 34, 7, Article 190 (Aug. 2025), 33 pages. doi:10.1145/3715908 - source
    • cites [71] [71] Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, and Tao Yu. 2024. EvoR: Evolving Retrieval for Code Generation. In Findings of the Association for Computational Linguistics: EMNLP 2024 , Yaser AlOnaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 2538-2554. doi:10.18653/v1/2024.findings-emnlp.143 - source
    • cites [89] [89] Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. 2023. DocPrompting: Generating Code by Retrieving the Docs. In The Eleventh International Conference on Learning Representations . https://openreview.net/ forum?id=ZTCxT2t2Ru - source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.2 Experimental Design

    Prior works categorize code features into three dimensions: code naming, code format, and implementation details.

    “Prior works categorize code features into three dimensions: code naming [64], code format [55], and implementation details [13].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [64] [64] Simone Scalabrino, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2016. Improving code readability models with textual features. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC) . 1-10. doi:10.1109/ICPC.2016.7503707 - source
    • cites [55] [55] Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A simpler model of software readability. In Proceedings of the 8th Working Conference on Mining Software Repositories (Waikiki, Honolulu, HI, USA) (MSR '11) . Association for Computing Machinery, New York, NY, USA, 73-82. doi:10.1145/1985441.1985454 - source
    • cites [13] [13] Songqiang Chen, Congying Xu, Jingyi Chen, Jialun Cao, Jiarong Wu, and Shing-Chi Cheung. 2026. Can Emulating Semantic Translation Help LLMs with Code Translation? A Study Based on Pseudocode. ACM Trans. Softw. Eng. Methodol. (Jan. 2026). doi:10.1145/3790101 Just Accepted. - source

      The citation to this paper is attached to “implementation details,” not to the other two dimensions. The experimental-strategies passage (p. 8) explicitly discusses implementation details of the original program, but it does not categorize code features into the three stated dimensions. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § 3.4 Performance Metrics

    The study used Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.

    “Following existing works evaluating LLM code generation [12, 23] and common evaluation practices over LiveCodeBench and DevEval, we use Pass@1 as the evaluation metric for all RQs rather than code similarity scores, as it is more appropriate for problem-solving and code generation tasks.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [12] [12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Such source
    • cites [23] [23] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net. https://openreview.net/forum?id=hQwb-lbM6EL - source

      Section C.6 (p. 20) confirms that InCoder used pass@1 to evaluate code synthesis on HumanEval and MBPP, supporting the citation as a precedent for that metric. It does not say pass@1 is more appropriate than code-similarity scores: it also reports pass@10 and pass@100 for HumanEval, while Table 1 (p. 5) reports exact match alongside pass rate. The citing authors’ choice to use pass@1 for all their research questions is their own methodology, not a finding attributed to InCoder. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 3.2 Experimental Design

    ICL is not a universally beneficial solution; its effectiveness remains context-dependent.

    “However, ICL is not a universally beneficial solution, as reported by [50, 77, 80], indicating that ICL's effectiveness remains context-dependent.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [50] [50] Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2024. Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 5506-5521. doi:10.18653/v1/2024.naacl-long.308 - source
    • cites [77] [77] Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2025. CodeRAG-Bench: Can Retrieval Augment Code Generation?. In Findings of the Association for Computational Linguistics: NAACL 2025 . Association for Computational Linguistics, Albuquerque, New Mexico, 3199-3214. doi:10.18653/v1/2025. findings-naacl.176 - source

      Section 3.3 reports that adding retrieved documents helps some code-generation models and tasks but gives GPT models no gain in the setups discussed. The passages searched for “ICL” and “in-context learning” address retrieval-augmented code generation, not whether in-context learning itself is universally beneficial; page numbers were unavailable in our copy. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [80] [80] Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. REPOFORMER: selective retrieval for repository-level code completion. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML'24) . JMLR.org, Article 2183, 21 pages. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Experimental Design

    Ranking retrieved ICL examples by similarity score with the most similar closest to the target question is generally more effective than random or reversed ordering.

    “The retrieved ICL examples are then included in the prompt in the order of their similarity scores calculated by the retriever with the most similar example placed closest to the target question, as this ranking strategy is reported to be generally more effective compared to random or reversed ordering [24].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [24] [24] Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R. Lyu. 2023. What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? . In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, Los Alamitos, CA, USA, 761-773. doi:10.1109/ASE56229.2023.00109 - source

  • May combine two claims.

    Premisecitation
    § 6.2 Study on LLMs' Preference on Code Snippets

    Existing techniques for optimizing LLM behavior on code snippets are mainly driven by machine learning approaches and lack explanations of LLM preference or predictable results before actual generation.

    “These optimization techniques are mainly driven by machine learning approaches, lacking explanations on preference of LLM or predictable results before the actual generation.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Jain et al. [34] [34] Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021. Contrastive Code Representation Learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5954-5971. doi:10.18653/v1/ 2021.emnlp-main.482 - source
    • cites Wang et al. [76] [76] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained EncoderDecoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8696-8708. doi:10.18653/v1/2021.emnlp-main.685 - source

      The Introduction describes existing code-related techniques as adaptations of machine-learning pre-training methods. It criticizes them for overlooking code structure, but does not say they lack explanations of LLM preferences or predictable results before generation. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Henkel et al. [27] [27] Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. 2022. Semantic Robustness of Models of Source Code. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . 526-537. doi:10.1109/SANER53432.2022.00070 - source

  • May combine two claims.

    Premisenone given
    § 6.1 Advancements in In-Context Learning

    Existing works on ICL for coding tasks treat code examples as monotonic units and do not explore the contributing features within a code snippet.

    “Although existing works advance in optimizing ICL for various tasks including coding, they consider code examples as monotonic units and do not explore the contributing features in a code snippet.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Liu et al. [47] [47] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures . Association for Computational Linguistics, Dublin, Ireland and Online, 100-114. doi:10.18653/v1/2022.deelio-1.10 - source

      The Trial section describes selecting whole in-context examples by similarity to a test sample, which supports the general point that this work selects examples as units. But Section 3, Experimental Setup, lists sentiment analysis, table-to-text generation, and question answering—not coding. The paper does not establish that existing coding-task work treats code snippets as indivisible units or neglects their contributing features. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Min et al. [51] [51] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 11048-11064. doi:10.18653/v1/2022.emnlp-main.759 - source

      The Introduction describes experiments isolating features of in-context demonstrations, including label space, input distribution, and format. The Related Work and Experimental Details passages do not assess coding-task examples or whether prior coding work examines features within code snippets; a search for coding tasks and code snippets found no such passage. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Gao et al. [24] [24] Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R. Lyu. 2023. What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? . In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, Los Alamitos, CA, USA, 761-773. doi:10.1109/ASE56229.2023.00109 - source

      The Introduction says earlier studies mainly selected and arranged demonstration examples, and the paper examines examples by selection, order, and number. The Conclusion and Future Work leaves other aspects of source code, including quality and naturalness, for future study. This supports the narrower point that this paper does not investigate features within snippets, but it does not establish that all existing ICL work treats code examples as indivisible units or neglects those features. (LLM, conf=0.79)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 4.1 RQ1: Effective ICL-Incorporated Information

    LLMs infer general logic correctly but produce incorrect input/output formats.

    “The LLM infers the general logic correctly but produces incorrect input/output formats”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 RQ1: Effective ICL-Incorporated Information

    BM25-based RAG covers 48.7% of variables and 38.9% of functions/classes used in the ground truth, while embedding models reach 61%-66%.

    “Similarly, BM25-based RAG covers 48.7% of variables and 38.9% of functions/classes used in the ground truth, while embedding models reach 61%-66%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 RQ1: Effective ICL-Incorporated Information

    Augmenting BM25 retrievals with unittests and required function/variable information yields substantial performance gains, achieving results comparable to embedding-based retrievers.

    “As shown in Table 2, augmenting BM25 retrievals with unittests and required function/variable information (Row 3) yields substantial performance gains ( 𝑝 < 10 -10 ; 𝑝 𝐻𝐵 < 10 -10 and all Bootstrap CI above zero for all models), achieving results comparable to embedding-based retrievers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 RQ1: Effective ICL-Incorporated Information

    For documents retrieved by BM25, unittest recall is 2.38%, whereas embedding-based retrievers achieve recall ranging from 51.93% to 66.92%.

    “We observed that for documents retrieved by BM25, unittest recall is only 2.38%, whereas the embedding-based retrievers achieve much higher recall, ranging from 51.93% to 66.92%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 RQ3: Effect of Different Code Naming Styles

    On LeetCode, CN generally outperforms VN for Python across all models, while preference is weaker for Java.

    “On LeetCode, CN generally outperforms VN for Python across all models, while the preference is weaker for Java (e.g., Qwen2.532B-Ins slightly favors VN).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 RQ3: Effect of Different Code Naming Styles

    VHR and VMR exhibit the largest degradation and are generally even worse than VOE.

    “In contrast, VHR and VMR, which replace identifiers with high-frequency names and their combinations, exhibit the largest degradation ( 𝑝 < 10 -3 ; 𝑝 𝐻𝐵 < 10 -3 and all Bootstrap CI below zero under all combinations of models and mutations) and are generally even worse than VOE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 RQ2: Effect of Different Code Features

    DS-Coder presents a unique behavioral profile on LeetCode: performance lags under Origin, becomes competitive under format and implementation detail mutation configurations.

    “In addition, we observe that DS-Coder presents a unique behavioral profile on LeetCode: its performance significantly lags behind other models under Origin configuration, while it becomes competitive under format and implementation detail mutation configurations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning
    § 4.2 RQ2: Effect of Different Code Features

    FOE configuration causes minimal performance degradation on LeetCode, compared with a relatively significant decrease on DevEval.

    “Among all mutation operators on identifier names, FOE configuration causes minimal performance degradation on LeetCode, compared with a relatively significant decrease on DevEval.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 7 Conclusion

    Despite the overall success of ICL across diverse scenarios, current LLMs remain limited in their ability to generalize problem-solving insights based on reflection and inference.

    “Despite the overall success of ICL across diverse scenarios, current LLMs remain limited in their ability to generalize problem-solving insights based on reflection and inference.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4.2 RQ2: Effect of Different Code Features

    For DevEval, identifier obfuscation remains the dominant failure factor.

    “For DevEval tasks depending on external context, although the absolute performance is lower due to task difficulty, identifier obfuscation remains the dominant failure factor.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 RQ2: Effect of Different Code Features

    Under FVE, DevEval performance drops from 37.91% to 25.58% for GPT-4o-mini, 24.67% to 17.66% for Qwen-7B, and 36.79% to 26.91% for Qwen-32B.

    “For instance, under FVE, DevEval performance drops from 37.91% to 25.58% for GPT-4o-mini, 24.67% to 17.66% for Qwen-7B, and 36.79% to 26.91% for Qwen32B.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 RQ2: Effect of Different Code Features

    Qwen-32B demonstrates remarkable robustness across all mutation operators, maintaining performance above 87% for LeetCode and above 30% in nearly all DevEval configurations except FVE.

    “Qwen-32B demonstrates remarkable robustness across all mutation operators, maintaining performance above 87% for all configurations in LeetCode, and above 30% in nearly all DevEval configurations except FVE.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 1 Introduction

    Explicit contextual information (e.g., input/output demonstrations, required function/variable information, repository structure) and meaningful identifier naming are the most beneficial properties among the studied ICL components.

    “We identify explicit contextual information (e.g., input/output demonstrations, required function/variable information, repository structure) and meaningful identifier naming as the most beneficial properties among the studied ICL components”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 3.2 Experimental Design

    The study excluded ground-truth documents from the retrieval corpus to prevent data leakage and ensure retrieved examples represent only external knowledge available to the model.

    “To simulate a common ICL scenario, we exclude ground-truth documents from the retrieval corpus.”
    • Also stated in: another passage · plain
      “This prevents inadvertent data leakage and ensures retrieved examples represent only external knowledge available to the model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.1 RQ1: Effective ICL-Incorporated Information

    Retrieved examples neither meaningfully influence the generation process nor introduce systematic bias.

    “indicating that retrieved examples neither meaningfully influence the generation process nor introduce systematic bias”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.1 Overall Pipelines

    Java is prevalent in real-world development and academic instruction.

    “To investigate the effectiveness of ICL code examples across programming languages, we included Java for LeetCode questions due to its prevalence in real-world development and academic instruction, as supported by the statistics from IEEE Spectrum [32] and the developer survey from Stack Overflow [70].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [32] [32] IEEE Spectrum. 2025. The Top Programming Languages 2025. https://spectrum.ieee.org/top-programming-languages2025 Accessed: 2026-04-27. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [70] [70] Stack Overflow. 2025. Technology | 2025 Stack Overflow Developer Survey. https://survey.stackoverflow.co/2025/ technology#most-popular-technologies-language-language Accessed: 2026-04-27. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Experimental Design

    The study conducted a sanity check to ensure runtime behavior preservation by directly executing the mutated solution on the original tests.

    “We conducted a sanity check to ensure runtime behavior preservation by directly executing the mutated solution on the original tests.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.2 Experimental Design

    The study conducted an AST structure comparison between the original code and the mutated code to verify that structural integrity is maintained and that the two versions only differ in identifier names.

    “We conducted an AST structure comparison between the original code and the mutated code to verify that the structural integrity is maintained and that the two versions only differ in identifier names.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 3.2 Experimental Design

    Existing studies have indicated different identifier naming styles as a factor that may affect code comprehension for both human developers and LLMs.

    “Existing studies have indicated different identifier naming styles as a factor that may affect code comprehension for both human developers [3, 65] and LLMs [29, 75].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [3] [3] Dave Binkley, Marcia Davis, Dawn Lawrie, and Christopher Morrell. 2009. To camelcase or under\_score. In 2009 IEEE 17th International Conference on Program Comprehension . 158-167. doi:10.1109/ICPC.2009.5090039 - source
    • cites [65] [65] Andrea Schankin, Annika Berger, Daniel V. Holt, Johannes C. Hofmeister, Till Riedel, and Michael Beigl. 2018. Descriptive compound identifier names improve source code comprehension. In Proceedings of the 26th Conference on Program Comprehension (Gothenburg, Sweden) (ICPC '18) . Association for Computing Machinery, New York, NY, USA, 31-40. doi:10.1145/3196321.3196332 - source
    • cites [29] [29] Chao Hu, Yitian Chai, Hao Zhou, Fandong Meng, Jie Zhou, and Xiaodong Gu. 2024. How Effectively Do Code Language Models Understand Poor-Readability Code?. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE '24) . Association for Computing Machinery, New York, NY, USA, 795-806. doi:10.1145/3691620.3695072 - source
    • cites [75] [75] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023. ReCode: Robustness Evaluation of Code Generation Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, Toronto, Canada, 13818-13843. doi:10.18653/v1/2023.acl-long.773 - source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.1 Overall Pipelines

    The study excluded tasks whose ground truth answers failed on the provided test cases from both benchmarks.

    “For a clean evaluation, we exclude tasks whose ground truth answers failed on the provided test cases from both benchmarks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 3.3 Selection of LLMs

    The study adopted temperature=0 to minimize the randomness of LLMs.

    “For all LLMs, we adopt temperature=0 to minimize the randomness of LLMs”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Overall Pipelines

    These two scenarios (competition and repository-level) are summarized as representative dimensions of code generation benchmarks by Jiang et al. [35].

    “These two scenarios (competition and repository-level) are summarized as representative dimensions of code generation benchmarks by Jiang et al. [35].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jiang et al. [35] [35] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A Survey on Large Language Models for Code Generation. ACM Trans. Softw. Eng. Methodol. 35, 2, Article 58 (Jan. 2026), 72 pages. doi:10.1145/3747588 - source

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 3.2 Experimental Design

    The runtime behavior of mutated code is verified to be preserved through executing the tests.

    “Similar to RQ2, the runtime behavior of mutated code is verified to be preserved through executing the tests.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.1 RQ1: Effective ICL-Incorporated Information

    For self-contained LeetCode questions, functions extracted from ground-truth solutions are usually self-defined helpers or standard-library/third-party APIs that LLMs already know, and most extracted variables are local variables whose names can be arbitrarily assigned.

    “In contrast, for the self-contained LeetCode questions, the functions extracted from ground-truth solutions are usually self-defined helpers or standardlibrary/third-party APIs that LLMs already know, and most extracted variables are local variables whose names can be arbitrarily assigned.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 4.3 RQ3: Effect of Different Code Naming Styles

    Identifier names may largely affect code comprehension for both humans and LLMs.

    “Prior works indicated that identifier names may largely affect code comprehension for both humans and LLMs [8, 64].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] [8] Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2010. Exploring the Influence of Identifier Names on Code Quality: An Empirical Study. In 2010 14th European Conference on Software Maintenance and Reengineering . 156-165. doi:10.1109/CSMR.2010.27 - source
    • cites [64] [64] Simone Scalabrino, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2016. Improving code readability models with textual features. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC) . 1-10. doi:10.1109/ICPC.2016.7503707 - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § 4.2 RQ2: Effect of Different Code Features

    Different LLMs may prefer unique representation formats when processing source code in context.

    “This observation may suggest that different LLMs may prefer unique representation formats when processing source code in context.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation
    § 5.2 Threats to Validity

    They systematically categorized code features into three fundamental dimensions based on established literature and designed mutation operators covering each dimension.

    “we refer to prior works focusing on code readability, which propose clear definitions regarding factors contained in code snippets [7, 55, 64], systematically categorize code features into three fundamental dimensions (semantics, format, and implementation details) based on established literature [9, 10, 16, 29, 82], and design mutation operators that comprehensively cover each dimension.”
    Quote in the paperSubstantial claim0 of 8 sources checked
    • cites [7] [7] Raymond P.L. Buse and Westley R. Weimer. 2010. Learning a Metric for Code Readability. IEEE Transactions on Software Engineering 36, 4 (2010), 546-558. doi:10.1109/TSE.2009.70 - source
    • cites [55] [55] Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A simpler model of software readability. In Proceedings of the 8th Working Conference on Mining Software Repositories (Waikiki, Honolulu, HI, USA) (MSR '11) . Association for Computing Machinery, New York, NY, USA, 73-82. doi:10.1145/1985441.1985454 - source
    • cites [64] [64] Simone Scalabrino, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2016. Improving code readability models with textual features. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC) . 1-10. doi:10.1109/ICPC.2016.7503707 - source
    • cites [9] [9] Jialun Cao, Songqiang Chen, Wuqi Zhang, Hau Ching Lo, Yeting Li, and Shing-Chi Cheung. 2025. CodeCleaner: Mitigating Data Contamination for LLM Benchmarking. In Proceedings of the 16th International Conference on Internetware (Internetware '25) . Association for Computing Machinery, New York, NY, USA, 71-83. doi:10.1145/3755881.3755901 - source
    • cites [10] [10] Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu, and Baishakhi Ray. 2022. NatGen: generative pre-training by 'naturalizing' source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022) . Association for Computing Machinery, New York, NY, USA, 18-30. doi:10.1145/3540250.3549162 - source
    • cites [16] [16] Yaniv David, Uri Alon, and Eran Yahav. 2020. Neural reverse engineering of stripped binaries using augmented control flow graphs. Proc. ACM Program. Lang. 4, OOPSLA, Article 225 (Nov. 2020), 28 pages. doi:10.1145/3428293 - source
    • cites [29] [29] Chao Hu, Yitian Chai, Hao Zhou, Fandong Meng, Jie Zhou, and Xiaodong Gu. 2024. How Effectively Do Code Language Models Understand Poor-Readability Code?. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE '24) . Association for Computing Machinery, New York, NY, USA, 795-806. doi:10.1145/3691620.3695072 - source
    • cites [82] [82] Fan Yan and Ming Li. 2021. Towards Generating Summaries for Lexically Confusing Code through Code Erosion. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 . International Joint Conferences on Artificial Intelligence Organization, 3721-3727. doi:10.24963/ijcai.2021/512 Main Track. - source

  • May combine two claims.

    Interpretationown data
    § 4.2 RQ2: Effect of Different Code Features

    This stability suggests superior generalization capabilities and reduced dependence on specific code features.

    “This stability suggests superior generalization capabilities and reduced dependence on specific code features of the powerful LLM for both contest-level and repository-level code generation tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.2 Threats to Validity

    Only a limited subset of LiveCodeBench-v5 and no DevEval tasks may have been included in the training data.

    “Therefore, only a limited subset of LiveCodeBench-v5 (May 2023 to January 2025) and no DevEval tasks (May 2024) may have been included in the training data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 5.1 Statistical Assumptions Validation

    They ground conclusions on Wilcoxon signed-rank test and Bootstrap CI, reporting t-test and LMM only as concordance checks.

    “We therefore ground the conclusions on the Wilcoxon signed-rank test together with the Bootstrap CI on the mean paired difference and report the paired 𝑡 -test and LMM only as concordance checks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation
    § 4.2 RQ2: Effect of Different Code Features

    LLMs can defend against semantic noise caused by incorrect format for Python, while struggling with correct code with irregular format in Java.

    “LLMs can defend against the semantic noise caused by incorrect format for the familiar Python language, while struggling with the correct code with irregular format in Java, which is relatively less familiar to LLMs as reported by previous works [72, 74].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [72] [72] Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 20032-20051. doi:10.18653/v1/2024.emnlp-main.1118 - source
    • cites [74] [74] Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, and Jie M. Zhang. 2026. A Study of LLMs' Preferences for Libraries and Programming Languages. In Findings of the Association for Computational Linguistics: ACL 2026 , Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 331-351. doi:10.18653/v1/2026.findings-acl.15 - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5.2 Threats to Validity

    They extend Python code generation tasks in LiveCodeBench to Java to strengthen generalizability.

    “Furthermore, we extend Python code generation tasks in LiveCodeBench to Java to strengthen the generalizability of our findings.”
    Quote in the paperNot a substantial claim
41 of 41

  • The paper relies on this without support.

    Most prior work treats code examples as monolithic units, without disentangling the contributions of individual code features such as identifier naming, formatting, and implementation details.

    “Second, most prior work treats code examples as monolithic units, without disentangling the contributions of individual code features such as identifier naming, formatting, and implementation details.”

  • The paper relies on this without support.

    Python is the officially supported programming language for code generation tasks in LiveCodeBench and DevEval.

    “Python is the officially supported programming language for code generation tasks in LiveCodeBench and DevEval.”

  • The authors state this as an assumption.

    Ground Truth configuration provides an upper bound of ICL effectiveness on contest-level tasks.

    “This configuration provides a clean setup to mitigate the effect of retrieved document quality, thus serving as a ground to investigate the upper bound of ICL effectiveness on contest-level tasks [77].”

  • The paper relies on this without support.

    The knowledge cutoff dates of the four studied LLMs accurately determine which LeetCode questions could have appeared in their pre-training data, so that questions released after the cutoff are guaranteed unseen.

    “we partitioned the LeetCode questions into two subsets for each model according to the knowledge cutoff date of each LLM: questions released before the cutoff (potentially seen during pre-training) and after the cutoff (guaranteed unseen)”
4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=hSyW5go0v8
  2. 10.5555/3495724.3495883 · DOI
  3. Max Brunsfeld. 2018. Tree-sitter: An Incremental Parsing System for Programming Tools. https://tree- sitter.github.io/ tree- sitter Open-source project. Presented at Strange Loop 2018..
  4. 10.18653/v1/2024.findings- · DOI
  5. 10.18653/v1/N19- · DOI
  6. 10.18653/v1/2024.emnlp- · DOI
  7. 10.18653/v1/2020.findings- · DOI
  8. 10.18653/v1/2026.findings- · DOI
  9. 10.18653/v1/2021.emnlp- · DOI
  10. Howard Levene. 1960. Robust Tests for Equality of Variances. In Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, Ingram Olkin, Sudhish G. Ghurye, Wassily Hoeffding, William G. Madow, and Henry B. Mann (Eds.). Stanford University Press, Stanford, CA, 278–292.
  11. 10.5555/3495724.3496517 · DOI
  12. 10.18653/v1/2024.findings- · DOI
  13. 10.18653/v1/2022.deelio- · DOI
  14. 10.18653/V1/2022.ACL- · DOI
  15. 10.18653/v1/2024.naacl- · DOI
  16. 10.18653/v1/2022.emnlp- · DOI
  17. 10.18653/v1/2024.naacl- · DOI
  18. 10.1111/j.1541- · DOI
  19. 10.1007/978-94- · DOI
  20. 10.1093/biomet/52.3- · DOI
  21. 10.18653/v1/2024.findings- · DOI
  22. 10.18653/v1/2024.emnlp- · DOI
  23. 10.18653/v1/2026.findings- · DOI
  24. 10.18653/v1/2023.acl- · DOI
  25. 10.18653/v1/2021.emnlp- · DOI
  26. 10.18653/v1/2025.findings- · DOI
  27. 10.18653/v1/2023.acl- · DOI
  28. 10.18653/v1/2022.emnlp- · DOI

Authors
Li, D., Chen, S., Cao, J., & Cheung, S.-C.
Affiliations
  • The Hong Kong University of Science and Technology, Hong Kong, China
  • Hong Kong University of Science and Technology (HK)
ORCID
Venue
Proceedings of the ACM on Software Engineering
Publisher
Association for Computing Machinery (ACM)
Published
2026-10-01
Volume
3
Issue
ISSTA
Pages
4214-4239
ISSN
2994-970X (ISSN-L)
DOI
10.1145/3832275
OpenAlex
W4415195398
Semantic Scholar
bcbca208bdf17f1b58348558bba3fa84e6ec8c4d
Cited by
0
References
47
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (92582201)
  • Hong Kong SAR Research Grant Council\/General Research Fund (16206524)
  • Hong Kong SAR Research Grant Council\/Theme-based Research Scheme (T41-517/25-N)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
in-context learning, code generation, large language models, code examples, identifier naming, code formatting, retrieval-augmented learning, reflective learning, variable names

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review