CoChat
Sign Up
PreprintCited by 96Software Engineering ResearchTopic ModelingTeaching and Learning Programming

ReACC: A Retrieval-Augmented Code Completion Framework

Open access
Lu, Shuai, Duan, Nan, Han, Hojae, Guo, Daya, Hwang, Seung-won, Svyatkovskiy, Alexey·2022·arXiv
DOI 10.48550/arxiv.2203.07722arXiv 2203.07722
Open this paper in CoChat — free accountOpen PDF

Abstract

Code completion, which aims to predict the following code token(s) according to the code context, can improve the productivity of software development. Recent work has proved that statistical language modeling with transformers can greatly improve the performance in the code completion task via learning from large-scale source code datasets. However, current approaches focus only on code context within the file or project, i.e. internal context. Our distinction is utilizing "external" context, inspired by human behaviors of copying from the related code snippets when writing code. Specifically, we propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We adopt a stage-wise training approach that combines a source code retriever and an auto-regressive language model for programming language. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.

AI snapshot

Written from the paper’s full text.

TL;DR

ReACC is a retrieval-augmented code completion framework that retrieves lexically and semantically similar code from an external codebase and concatenates it with the unfinished code for a CodeGPT-adapted generator. It reports state-of-the-art performance on the CodeXGLUE code completion benchmark for Python and Java.

Key findings

  • On CodeXGLUE, ReACC-hybrid achieves PY150 perplexity 2.311, exact match 46.26, edit similarity 74.41; Java Corpus perplexity 3.327, exact match 30.70, edit similarity 64.73 (Table 3).
  • On a newly created PY150 test set, ReACC-hybrid reaches exact match 40.15 and edit similarity 72.01, compared with CodeGPT-adapted 38.77/70.07 and X-CodeGPT 39.41/70.97 (Table 4).
  • On CodeNet Python algorithm-domain completion, ReACC-hybrid reaches exact match 64.74 and edit similarity 84.93 vs CodeGPT-adapted 46.38/74.10; the paper reports 10% and 18% absolute improvement in edit similarity and exact match, respectively (Table 5).
  • In zero-shot code clone detection on CodeNet, ReACC-retriever gets Python MAP@100 9.60 / precision 27.04 and Java 9.31 / 27.55; BM25 gets Python 10.32/23.17 and Java 8.67/25.85 (Table 2). Ablation shows query truncation is the most essential factor, and identifier renaming contributes more than dead code insertion (Table 6).

Methods

  • ReACC combines a hybrid retriever (BM25 sparse + dense dual-encoder initialized from GraphCodeBERT, contrastively pretrained with in-batch negatives, data augmentation via identifier renaming and dead-code insertion, API sequence input, random query truncation) and a CodeGPT-adapted decoder-only generator; the retrieved next fragment \(c'_s\) is concatenated with context \(x\).
  • Retriever pretraining uses CodeSearchNet: 1.6M Java methods and 1.2M Python functions. Evaluation uses CodeXGLUE PY150 and GitHub Java Corpus for completion, and CodeNet for zero-shot clone detection; the completion retrieval database is the training set of PY150/Java Corpus.
  • Metrics: token-level perplexity; line-level exact match and edit similarity; clone detection MAP@K and precision at 1. Hybrid score is \(\text{sim}(q,c) + \alpha \cdot \text{BM25}(q,c)\) with \(\alpha = 0.9\).

Limitations

  • Paper states the Java Corpus retrieval database is much smaller (12,934 training files), making retrieval harder and improvements smaller than on PY150.
  • Paper states that for token-level completion, retrieval is not applied at each step; it retrieves after predicting the first 100 tokens for efficiency.
  • Evident from the text: evaluation is limited to Python and Java; the GitHub Copilot comparison is qualitative examples only, not a systematic benchmark; the dense retriever relies on semantic-preserving transformations for positives because labeled similar-code pairs are rare.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.07722

Cites

81 of 81 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 12 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § Abstract

    Current code completion approaches focus only on code context within the file or project (internal context).

    “However, current approaches focus only on code context within the file or project, i.e. internal context.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Liu et al., 2020 source

      The Introduction (page unavailable) describes existing completion models as using source-code sequences and considering tokens in the source-code file. It does not establish that current approaches use *only* context within the file or project, or define that scope as “internal context.” (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Clement et al., 2021 source

      Section 2 describes code completion using context from the source file, and Section 5.2 contrasts a method-completion approach using only the focal method with one using file-level context. These passages support the use of internal context in the approaches discussed, but they do not establish that current code-completion approaches use only file- or project-internal context. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Findingown data
    § 5.4 Results

    Among the three retriever configurations in ReACC, the hybrid retriever performs best on almost all metrics except exact match on the new PY150 test set.

    “Among three configurations of the retriever in ReACC, hybrid retriever performs best on almost all metrics except the exact match score in the new test set of PY150.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 5.5 Analysis

    When fragment alignment is removed from the generator, performance suffers slightly.

    “When fragment alignment is removed from generator, i.e. using the retrieved code snippet itself for generator, performance suffers slightly.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 5.5 Analysis

    Comparing the two semantic-preserving transformations, identifier renaming contributes more than dead code insertion.

    “Comparing the two semanticpreserving transformations, identifier renaming contributes more than dead code insertion.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Abstract

    ReACC achieves state-of-the-art performance on the CodeXGLUE benchmark.

    “achieving a state-of-the-art performance on CodeXGLUE benchmark.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • May combine two claims. 8
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 5.5 Analysis

    Codex is more powerful than CodeGPT because it is a large-scale pretrained model trained on all source codes in GitHub based on GPT-3.

    “It worth noting that Codex is more powerful than CodeGPT since it is a large-scale pretrained model that is trained on all source codes in GitHub based on GPT-3 (Brown et al., 2020).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Brown et al., 2020 source

      Section 2.1 (p. 8) describes GPT-3 as a large model, but does not identify Codex as a GPT-3-based model. Searches for Codex, CodeGPT, and GitHub source-code training found no passage addressing their comparison or Codex’s training data. The claim therefore cannot be verified from this paper. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 5.4 Results

    BM25 shows comparable results with dense retriever and even performs better in perplexity and exact match metrics.

    “Another finding is that BM25 shows comparable results with dense retriever and even performs better in perplexity and exact match metrics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.5 Analysis

    ReACC significantly outperforms CodeGPT-adapted on the CodeNet Python dataset by 10% absolute improvement in edit similarity and 18% absolute improvement in exact match.

    “Table 5 reveals that ReACC significantly outperforms CodeGPTadapted in CodeNet by 10% and 18% absolute improvement in edit similarity and exact match, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 5.4 Results

    ReACC (with hybrid retriever) outperforms other baselines and achieves state-of-the-art performance on the CodeXGLUE code completion benchmark.

    “ReACC framework with the hybrid retriever outperforms consistently than other baselines on all datasets”
    • Also stated in: § 6 Conclusion · plain
      “On the CodeXGLUE benchmark, ReACC achieves a state-of-the-art performance in the code completion task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § 2.2 Retrieval on code intelligence

    In the code completion scenario, using code as query, particularly partial code, is a more difficult task because there are few labeled data with semantically similar code pairs and semantics in query is incomplete.

    “In the code completion scenario, however, we focus on using code as query, particularly partial code, which is a more difficult task since there are few labeled data with semantically similar code pairs and in partial code search, semantics in query is incomplete.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction

    The cross-encoder model has high computational complexity.

    “since the cross-encoder model has a high computational complexity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction

    The dense retriever can capture syntactic and semantic information by mapping a code snippet to a dense vector.

    “The dense retriever, on the contrary, can capture syntactic and semantic information by mapping a code snippet to a dense vector.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning
    § 1 Introduction

    The sparse retriever can capture lexical information and is sensitive to the names of code identifiers.

    “The sparse retriever can capture lexical information and is sensitive to the names of code identifiers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § 3.2 Retriever

    Searching semantically equivalent code requires extra code compilation and execution costs, which is unrealistic in a large database.

    “Searching semantically equivalent code requires extra code compilation and execution costs (Massalin, 1987; Churchill et al., 2019), which is unrealistic in a large database.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Massalin, 1987 source
    • cites Churchill et al., 2019 source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 Introduction

    A software repository usually contains 7-23% cloned codes.

    “leading a software repository usually containing 7-23% cloned codes (Svajlenko and Roy, 2015).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Svajlenko and Roy, 2015 source

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction

    The labeled data containing similar code pairs is rare.

    “As the labeled data containing similar code pairs is rare”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning
    § 4.3 Experiment Setup

    The code clone detection experiment is implemented in partial search way, which is ideally adapted to code completion scenarios.

    “We implement the code clone detection experiment in the partial search way, which is ideally adapted to code completion scenarios as it accepts a partial program as a query while maintaining the same goal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.2 Retriever

    Dead code insertion is one of the most common techniques for code obfuscation.

    “Dead code insertion is one of the most common techniques for code obfuscation (You and Yim, 2010)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites You and Yim, 2010 source

  • The quote only partly states this claim.

    Interpretationown data
    § 5.5 Analysis

    ReACC is more effective in a specific domain (algorithm domain) than in general domains.

    “According to the findings, ReACC is more effective in a specific domain.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: PY150 and Java Corpus datasets are extracted from GitHub repositories distributed in a wide domain.
    • Rests on an assumption: The CodeNet dataset can be considered in the algorithm domain.

  • May combine two claims.

    Interpretationreasoning
    § 3.2 Retriever

    Because programmers tend to copy tokens directly, a code with distinct semantics but substantial lexical similarity can help with code completion.

    “Because programmers tend to copy tokens directly, a code with distinct semantics but substantial lexical similarity can help with code completion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3 Generator

    CodeGPT-adapted has shown promising results in the code completion task in CodeXGLUE benchmark.

    “CodeGPT-adapted has shown promising results in the code completion task in CodeXGLUE benchmark (Lu et al., 2021)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lu et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 5.3 Experiment Setup

    The new dataset built from PY150 retains the train/valid/test split.

    “create a new dataset that retains the train/valid/test split, as seen in Table 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 5.3 Experiment Setup

    Except for X-CodeGPT, all other baseline models were fine-tuned with the same settings.

    “Except for X-CodeGPT, all other baseline models are fine-tuned with the same settings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 6 Conclusion

    Our method can adopt any architecture that can perform code completion as the generator.

    “Our method can adopt any architecture that can perform code completion as the generator.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 5.3 Experiment Setup

    The retrieval database for test did not use the contrastive pre-training corpus (CodeSearchNet) to avoid duplication between CodeXGLUE and CodeSearchNet, as both are extracted from GitHub.

    “We don't use the contrastive pre-training corpus (i.e., CodeSearchNet) in order to avoid the duplication between CodeXGLUE and CodeSearchNet as they are both extracted from GitHub.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: CodeXGLUE and CodeSearchNet are both extracted from GitHub, so using CodeSearchNet as retrieval database would risk duplication.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 5.3 Experiment Setup

    The hybrid retriever's alpha was set to 0.9 based on dev set results for both PY150 and Java Corpus datasets.

    “let α = 0 . 9 based on the results on dev set for both PY150 and Java Corpus datasets.”
    Quote in the paperNot a substantial claim
21 of 21

  • The paper relies on this without support.

    Identifier names typically convey meanings for humans.

    “identifier names typically convey the meanings for humans”

  • The paper relies on this without support.

    PY150 and Java Corpus datasets are extracted from GitHub repositories distributed in a wide domain.

    “Both PY150 and Java Corpus datasets are extracted from GitHub repositories which are distributed in a wide domain.”

  • The paper relies on this without support.

    The CodeNet dataset can be considered in the algorithm domain.

    “which can be considered in algorithm domain.”

  • The paper relies on this without support.

    CodeXGLUE and CodeSearchNet are both extracted from GitHub, so using CodeSearchNet as retrieval database would risk duplication.

    “as they are both extracted from GitHub.”

  • The paper relies on this without support.

    eWASH requires codes parsed into ASTs, but CodeXGLUE codes are tokenized and cannot be parsed.

    “Since eWASH requires codes parsed into ASTs but codes in CodeXGLUE have been tokenized and cannot be parsed”

  • The paper relies on this without support.

    Some people frequently write code in more specific domains, such as data mining/pattern recognition or algorithm domains.

    “As some people frequently write codes in a more specific domain, e.g., data mining/pattern recognition domain for Kaggle 3 users, algorithm domain for ACM community, etc.”
6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
  2. Nils Reimers and Iryna Gurevych. 2019. Sentencebert: Sentence embeddings using siamese bertnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992.

Cited by

Sorted by citations

Related

Authors
Lu, S., Duan, N., Han, H., Guo, D., Hwang, S.-W., & Svyatkovskiy, A.
Affiliations
  • Microsoft Research Asia (China) (CN)
  • Seoul National University (KR)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
6227-6240
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2203.07722
arXiv
2203.07722
OpenAlex
W4226350810
Cited by
96
References
70
Citations by year
2026: 18 · 2025: 35 · 2024: 22 · 2023: 16 · 2022: 5
FWCI
17.43
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Seoul National University Open Repository (Seoul National University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Microsoft Research
  • Microsoft Research Asia
  • Institute for Information and Communications Technology Promotion
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
code completion, retrieval-augmented generation, external context, code search, programming language models, Python, Java, CodeXGLUE

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review