The DOI matches its DataCite record; 4 of the 4 references it cites were found in a registry or an index.
RepoFusion: Training Code Models to Understand Your Repository
Abstract
Despite the huge success of Large Language Models (LLMs) in coding assistants like GitHub Copilot, these models struggle to understand the context present in the repository (e.g., imports, parent classes, files with similar names, etc.), thereby producing inaccurate code completions. This effect is more pronounced when using these assistants for repositories that the model has not seen during training, such as proprietary software or work-in-progress code projects. Recent work has shown the promise of using context from the repository during inference. In this work, we extend this idea and propose RepoFusion, a framework to train models to incorporate relevant repository context. Experiments on single-line code completion show that our models trained with repository context significantly outperform much larger code models as CodeGen-16B-multi ($\sim73\times$ larger) and closely match the performance of the $\sim 70\times$ larger StarCoderBase model that was trained with the Fill-in-the-Middle objective. We find these results to be a novel and compelling demonstration of the gains that training with repository context can bring. We carry out extensive ablation studies to investigate the impact of design choices such as context type, number of contexts, context length, and initialization within our framework. Lastly, we release Stack-Repo, a dataset of 200 Java repositories with permissive licenses and near-deduplicated files that are augmented with three types of repository contexts. Additionally, we are making available the code and trained checkpoints for our work. Our released resources can be found at \url{https://huggingface.co/RepoFusion}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Retrieve-and-edit [100], RepoFusion [243], and EDITAS [244] employ multiple encoders for retrieved contents or edit sequences, then fuse the information for subsequent decoder.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [243]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 1×
“The generated code can often be challenging to understand or debug, resulting in developers spending significant time editing and revising the code [33, 23, 3, 5].”
BackgroundEfficient Training of Language Models to Fill in the Middle
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14255 · Cited by 47
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 1×
“The generated code can often be challenging to understand or debug, resulting in developers spending significant time editing and revising the code [33, 23, 3, 5].”
BackgroundDo Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 1×
“Moreover, excessive dependence on these models can result in situations where users overlook errors in their code [1] or become overly self-assured, leading to the introduction of mistakes [29].”
Background
Cited by
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240
AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology
Nguyen, M. H., Chau, T. P., Nguyen, P. X., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11912 · Cited by 21
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
FullStack Bench: Evaluating LLMs as Full Stack Coders
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.00535 · Cited by 2
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Liu, X., Lan, B., Hu, Z., Liu, Y., Zhang, Z., Wang, F., Shieh, M., & Zhou, W. (2024) · arXiv · doi:10.48550/arxiv.2408.03910 · Cited by 1
Metadata
- Authors
- Shrivastava, D., Kocetkov, D., de Vries, H., Bahdanau, D., & Scholak, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2306.10998
- References
- 4
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RepoFusion: Training Code Models to Understand Your Repository. https://app.cochat.ai/papers/10.48550/arxiv.2306.10998
- The paper
- Shrivastava, D., Kocetkov, D., de Vries, H., Bahdanau, D., & Scholak, T. (2023). RepoFusion: Training Code Models to Understand Your Repository. arXiv. https://doi.org/10.48550/arxiv.2306.10998
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.