Using pre-trained language models to resolve textual and semantic merge conflicts (experience paper)
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Program merging is standard practice when developers integrate their individual changes to a common code base. When the merge algorithm fails, this is called a merge conflict. The conflict either manifests as a textual merge conflict where the merge fails to produce code, or as a semantic merge conflict where the merged code results in compiler errors or broken tests. Resolving these conflicts for large code projects is expensive because it requires developers to manually identify the sources of conflicts and correct them. In this paper, we explore the feasibility of automatically repairing merge conflicts (both textual and semantic) using k-shot learning with pre-trained large neural language models (LM) such as GPT-3. One of the challenges in leveraging such language models is fitting the examples and the queries within a small prompt (2048 tokens). We evaluate LMs and k-shot learning for both textual and semantic merge conflicts for Microsoft Edge. Our results are mixed: on one-hand, LMs provide the state-of-the-art (SOTA) performance on semantic merge conflict resolution for Edge compared to earlier symbolic approaches; on the other hand, LMs do not yet obviate the benefits of special purpose domain-specific languages (DSL) for restricted patterns for program synthesis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1×
“Language models have been used for sentiment analysis[18], question and answering[26], or even code generation[6].”
MethodMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 1×
“Language models have been used for sentiment analysis[18], question and answering[26], or even code generation[6].”
MethodOn the Nature of Merge Conflicts: A Study of 2,731 Open Source Java Projects Hosted by GitHub
Ghiotto, G., Murta, L., Barros, M., & van der Hoek, A. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2871083 · Cited by 67 · cited 4×
“Recent studies [7, 9] have shown that such a bad merge in the code delayed the development cycle or caused damage by simply leaving bugs in the code.”
BackgroundResultInfluentialProgram merge conflict resolution via neural transformers
Svyatkovskiy, A., Fakhoury, S., Ghorbani, N., Mytkowicz, T., Dinella, E., Bird, C., Jang, J., Sundaresan, N., & Lahiri, S. K. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549163 · Cited by 38
Self-citation
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Automatic Root Cause Analysis via Large Language Models for Cloud Incidents
Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., et al. (2024) · Proceedings of the Nineteenth European Conference on Computer Systems · doi:10.1145/3627703.3629553 · Cited by 190
“Zhang et al. [55] proposes to use prompting for LLM to improve code version control.”
MethodLarge Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
Assess and Summarize: Improve Outage Understanding with Large Language Models
Jin, P., Zhang, S., Ma, M., Li, H., Kang, Y., Li, L., Liu, Y., Qiao, B., Zhang, C., Zhao, P., et al. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613891 · Cited by 50
“In recent years, the rise of LLM has brought new opportunities to the field of software engineering [2, 13, 25, 26, 30].”
BackgroundMethodCan GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
“Materials used in this study, such as the protocols, GPT-4 generated data, and exact prompts, are available in the supplemental materials [5].”
Method
Metadata
- Authors
- Zhang, J., Mytkowicz, T., Kaufman, M., Piskac, R., & Lahiri, S. K.
- Venue
- Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2022-07-18
- Pages
- 77-88
- OpenAlex
- W4285490371
- Cited by
- 39
- References
- 4
- FWCI
- 6.52
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.