The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Abstract
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [24]
In the citing paper
“However, the quality of code produced by Copilot was lower.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“The study concluded that Copilot tended to result in more lines of code than with the human-based pair-programming in the same amount of time.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Meanwhile, other studies which have included human developers [21-23] have not considered security.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [11]
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
How We Refactor, and How We Know It
Murphy-Hill, E., Parnin, C., & Black, A. P. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.41 · Cited by 547
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
AI-Based Software Defect Predictors: Applications and Benefits in a Case Study
Tosun, A., Bener, A., & Kale, R. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i2.18807 · Cited by 29
Research works printed but not linked: 7
- Ritchie Schacher Adam Archer and Scott Will . [n.d.]. Program in Pairs . Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/ Ritchie Schacher Adam Archer and Scott Will. [n.d.]. Program in Pairs. Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/
- Scott Carey . 2021 . Developers react to GitHub Copilot . Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html Scott Carey. 2021. Developers react to GitHub Copilot. Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html
- Matteo Ciniselli , Nathan Cooper , Luca Pascarella , Antonio Mastropaolo , Emad Aghajani , Denys Poshyvanyk , Massimiliano Di Penta, and Gabriele Bavota . 2021 . An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering ( 2021). Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering (2021).
- Gail C Murphy , Mik Kersten , and Leah Findlater . 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 ( 2006 ), 76--83. Gail C Murphy, Mik Kersten, and Leah Findlater. 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 (2006), 76--83.
- Hammond Pearce , Benjamin Tan , Baleegh Ahmad , Ramesh Karri , and Brendan Dolan-Gavitt . 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 ( 2021 ). Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 (2021). · arXiv
- Julian Aron Prenner and Romain Robbes . 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 ( 2021 ). Julian Aron Prenner and Romain Robbes. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 (2021). · arXiv
- Dominik Sobania , Martin Briesch , and Franz Rothlauf . 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 ( 2021 ). Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 (2021). · arXiv
Cited by
Sorted by citations
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
Metadata
- Authors
- Imai, S.
- Affiliations
- Colby College (US)
- Corresponding author
- Imai, Saki
- ORCID
- Imai, Saki
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 319-321
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4312763760
- Cited by
- 116
- References
- 3
- Citations by year
- 2026: 23 · 2025: 44 · 2024: 39 · 2023: 10
- FWCI
- 20.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, pair programming, AI-assisted programming, code quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is GitHub copilot a substitute for human pair-programming?: an empirical study. https://app.cochat.ai/papers/10.1145/3510454.3522684
- The paper
- Imai, S. (2022). Is GitHub copilot a substitute for human pair-programming?: an empirical study. Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. https://doi.org/10.1145/3510454.3522684
