The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Abstract
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [24]
In the citing paper
“However, the quality of code produced by Copilot was lower.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“The study concluded that Copilot tended to result in more lines of code than with the human-based pair-programming in the same amount of time.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Meanwhile, other studies which have included human developers [21-23] have not considered security.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [11]
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
How We Refactor, and How We Know It
Murphy-Hill, E., Parnin, C., & Black, A. P. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.41 · Cited by 547
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
AI-Based Software Defect Predictors: Applications and Benefits in a Case Study
Tosun, A., Bener, A., & Kale, R. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i2.18807 · Cited by 29
Research works printed but not linked: 7
- Ritchie Schacher Adam Archer and Scott Will . [n.d.]. Program in Pairs . Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/ Ritchie Schacher Adam Archer and Scott Will. [n.d.]. Program in Pairs. Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/
- Scott Carey . 2021 . Developers react to GitHub Copilot . Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html Scott Carey. 2021. Developers react to GitHub Copilot. Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html
- Matteo Ciniselli , Nathan Cooper , Luca Pascarella , Antonio Mastropaolo , Emad Aghajani , Denys Poshyvanyk , Massimiliano Di Penta, and Gabriele Bavota . 2021 . An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering ( 2021). Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering (2021).
- Gail C Murphy , Mik Kersten , and Leah Findlater . 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 ( 2006 ), 76--83. Gail C Murphy, Mik Kersten, and Leah Findlater. 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 (2006), 76--83.
- Hammond Pearce , Benjamin Tan , Baleegh Ahmad , Ramesh Karri , and Brendan Dolan-Gavitt . 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 ( 2021 ). Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 (2021). · arXiv
- Julian Aron Prenner and Romain Robbes . 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 ( 2021 ). Julian Aron Prenner and Romain Robbes. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 (2021). · arXiv
- Dominik Sobania , Martin Briesch , and Franz Rothlauf . 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 ( 2021 ). Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 (2021). · arXiv
Cited by
Sorted by citations
Items 101–131 of 131
Miranda, P. C., & de Andrade, M. H. S. (2024) · doi:10.5753/cbsoft_estendido.2024.4100
A Retrospective on How Developers Seek, Relate, and Collect Information About Code
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3534071
“One of the key challenges surfacing in these recent studies of LLMs are that no matter how good they get, they are wrong often enough that they put even greater pressure on program comprehension, to detect and overcome all of its failures [13].”
Why Personalizing Deep Learning-Based Code Completion Tools Matters
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725732
Ramachandran, R. (2025) · International Conference on Software Engineering and Computer Science (CSECS) · doi:10.1109/csecs64665.2025.11009821
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
OR for Everyone: Solving OR Problems as Non-experts with Generative AI
Maurischat, J., Bogs, S., Walther, G., & Kirchhof, O. (2025) · Lecture notes in operations research · doi:10.1007/978-3-031-92575-7_18
Determining Code Proficiency Levels from Python Textbooks
Rojpaisarnkit, R., Robles, G., Gonzalez-Barahona, J. M., Matsumoto, K., & Kula, R. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769864 · Cited by 1
“Other studies note that although LLMs provide convenience, they occasionally produce code that developers struggle to understand [19, 54].”
Automated Update of Android Deprecated API Usages With Large Language Models
Mahmud, T., Duan, B., Che, M., Yasmin, A., Ngu, A. H. H., & Yang, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3627897
“Other studies are looking at how LLMs change the way we make software [34], [49], help developers work faster [19], [78], and find security problems [57].”
Costa, A. C. M. P., Fantini, J. B. O., de Souza, É. F., Silva, G. B. E., dos Santos, L. B. R., Felizardo, K. R., & Meinerz, G. V. (2025) · Simpósio Brasileiro de Qualidade de Software (SBQS) · doi:10.5753/sbqs.2025.13875
“Specifically, studies [46] (ID 93) and [47] (ID 95) evaluated GitHub Copilot as an automated partner in pair programming, assessing its effectiveness in replicating the supportive role typically performed by a human collaborator, and highlighting its potential to transform collaborative development…”
Revolutionizing Software Development with Machine Learning: A Comprehensive Review
Kaur, C., Kapur, S., Vandana, V., & Agrawal, T. (2025) · IEEE Space, Aerospace and Defence Conference (SPACE) · doi:10.1109/space65882.2025.11171421
Hein, D. K. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-14518-5_12
Yapay Zeka Destekli Kod Üretimi İle İnsan Yazımı Kodların Karşılaştırılması: Python Örneği
EYGÜ, S., & Keskinkılıç, M. (2026) · Bilgisayar bilimleri ve mühendisliği dergisi. · doi:10.54525/bbmd.1781375
Matejić, T., Mihailović, Đ., & Dukić, A. (2025) · Limes plus. · doi:10.69899/limes-plus25222046m
Avaliação de Qualidade de Código Java gerado por Large Language Models
Oliveira, M. T., Silveira, P. M. O., & de Andrade, M. H. S. (2025) · Escola Regional de Engenharia de Software (ERES) · doi:10.5753/eres.2025.16845
Yardım, A., Serafini, R., Jost, N., Ortloff, A.-M., Speckels, J. G., & Naiakshina, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791693
Engineering Students' Usage and Perceptions of GitHub Copilot in Open-Source Projects
Rani, N., Munnangi, J. R., Spangler, A. M., & Honeycutt, D. R. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59352
de Oliveira, L. A. F., & de Oliveira, C. C. (2026) · Research Society and Development · doi:10.33448/rsd-v15i4.50966 · Cited by 1
Ghodke, P., Kumbhar, B., & Sadam, R. (2026) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3796563.3796589
“Other research shows that developers actually spend a lot of time fixing bugs in AI-generated code [5], which means developers need to learn the right way to use these tools to actually get the productivity benefits.”
The Energy-Throughput Trade-Off in Lossless-Compressed Source Code Storage
Ferragina, P., & Tosoni, F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c67878.2026.00027
Learning Programming with Generative AI: A Pre-University Pilot Study
Yong, S.-T., Chin, L.-Y., Matthews, R., & Ramayah, B. (2026) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet69664.2026.11561564
AI Dependency’s Influence on Computational Thinking: Implications for Augmented Education
Crosby, M. E., Auernheimer, B., Doliashvili, M., Ogata, B., Mosier, S., & Ogawa, M.-B. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29548-4_1
From Unfinished to Done: Bridging the MDE Implementation Gap with Constrained LLMs
Kirchhof, J. C., Netz, L., Mertens, D., & Rumpe⋆, B. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830316
“Although LLMs are rapidly evolving, their unconstrained use correlates with increased code churn and decreased code quality [28, 32].”
Huang, Z., Liu, C., Wang, Z., Cui, X., Deng, X., & Xu, M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3830237
Software Engineering Reliability in the Era of Generic and Agentic AI
Nwodo, K., & Stavrou, A. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18585-3_9
Beyond Coding: How Students Use Generative AI Across the Full Software Development Lifecycle
Matti, S., Szendrei, R., & Pintér, B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32732-1_14
Dugyala, R. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11689630
Pérez-Morera, D., Quesada–López, C., Lara, A., González, A., & Sánchez, S. (2026) · IEEE Jornadas Costarricenses de Investigación en Computación e Informática (JoCICI) · doi:10.1109/jocici69279.2026.11701712
“This can also help with the logistical complexity of getting collaborators at the same time at the same place [12]–[14].”
InfluentialCopilot-Driven Development in Practice: Building a Monolith and Migrating to Microservices
Bitgin, A. K., Demir, O., Ulurak, E., Kıraç, F., & Sözer, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39236-7_33
Learning Outcomes, Human–AI Trust, and Behavioral Patterns in GenAI-Assisted Pair ProgrammingPreprint
Sun, D., Xu, F., Shi, J., & Yang, L. (2026) · Research Square · doi:10.21203/rs.3.rs-11139158/v1
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · arXiv · doi:10.48550/arxiv.2404.10100
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
Background
Metadata
- Authors
- Imai, S.
- Affiliations
- Colby College (US)
- Corresponding author
- Imai, Saki
- ORCID
- Imai, Saki
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 319-321
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4312763760
- Cited by
- 116
- References
- 3
- Citations by year
- 2026: 23 · 2025: 44 · 2024: 39 · 2023: 10
- FWCI
- 20.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, pair programming, AI-assisted programming, code quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
