The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Abstract
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [24]
In the citing paper
“However, the quality of code produced by Copilot was lower.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“The study concluded that Copilot tended to result in more lines of code than with the human-based pair-programming in the same amount of time.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Meanwhile, other studies which have included human developers [21-23] have not considered security.”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [21]
In the citing paper
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [11]
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
How We Refactor, and How We Know It
Murphy-Hill, E., Parnin, C., & Black, A. P. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.41 · Cited by 547
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452
AI-Based Software Defect Predictors: Applications and Benefits in a Case Study
Tosun, A., Bener, A., & Kale, R. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i2.18807 · Cited by 29
Research works printed but not linked: 7
- Ritchie Schacher Adam Archer and Scott Will . [n.d.]. Program in Pairs . Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/ Ritchie Schacher Adam Archer and Scott Will. [n.d.]. Program in Pairs. Retrieved December 31, 2021 from https://www.ibm.com/garage/method/practices/code/practice_pair_programming/
- Scott Carey . 2021 . Developers react to GitHub Copilot . Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html Scott Carey. 2021. Developers react to GitHub Copilot. Retrieved December 31, 2021 from https://www.infoworld.com/article/3624688/developers-react-to-github-copilot.html
- Matteo Ciniselli , Nathan Cooper , Luca Pascarella , Antonio Mastropaolo , Emad Aghajani , Denys Poshyvanyk , Massimiliano Di Penta, and Gabriele Bavota . 2021 . An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering ( 2021). Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering (2021).
- Gail C Murphy , Mik Kersten , and Leah Findlater . 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 ( 2006 ), 76--83. Gail C Murphy, Mik Kersten, and Leah Findlater. 2006. How are Java software developers using the Elipse IDE? IEEE software 23, 4 (2006), 76--83.
- Hammond Pearce , Benjamin Tan , Baleegh Ahmad , Ramesh Karri , and Brendan Dolan-Gavitt . 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 ( 2021 ). Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2021. Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs? arXiv preprint arXiv:2112.02125 (2021). · arXiv
- Julian Aron Prenner and Romain Robbes . 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 ( 2021 ). Julian Aron Prenner and Romain Robbes. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. arXiv preprint arXiv:2111.03922 (2021). · arXiv
- Dominik Sobania , Martin Briesch , and Franz Rothlauf . 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 ( 2021 ). Dominik Sobania, Martin Briesch, and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv preprint arXiv:2111.07875 (2021). · arXiv
Cited by
Sorted by citations
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
Prompting Is All You Need: Automated Android Bug Replay with Large Language Models
Feng, S., & Chen, C. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608137 · Cited by 245
Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160
“The online help-seeking behavior of developers is also changing along with other behavior shifts.”
BackgroundTaking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 247
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundKosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024) · Mathematics · doi:10.3390/math12050629 · Cited by 140
On the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 100
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 125
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70
“Positioned as potential AI pair programmers (Moradi Dakhel et al. 2023; Imai 2022; Bird et al. 2022) in software projects, LLM-based code generators are poised to play a crucial role in the quality of the overall software project.”
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 78
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2023) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3605731.3605886 · Cited by 52
Hajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 87
“A prominent example is GitHub Copilot [9], an AI pair programmer based on OpenAI Codex [5], [10] that is already used by more than a million developers [11].”
MethodWeisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
“…language models that have been exposed to source code in their pre-training have demonstrated a high degree of aptitude in performing a variety of tasks: converting natural language to code (e.g. [2, 16, 19, 56]), converting code to natural language documentation (e.g. [16, 31]) or explanations…”
Navigating software development in the ChatGPT and GitHub copilot era
France, S. L. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.05.009 · Cited by 38
Unveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38
“These investigations include studies on the impact of the recommender on developers [71, 93, 109, 118], robustness assessment [82, 113, 115], empirical evaluation of correctness [88, 115], and scrutiny of security aspects [47, 92, 97].”
BackgroundResultPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 41
“…research comparing the quality of code generated by Copilot with human-written code indicates that while Copilot increases productivity in terms of the number of lines of code, the quality of its output is generally inferior, with higher failure rates on medium and complex tasks [14], [75].”
BackgroundZhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30
An Industry Case Study on Adoption of AI-based Programming Assistants
Davila, N., Wiese, I., Steinmacher, I., da Silva, L. L., Kawamoto, A. L. S., Favaro, G. J. P., & Nunes, I. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3643648 · Cited by 48
Demystifying practices, challenges and expected features of using GitHub copilot
Zhang, B., Liang, P., Zhou, X., Ahmad, A., & Waseem, M. (2023) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194023410048 · Cited by 26
Detecting Code Smells using ChatGPT: Initial Insights
Silva, L. L., da Silva, J. R., Montandon, J. E., Andrade, M., & Valente, M. T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690742 · Cited by 25
Enhancing Exploratory Testing by Large Language Model and Knowledge Graph
Su, Y., Liao, D., Xing, Z., Huang, Q. Y., Xie, M., Lu, Q., & Xu, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639157 · Cited by 30
Large language model evaluation for high‐performance computing software development
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8269 · Cited by 29
Automatic Library Migration Using Large Language Models: First Results
Almeida, A., Xavier, L., & Valente, M. T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690746 · Cited by 18
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18
Song, F., Agarwal, A., & Wen, W. (2024) · SSRN Electronic Journal (preprint) · doi:10.2139/ssrn.4856935 · Cited by 30
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 26
“Recent studies have begun to explore interaction modes of human-AI pair programming [4, 57, 61] and related performance outcomes [22, 25, 57] from case studies to experimental studies.”
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 38
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 18
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
Agossah, A., Krupa, F., Silva, M. P. D., & Callet, P. L. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3603362 · Cited by 14
Impacts of the Usage of Generative Artificial Intelligence on Software Development Process
Santos, P. D. O., Figueiredo, A. C., Moura, P., Diirr, B., Alvim, A. C. F., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658337 · Cited by 17
“GitHub Copilot 2 é uma ferramenta de desenvolvimento de software que oferece geração de linhas código, trechos de código ou até programas inteiros baseados em código e comentários exis-tentes [20].”
BackgroundRethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 15
“However, other empirical studies (Imai 2022) reveal that Copilot results in higher productivity but with lower quality of code.”
A disruptive research playbook for studying disruptive innovations
Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3678172 · Cited by 15
Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot
OBrien, D., Biswas, S., Imtiaz, S. M., Abdalkareem, R., Shihab, E., & Rajan, H. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639176 · Cited by 12
Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace
Butler, J., Suh, J., Haniyur, S., & Hadley, C. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00034 · Cited by 12
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715108 · Cited by 11
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 19
“Also, when it comes to comparison between students’ generated code and AI-CA’s generated code, the AI-CA’s output quality is often lower in quality when compared to human-written code, especially for complex tasks (Imai, 2022 ; Vaithilingam et al., 2022 ).”
ChatGPT vs. Stack Overflow: An Exploratory Comparison of Programming Assistance Tools
Liu, J., Tang, X., Li, L., Chen, P., & Liu, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c60940.2023.00105 · Cited by 15
Nikolaidis, N., Flamos, K., Gulati, K., Feitosa, D., Ampatzoglou, A., & Chatzigeorgiou, A. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c62648.2024.00018 · Cited by 11
Retracted: Autonomous Crowdsensing: Operating and Organizing Crowdsensing for Sensing Automation
Wu, W., Yang, W., Li, J., Zhao, Y. B., Zhu, Z., Chen, B., Qiu, S., Peng, Y., & Wang, F. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3355508 · Cited by 12
“Moreover, LLMs can automatically generate code snippets or function structures based on task descriptions or requirements [32].”
BackgroundRetractedDeveloper Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
Stray, V., Brandtzæg, E. G., Wivestad, V., Barbala, A., & Moe, N. B. (2026) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2026.880 · Cited by 9
“GenAI tools assist developers by providing intelligent code suggestions, automating repetitive tasks, enhancing problem-solving capabilities (Dohmke, 2023; Imai, 2022), and improved coding practices (Zhang et al., 2023).”
Case study: using AI-assisted code generation in mobile teams
Vasiliniuc, M.-S., & Groza, A. P. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398656 · Cited by 18
Human-Machine Teaming and Team Effectiveness in AI Tools for Software Engineering
Rauf, I., Sharp, H., López, T., & Michel Wermelinger, B. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00017 · Cited by 8
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 19
Beyond Snippet Assistance: A Workflow-Centric Framework for End-to-End AI-Driven Code Generation
Sonkin, V., & Tudose, C. (2025) · Computers · doi:10.3390/computers14030094 · Cited by 7
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 14
Generative AI for Code Generation: Software Reuse Implications
Kapitsaki, G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66459-5_3 · Cited by 7
Serafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 6
“This reflects findings by Imai [31], who noted that while tools like GitHub Copilot increase productivity by offering a starting point, much of the generated code is later discarded due to quality concerns.”
Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 10
Dockerfile Flakiness: Characterization and Repair
Shabani, T., Nashid, N., Alian, P., & Mesbah, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00238 · Cited by 14
“Leveraging the ability of LLMs to solve programming tasks across different domains [23]–[33], and the effectiveness of retrieval-augmented generation (RAG) techniques [34]–[36], we pro-pose F LAKI D OCK , an automated approach using LLMs to repair Dockerfile flakiness.”
An Analysis of the Costs and Benefits of Autocomplete in IDEs
Jiang, S., & Coblenz, M. J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660765 · Cited by 8
“Liang et al. [21] also found that Copilot reduces keystrokes but that participants found it potentially annoying and unhelpful in brainstorming potential solutions.”
Exploring the Productivity Impacts of Generative AI in Organizations
Semujanga, B., & Mikalef, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72234-9_9 · Cited by 8
Batista, S. S., Branco, B., Castro, O., & Avelino, G. A. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701673 · Cited by 6
Measuring the Runtime Performance of C++ Code Written by Humans Using Github Copilot
Erhabor, D., Udayashankar, S., Nagappan, M., & Al-Kiswany, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00059 · Cited by 11
The Transformative Influence of LLMs on Software Development & Developer Productivity
Jalil, S. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166110 · Cited by 15
“Perceived benefits encompass the easy execution of routine tasks, swift generation of boilerplate code, code refactoring, project architecture creation, and unit test generation [23], [25], [27], [39].”
InfluentialVSCuda: LLM based CUDA extension for Visual Studio Code
Chen, B., Mustakin, N., Hoang, A., Fuad, S., & Wong, D. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3624062.3624064 · Cited by 6
Agentic Much? Adoption of Coding Agents on GitHub
Robbes, R., Matricon, T., Degueule, T., Hora, A., & Zacchiroli, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3822180 · Cited by 56
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5
Bad Smell Detection using Google Gemini
Amorim, L., da Costa, I. M., Alves, L., & Figueiredo, E. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00221 · Cited by 4
“Imai [20] evaluated GitHub Copilot’s performance in collaborative programming, noting its ability to generate code but also highlighting quality concerns.”
An Empirical Study of Knowledge Transfer in AI Pair Programming
Welter, A., Schneider, N., Dick, T., Weis, K., Tinnes, C., Wyrich, M., & Apel, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00022 · Cited by 9
“Imai [11] investigated changes in productivity and code quality in pair programming with G IT H UB C OPILOT compared to a human partner.”
Developers’ Dilemma: Opportunities and Pitfalls of Generative AI for Software Development
Zacharias, J., Popova, A., von Zahn, M., Chen, J., & Hinz, O. F. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00998-y · Cited by 8
Zhu, Z., Yu, J., & Luo, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790697 · Cited by 11
A fine-grained taxonomy of code review feedback in TypeScript projects
Davila, N., Nunes, I., & Wiese, I. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10604-y · Cited by 4
“For instance, there are reviewer recommenders and visualization techniques (Davila and Nunes2021)and,morerecently,automationapproachestorecommendcodechanges(Tufano etal.2021,2022).”
Code Generation of Smart Contracts with LLMs: A Case Study on Hyperledger Fabric
Olivieri, L., Beste, D., Negrini, L., Schönherr, L., Ciná, A. E., & Ferrara, P. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00034 · Cited by 3
“Various studies have affirmed the positive impact of these language models on programming productivity [16]–[18].”
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 6
“…our study within the existing literature, organized into three main streams: empirical evaluations of ACAT performance and perceptions in general contexts, the integration and impact of ACATs in programming education, and analyses of developers’ interaction behaviors with these tools.”
How I Learned to Stop Worrying and Love ChatGPT
Przymus, P., Fejzer, M., Narębski, J., & Stencel, K. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645073 · Cited by 4
Edutainment in the Context of Science Communication in Virtual Reality
DSouza, P., Galeote, D. F., Legaki, N.-Z., & Hamari, J. (2024) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3665463.3678827 · Cited by 3
Su, Y., Xing, Z., Wang, C., Chen, C., Xu, X., Lu, Q., & Zhu, L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715752 · Cited by 10
“…of LLMs in NLP and vision, researchers have begun to explore their applicability or impact on software engineering, such as automated code completion [20, 30], software development [16, 33], security vulnerabilities [14, 31], software exploration [42, 44, 49] and software testing [26, 27, 39].”
Rzanny, P., & Ochodek, M. (2025) · ACM Computing Surveys · doi:10.1145/3767321 · Cited by 4
How much SPACE do metrics have in GenAI assisted software development?
Sikand, S., Phokela, K. K., Sharma, V. S., Singi, K., Kaulgud, V., Tung, T., Sharma, P., & Burden, A. P. (2024) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3641399.3641419 · Cited by 3
“It is also a good representative of quality [1, 8, 9] and adaptability aspects of using a tool.”
BackgroundMethodInfluentialAssessing GitHub Copilot in Solidity Development: Capabilities, Testing, and Bug Fixing
Baralla, G., Ibba, G., & Tonelli, R. (2024) · IEEE Access · doi:10.1109/access.2024.3486365 · Cited by 4
“While Copilot has been found to increase productivity by generating substantial amounts of code, it often produces lower-quality outputs that require significant debugging, highlighting the importance of balancing productivity with code quality [19].”
BackgroundCreating benchmarkable components to measure the quality of AI-enhanced developer tools
Paradis, E., Murillo, A., Pandey, M., D’Angelo, S., Macvean, A., Ferrari-Church, B., & Hughes, M. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706674 · Cited by 2
Brandebusemeyer, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00043 · Cited by 3
“Researchers have so far relied on code quality metrics [29], [31] and software developers’ perceived experience and productivity [15], [32] to evaluate these GenAI tools during coding.”
Investigating the Use of GitHub Copilot for Green Software
Stivala, M., Fatima, I., & Lago, P. (2025) · Progress in IS · doi:10.1007/978-3-031-85284-8_13 · Cited by 2
Language-Model Assisted Learning How to Program?
Leidner, J. L., & Reiche, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-50485-3_41 · Cited by 4
“In 2022, Imai, when studying human-computer “pair” programming found that programming with Copilot helps generate more lines of code (LoC) than human pair-programming in the same period of time, but at a lower quality level [6].”
Robustness-Enhanced Assertion Generation Method Based on Code Mutation and Attack Defense
Li, M., Chen, S., Fan, G., Zhang, L., Wu, H., Xue, X., & Feng, Z. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-54528-3_16 · Cited by 2
Transforming Software Architecture Design With Intelligent Assistants-A Comparative Analysis
Ramachandran, R. (2025) · IEEE SoutheastCon · doi:10.1109/southeastcon56624.2025.10971683 · Cited by 2
Mehmood, S., Janjua, U. I., & Ahmed, A. (2023) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit60620.2023.00013 · Cited by 2
“By incorporating these innovations, the research aimed to provide nuanced insights into GitHub Copilot's capabilities in test case generation, addressing a gap in the literature [14],[15],[16],[17] and contributing to more efficient and effective software testing processes [19],[20].”
MethodISiCell: involving biologists in the design process of agent-based models in cell biologyPreprint
Cogoni, F., Bernard, D., Kazhen, R., Valitutti, S., Lobjois, V., & Cussat‐Blanc, S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.30.547165 · Cited by 2
“AI-based Pair programming such as github copilot [49, 50] would be interesting 499 tools to facilitate the handling of the platform for new modelers especially for the 500”
MethodTowards Decoding Developer Cognition in the Age of AI Assistants
Haque, E. A., Brown, C., LaToza, T. D., & Johnson, B. (2025) · arXiv · doi:10.48550/arxiv.2501.02684 · Cited by 1
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
An Assessment of ML-based Sentiment Analysis for Intelligent Web Filtering
Paspallis, N., & Panayiotou, P. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3652039 · Cited by 1
Avaliando o Papel do GPT-4o na Aprendizagem de Engenharia de Software: Um Relato de Experiência
Branco, B. M. D. C. C., Neto, A. L. D. M., Avelino, G. A., & Machado, V. P. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.10716 · Cited by 1
Salomon, M., Chin, K. D., Holmes, R., Fritz, T., & Murphy, G. C. (2025) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3758317.3759680 · Cited by 3
Lira, W. A. L., Neto, P. D. A. D. S., Avelino, G. A., & Osório, L. F. M. (2025) · Simpósio Brasileiro de Qualidade de Software (SBQS) · doi:10.5753/sbqs.2025.13853 · Cited by 1
“These models have shown significant promise in similar tasks, with one notable example being GitHub Copilot 1 , a tool that leverages LLMs to assist developers in coding tasks [26] [33] [31].”
Tives, H. A., Marioti, M. B., & Jaques, P. A. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12361 · Cited by 1
“Qualitative studies focused on natu-ral language-mediated interactions and observation of collaborative behaviors [Al Madi 2023, Imai 2022].”
Can We Trust the AI Pair Programmer? Copilot for API Misuse Detection and Correction
Mondal, S., Roy, C. K., Wang, H., Arguello, J., & Mathan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00025 · Cited by 1
Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants
Chen, V., He, J., Williams, B., Valentino, J., & Talwalkar, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786848 · Cited by 7
Impacts of Generative AI on Agile Teams' Productivity: A Multi-Case Longitudinal Study
Tomaz, R., Guenes, P., Araújo, A. A., Baldassarre, M. T., & Kalinowski, M. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793728 · Cited by 7
“However, this existing literature often consists of short-term experiments focused on individual, task-based scenarios [5, 11, 14].”
A Semi-Automated Approach for Resolving Data-Driven Architecture Mismatches
Karathanasis, C., Maikantis, T., Nikolaidis, N., Ampatzoglou, A., Chatzigeorgiou, A., & Mittas, N. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00007 · Cited by 1
“Even though the AI model evaluation varies a lot in different studies [18][19], we cannot overlook their accuracy in simple demonstrative problems and repetitive 2 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Background
Metadata
- Authors
- Imai, S.
- Affiliations
- Colby College (US)
- Corresponding author
- Imai, Saki
- ORCID
- Imai, Saki
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 319-321
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4312763760
- Cited by
- 116
- References
- 3
- Citations by year
- 2026: 23 · 2025: 44 · 2024: 39 · 2023: 10
- FWCI
- 20.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, pair programming, AI-assisted programming, code quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
