Is GitHub copilot a substitute for human pair-programming?: an empirical study
Abstract
This empirical study investigates the effectiveness of pair programming with GitHub Copilot in comparison to human pair-programming. Through an experiment with 21 participants we focus on code productivity and code quality. For experimental design, a participant was given a project to code, under three conditions presented in a randomized order. The conditions are pair-programming with Copilot, human pair-programming as a driver, and as a navigator. The codes generated from the three trials were analyzed to determine how many lines of code on average were added in each condition and how many lines of code on average were removed in the subsequent stage. The former measures the productivity of each condition while the latter measures the quality of the produced code. The results suggest that although Copilot increases productivity as measured by lines of code added, the quality of code produced is inferior by having more lines of code deleted in the subsequent trial.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Developers may receive code they do not understand when using LLM code generation tools.
“This convenience comes at a cost, however -developers may receive code they don't understand [24,79].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [24]
Imai found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
From Towards Decoding Developer Cognition in the Age of AI Assistants (2025) · cites this paper as [11]
Cites
How We Refactor, and How We Know It
Murphy-Hill, E., Parnin, C., & Black, A. P. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.41 · Cited by 547 · see citing works
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452 · see citing works
AI-Based Software Defect Predictors: Applications and Benefits in a Case Study
Tosun, A., Bener, A., & Kale, R. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i2.18807 · Cited by 29 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417 · see citing works
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 345 · see citing works
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217 · see citing works
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203 · see citing works
Prompting Is All You Need: Automated Android Bug Replay with Large Language Models
Feng, S., & Chen, C. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608137 · Cited by 180 · see citing works
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158 · see citing works
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160 · see citing works
“The online help-seeking behavior of developers is also changing along with other behavior shifts.”
BackgroundBridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 169 · see citing works
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120 · see citing works
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundKosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024) · Mathematics · doi:10.3390/math12050629 · Cited by 98 · see citing works
On the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 81 · see citing works
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 93 · see citing works
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70 · see citing works
“Positioned as potential AI pair programmers (Moradi Dakhel et al. 2023; Imai 2022; Bird et al. 2022) in software projects, LLM-based code generators are poised to play a crucial role in the quality of the overall software project.”
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 66 · see citing works
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2023) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3605731.3605886 · Cited by 42 · see citing works
Hajipour, H., Hassler, K., Holz, T., Schönherr, L., & Fritz, M. (2023) · 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 107 · see citing works
“A prominent example is GitHub Copilot [9], an AI pair programmer based on OpenAI Codex [5], [10] that is already used by more than a million developers [11].”
MethodWeisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37 · see citing works
“…language models that have been exposed to source code in their pre-training have demonstrated a high degree of aptitude in performing a variety of tasks: converting natural language to code (e.g. [2, 16, 19, 56]), converting code to natural language documentation (e.g. [16, 31]) or explanations…”
Unveiling ChatGPT’s usage in open source projects: A mining-based study
Tufano, R., Mastropaolo, A., Pepe, F., Dabic, O., Di Penta, M., & Bavota, G. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644918 · Cited by 38 · see citing works
“These investigations include studies on the impact of the recommender on developers [71, 93, 109, 118], robustness assessment [82, 113, 115], empirical evaluation of correctness [88, 115], and scrutiny of security aspects [47, 92, 97].”
BackgroundResultPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 37 · see citing works
“…research comparing the quality of code generated by Copilot with human-written code indicates that while Copilot increases productivity in terms of the number of lines of code, the quality of its output is generally inferior, with higher failure rates on medium and complex tasks [14], [75].”
BackgroundAn Industry Case Study on Adoption of AI-based Programming Assistants
Davila, N., Wiese, I., Steinmacher, I., da Silva, L. L., Kawamoto, A. L. S., Favaro, G. J. P., & Nunes, I. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3643648 · Cited by 30 · see citing works
Detecting Code Smells using ChatGPT: Initial Insights
Silva, L. L., da Silva, J. R., Montandon, J. E., Andrade, M., & Valente, M. T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690742 · Cited by 25 · see citing works
Enhancing Exploratory Testing by Large Language Model and Knowledge Graph
Su, Y., Liao, D., Xing, Z., Huang, Q. Y., Xie, M., Lu, Q., & Xu, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639157 · Cited by 23 · see citing works
Large language model evaluation for high‐performance computing software development
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8269 · Cited by 20 · see citing works
Automatic Library Migration Using Large Language Models: First Results
Almeida, A., Xavier, L., & Valente, M. T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690746 · Cited by 18 · see citing works
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18 · see citing works
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 19 · see citing works
“Recent studies have begun to explore interaction modes of human-AI pair programming [4, 57, 61] and related performance outcomes [22, 25, 57] from case studies to experimental studies.”
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 26 · see citing works
Generative AI for Software Development: A Family of Studies on Code Generation
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_7 · Cited by 16 · see citing works
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20 · see citing works
Agossah, A., Krupa, F., Silva, M. P. D., & Callet, P. L. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3603362 · Cited by 14 · see citing works
Impacts of the Usage of Generative Artificial Intelligence on Software Development Process
Santos, P. D. O., Figueiredo, A. C., Moura, P., Diirr, B., Alvim, A. C. F., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658337 · Cited by 14 · see citing works
“GitHub Copilot 2 é uma ferramenta de desenvolvimento de software que oferece geração de linhas código, trechos de código ou até programas inteiros baseados em código e comentários exis-tentes [20].”
BackgroundRethinking AI code generation: a one-shot correction approach based on user feedback
Le, K. T., & Andrzejak, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00451-y · Cited by 15 · see citing works
“However, other empirical studies (Imai 2022) reveal that Copilot results in higher productivity but with lower quality of code.”
A disruptive research playbook for studying disruptive innovations
Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3678172 · Cited by 15 · see citing works
Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot
OBrien, D., Biswas, S., Imtiaz, S. M., Abdalkareem, R., Shihab, E., & Rajan, H. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639176 · Cited by 12 · see citing works
Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace
Butler, J., Suh, J., Haniyur, S., & Hadley, C. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00034 · Cited by 12 · see citing works
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715108 · Cited by 11 · see citing works
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 16 · see citing works
“Also, when it comes to comparison between students’ generated code and AI-CA’s generated code, the AI-CA’s output quality is often lower in quality when compared to human-written code, especially for complex tasks (Imai, 2022 ; Vaithilingam et al., 2022 ).”
ChatGPT vs. Stack Overflow: An Exploratory Comparison of Programming Assistance Tools
Liu, J., Tang, X., Li, L., Chen, P., & Liu, Y. (2023) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c60940.2023.00105 · Cited by 11 · see citing works
Nikolaidis, N., Flamos, K., Gulati, K., Feitosa, D., Ampatzoglou, A., & Chatzigeorgiou, A. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c62648.2024.00018 · Cited by 11 · see citing works
Retracted: Autonomous Crowdsensing: Operating and Organizing Crowdsensing for Sensing Automation
Wu, W., Yang, W., Li, J., Zhao, Y. B., Zhu, Z., Chen, B., Qiu, S., Peng, Y., & Wang, F. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3355508 · Cited by 10 · see citing works
“Moreover, LLMs can automatically generate code snippets or function structures based on task descriptions or requirements [32].”
BackgroundRetractedCase study: using AI-assisted code generation in mobile teams
Vasiliniuc, M.-S., & Groza, A. P. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398656 · Cited by 8 · see citing works
Human-Machine Teaming and Team Effectiveness in AI Tools for Software Engineering
Rauf, I., Sharp, H., López, T., & Michel Wermelinger, B. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00017 · Cited by 8 · see citing works
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 9 · see citing works
Beyond Snippet Assistance: A Workflow-Centric Framework for End-to-End AI-Driven Code Generation
Sonkin, V., & Tudose, C. (2025) · Computers · doi:10.3390/computers14030094 · Cited by 7 · see citing works
Exploring GenAI in Software Development: Insights from a Case Study in a Large Brazilian Company
Pereira, G. V., Jackson, V., Prikladnicki, R., van der Hoek, A., Fortes, L., Araújo, C., Coelho, A., Chelli, L., & Ramos, D. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00035 · Cited by 7 · see citing works
Generative AI for Code Generation: Software Reuse Implications
Kapitsaki, G. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66459-5_3 · Cited by 7 · see citing works
Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study
Stray, V., Brandtzæg, E. G., Wivestad, V., Barbala, A., & Moe, N. B. (2026) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2026.880 · Cited by 9 · see citing works
“GenAI tools assist developers by providing intelligent code suggestions, automating repetitive tasks, enhancing problem-solving capabilities (Dohmke, 2023; Imai, 2022), and improved coding practices (Zhang et al., 2023).”
Serafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 6 · see citing works
“This reflects findings by Imai [31], who noted that while tools like GitHub Copilot increase productivity by offering a starting point, much of the generated code is later discarded due to quality concerns.”
Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 6 · see citing works
Dockerfile Flakiness: Characterization and Repair
Shabani, T., Nashid, N., Alian, P., & Mesbah, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00238 · Cited by 6 · see citing works
“Leveraging the ability of LLMs to solve programming tasks across different domains [23]–[33], and the effectiveness of retrieval-augmented generation (RAG) techniques [34]–[36], we pro-pose F LAKI D OCK , an automated approach using LLMs to repair Dockerfile flakiness.”
An Analysis of the Costs and Benefits of Autocomplete in IDEs
Jiang, S., & Coblenz, M. J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660765 · Cited by 6 · see citing works
“Liang et al. [21] also found that Copilot reduces keystrokes but that participants found it potentially annoying and unhelpful in brainstorming potential solutions.”
Exploring the Productivity Impacts of Generative AI in Organizations
Semujanga, B., & Mikalef, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72234-9_9 · Cited by 6 · see citing works
Batista, S. S., Branco, B., Castro, O., & Avelino, G. A. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701673 · Cited by 6 · see citing works
Measuring the Runtime Performance of C++ Code Written by Humans Using Github Copilot
Erhabor, D., Udayashankar, S., Nagappan, M., & Al-Kiswany, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00059 · Cited by 7 · see citing works
The Transformative Influence of LLMs on Software Development & Developer Productivity
Jalil, S. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166110 · Cited by 6 · see citing works
“Perceived benefits encompass the easy execution of routine tasks, swift generation of boilerplate code, code refactoring, project architecture creation, and unit test generation [23], [25], [27], [39].”
InfluentialVSCuda: LLM based CUDA extension for Visual Studio Code
Chen, B., Mustakin, N., Hoang, A., Fuad, S., & Wong, D. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3624062.3624064 · Cited by 6 · see citing works
Agentic Much? Adoption of Coding Agents on GitHub
Robbes, R., Matricon, T., Degueule, T., Hora, A., & Zacchiroli, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3822180 · Cited by 5 · see citing works
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5 · see citing works
Bad Smell Detection using Google Gemini
Amorim, L., da Costa, I. M., Alves, L., & Figueiredo, E. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00221 · Cited by 4 · see citing works
“Imai [20] evaluated GitHub Copilot’s performance in collaborative programming, noting its ability to generate code but also highlighting quality concerns.”
An Empirical Study of Knowledge Transfer in AI Pair Programming
Welter, A., Schneider, N., Dick, T., Weis, K., Tinnes, C., Wyrich, M., & Apel, S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00022 · Cited by 9 · see citing works
“Imai [11] investigated changes in productivity and code quality in pair programming with G IT H UB C OPILOT compared to a human partner.”
Developers’ Dilemma: Opportunities and Pitfalls of Generative AI for Software Development
Zacharias, J., Popova, A., von Zahn, M., Chen, J., & Hinz, O. F. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00998-y · Cited by 7 · see citing works
Zhu, Z., Yu, J., & Luo, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790697 · Cited by 6 · see citing works
A fine-grained taxonomy of code review feedback in TypeScript projects
Davila, N., Nunes, I., & Wiese, I. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10604-y · Cited by 4 · see citing works
“For instance, there are reviewer recommenders and visualization techniques (Davila and Nunes2021)and,morerecently,automationapproachestorecommendcodechanges(Tufano etal.2021,2022).”
Code Generation of Smart Contracts with LLMs: A Case Study on Hyperledger Fabric
Olivieri, L., Beste, D., Negrini, L., Schönherr, L., Ciná, A. E., & Ferrara, P. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00034 · Cited by 3 · see citing works
“Various studies have affirmed the positive impact of these language models on programming productivity [16]–[18].”
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 5 · see citing works
“…our study within the existing literature, organized into three main streams: empirical evaluations of ACAT performance and perceptions in general contexts, the integration and impact of ACATs in programming education, and analyses of developers’ interaction behaviors with these tools.”
How I Learned to Stop Worrying and Love ChatGPT
Przymus, P., Fejzer, M., Narębski, J., & Stencel, K. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645073 · Cited by 3 · see citing works
Edutainment in the Context of Science Communication in Virtual Reality
DSouza, P., Galeote, D. F., Legaki, N.-Z., & Hamari, J. (2024) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3665463.3678827 · Cited by 3 · see citing works
Su, Y., Xing, Z., Wang, C., Chen, C., Xu, X., Lu, Q., & Zhu, L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715752 · Cited by 3 · see citing works
“…of LLMs in NLP and vision, researchers have begun to explore their applicability or impact on software engineering, such as automated code completion [20, 30], software development [16, 33], security vulnerabilities [14, 31], software exploration [42, 44, 49] and software testing [26, 27, 39].”
Rzanny, P., & Ochodek, M. (2025) · ACM Computing Surveys · doi:10.1145/3767321 · Cited by 4 · see citing works
How much SPACE do metrics have in GenAI assisted software development?
Sikand, S., Phokela, K. K., Sharma, V. S., Singi, K., Kaulgud, V., Tung, T., Sharma, P., & Burden, A. P. (2024) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3641399.3641419 · Cited by 3 · see citing works
“It is also a good representative of quality [1, 8, 9] and adaptability aspects of using a tool.”
BackgroundMethodInfluentialAssessing GitHub Copilot in Solidity Development: Capabilities, Testing, and Bug Fixing
Baralla, G., Ibba, G., & Tonelli, R. (2024) · IEEE Access · doi:10.1109/access.2024.3486365 · Cited by 4 · see citing works
“While Copilot has been found to increase productivity by generating substantial amounts of code, it often produces lower-quality outputs that require significant debugging, highlighting the importance of balancing productivity with code quality [19].”
BackgroundCreating benchmarkable components to measure the quality of AI-enhanced developer tools
Paradis, E., Murillo, A., Pandey, M., D’Angelo, S., Macvean, A., Ferrari-Church, B., & Hughes, M. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706674 · Cited by 2 · see citing works
Brandebusemeyer, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00043 · Cited by 2 · see citing works
“Researchers have so far relied on code quality metrics [29], [31] and software developers’ perceived experience and productivity [15], [32] to evaluate these GenAI tools during coding.”
Investigating the Use of GitHub Copilot for Green Software
Stivala, M., Fatima, I., & Lago, P. (2025) · Progress in IS · doi:10.1007/978-3-031-85284-8_13 · Cited by 2 · see citing works
Language-Model Assisted Learning How to Program?
Leidner, J. L., & Reiche, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-50485-3_41 · Cited by 2 · see citing works
“In 2022, Imai, when studying human-computer “pair” programming found that programming with Copilot helps generate more lines of code (LoC) than human pair-programming in the same period of time, but at a lower quality level [6].”
Robustness-Enhanced Assertion Generation Method Based on Code Mutation and Attack Defense
Li, M., Chen, S., Fan, G., Zhang, L., Wu, H., Xue, X., & Feng, Z. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-54528-3_16 · Cited by 2 · see citing works
Transforming Software Architecture Design With Intelligent Assistants-A Comparative Analysis
Ramachandran, R. (2025) · IEEE SoutheastCon · doi:10.1109/southeastcon56624.2025.10971683 · Cited by 2 · see citing works
Mehmood, S., Janjua, U. I., & Ahmed, A. (2023) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit60620.2023.00013 · Cited by 2 · see citing works
“By incorporating these innovations, the research aimed to provide nuanced insights into GitHub Copilot's capabilities in test case generation, addressing a gap in the literature [14],[15],[16],[17] and contributing to more efficient and effective software testing processes [19],[20].”
MethodISiCell: involving biologists in the design process of agent-based models in cell biologyPreprint
Cogoni, F., Bernard, D., Kazhen, R., Valitutti, S., Lobjois, V., & Cussat‐Blanc, S. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.30.547165 · Cited by 2 · see citing works
“AI-based Pair programming such as github copilot [49, 50] would be interesting 499 tools to facilitate the handling of the platform for new modelers especially for the 500”
MethodTowards Decoding Developer Cognition in the Age of AI Assistants
Haque, E. A., Brown, C., LaToza, T. D., & Johnson, B. (2025) · arXiv · doi:10.48550/arxiv.2501.02684 · Cited by 1 · see citing works
“Adding to this, Imai [11] found that while Copilot increased code output compared to human pair programming, the generated code actually required more subsequent modification.”
An Assessment of ML-based Sentiment Analysis for Intelligent Web Filtering
Paspallis, N., & Panayiotou, P. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3652039 · Cited by 1 · see citing works
Avaliando o Papel do GPT-4o na Aprendizagem de Engenharia de Software: Um Relato de Experiência
Branco, B. M. D. C. C., Neto, A. L. D. M., Avelino, G. A., & Machado, V. P. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.10716 · Cited by 1 · see citing works
Salomon, M., Chin, K. D., Holmes, R., Fritz, T., & Murphy, G. C. (2025) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3758317.3759680 · Cited by 2 · see citing works
Lira, W. A. L., Neto, P. D. A. D. S., Avelino, G. A., & Osório, L. F. M. (2025) · Simpósio Brasileiro de Qualidade de Software (SBQS) · doi:10.5753/sbqs.2025.13853 · Cited by 1 · see citing works
“These models have shown significant promise in similar tasks, with one notable example being GitHub Copilot 1 , a tool that leverages LLMs to assist developers in coding tasks [26] [33] [31].”
Tives, H. A., Marioti, M. B., & Jaques, P. A. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12361 · Cited by 1 · see citing works
“Qualitative studies focused on natu-ral language-mediated interactions and observation of collaborative behaviors [Al Madi 2023, Imai 2022].”
Can We Trust the AI Pair Programmer? Copilot for API Misuse Detection and Correction
Mondal, S., Roy, C. K., Wang, H., Arguello, J., & Mathan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00025 · Cited by 1 · see citing works
Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants
Chen, V., He, J., Williams, B., Valentino, J., & Talwalkar, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786848 · Cited by 1 · see citing works
Impacts of Generative AI on Agile Teams' Productivity: A Multi-Case Longitudinal Study
Tomaz, R., Guenes, P., Araújo, A. A., Baldassarre, M. T., & Kalinowski, M. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793728 · Cited by 1 · see citing works
“However, this existing literature often consists of short-term experiments focused on individual, task-based scenarios [5, 11, 14].”
A Semi-Automated Approach for Resolving Data-Driven Architecture Mismatches
Karathanasis, C., Maikantis, T., Nikolaidis, N., Ampatzoglou, A., Chatzigeorgiou, A., & Mittas, N. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00007 · Cited by 1 · see citing works
“Even though the AI model evaluation varies a lot in different studies [18][19], we cannot overlook their accuracy in simple demonstrative problems and repetitive 2 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
BackgroundMiranda, P. C., & de Andrade, M. H. S. (2024) · doi:10.5753/cbsoft_estendido.2024.4100 · Cited by 1 · see citing works
A Retrospective on How Developers Seek, Relate, and Collect Information About Code
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3534071
“One of the key challenges surfacing in these recent studies of LLMs are that no matter how good they get, they are wrong often enough that they put even greater pressure on program comprehension, to detect and overcome all of its failures [13].”
Why Personalizing Deep Learning-Based Code Completion Tools Matters
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725732
Ramachandran, R. (2025) · International Conference on Software Engineering and Computer Science (CSECS) · doi:10.1109/csecs64665.2025.11009821
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
OR for Everyone: Solving OR Problems as Non-experts with Generative AI
Maurischat, J., Bogs, S., Walther, G., & Kirchhof, O. (2025) · Lecture notes in operations research · doi:10.1007/978-3-031-92575-7_18
Determining Code Proficiency Levels from Python Textbooks
Rojpaisarnkit, R., Robles, G., Gonzalez-Barahona, J. M., Matsumoto, K., & Kula, R. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769864
“Other studies note that although LLMs provide convenience, they occasionally produce code that developers struggle to understand [19, 54].”
Automated Update of Android Deprecated API Usages With Large Language Models
Mahmud, T., Duan, B., Che, M., Yasmin, A., Ngu, A. H. H., & Yang, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3627897
“Other studies are looking at how LLMs change the way we make software [34], [49], help developers work faster [19], [78], and find security problems [57].”
Costa, A. C. M. P., Fantini, J. B. O., de Souza, É. F., Silva, G. B. E., dos Santos, L. B. R., Felizardo, K. R., & Meinerz, G. V. (2025) · Simpósio Brasileiro de Qualidade de Software (SBQS) · doi:10.5753/sbqs.2025.13875
“Specifically, studies [46] (ID 93) and [47] (ID 95) evaluated GitHub Copilot as an automated partner in pair programming, assessing its effectiveness in replicating the supportive role typically performed by a human collaborator, and highlighting its potential to transform collaborative development…”
Revolutionizing Software Development with Machine Learning: A Comprehensive Review
Kaur, C., Kapur, S., Vandana, V., & Agrawal, T. (2025) · IEEE Space, Aerospace and Defence Conference (SPACE) · doi:10.1109/space65882.2025.11171421
- Authors
- Imai, S.
- Affiliations
- Colby College (US)
- Corresponding author
- Imai, Saki
- ORCID
- Imai, Saki
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 319-321
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4312763760
- Cited by
- 116
- References
- 3
- Citations by year
- 2026: 23 · 2025: 44 · 2024: 39 · 2023: 10
- FWCI
- 20.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, pair programming, AI-assisted programming, code quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
