The DOI matches its DataCite record; 12 of the 12 references it cites were found in a registry or an index.
Productivity Assessment of Neural Code Completion
Open accessAbstract
Neural code synthesis has reached a point where snippet generation is accurate enough to be considered for integration into human software development workflows. Commercial products aim to increase programmers' productivity, without being able to measure it directly. In this case study, we asked users of GitHub Copilot about its impact on their productivity, and sought to find a reflection of their perception in directly measurable user data. We find that the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Consistent with earlier research [32, 49], we acknowledge that these proxy metrics might not fully capture the usability of the interaction without potentially compromising it in some other way.”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Copilot's authors themselves reported that about two-thirds of shown completions are ignored by their end-user [49].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, optimising only acceptance rate results in worse completions [32, 49].”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as [49]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Ziegler et al. [49] proposes the acceptance rate of suggestions as a proxy for developers' perceived productivity”
From A Transformer-Based Approach for Smart Invocation of Automatic Code Completion (2024) · cites this paper as Ziegler et al. [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PLS-regression: a basic tool of chemometrics
Wold, S., Sjöström, M., & Eriksson, L. (2001) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/s0169-7439(01)00155-1 · Cited by 10,084
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
MethodWhat makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205
The SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Fast and Memory-Efficient Neural Code Completion
Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J. V., & Allamanis, M. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00045 · Cited by 95
A Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77
When Code Completion Fails: A Case Study on Real-World Completions
Hellendoorn, V. J., Proksch, S., Gall, H. C., & Bacchelli, A. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00101 · Cited by 76
Research works printed but not linked: 7
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732 · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, An- drew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCan- dlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 · arXiv
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Com- petence With APPS. CoRR abs/2105.09938 (2021). arXiv:2105.09938 https://arxiv.org/abs/2105.09938 · arXiv
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019. SPoC: Search-based Pseudocode to Code. In Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 11883ś11894. https://proceedings.neurips.cc/paper/2019/hash/ 7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html
- Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. 2022. Ex- pectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI '22 Late-Breaking Work: Proceedings of the 2022 Conference on Human Factors in Comput- ing Systems.
- Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. CoRR abs/2101.11149 (2021). arXiv:2101.11149 https://arxiv.org/abs/ 2101.11149 · arXiv
- Wen Zhou, Seohyun Kim, Vijayaraghavan Murali, and Gareth Ari Aye. 2021. Improving Code Autocompletion with Transfer Learning. CoRR abs/2105.05991 (2021). arXiv:2105.05991 https://arxiv.org/abs/2105. 05991 · arXiv
Cited by
Sorted by citations
Items 101–200 of 201
On the Generalizability of Deep Learning-based Code Completion Across Programming Language Versions
Ciniselli, M., Martin-Lopez, A., & Bavota, G. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644411 · Cited by 6
Hicks, C. M., Lee, C. S., & Foster-Marks, K. L. (2024) · doi:10.31234/osf.io/2gej5 · Cited by 6
Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets
Jahanshahi, M., & Mockus, A. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00018 · Cited by 6
Learning Program Behavioral Models from Synthesized Input–Output Pairs
Mammadov, T., Klakow, D., Koller, A., & Zeller, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748720 · Cited by 5
PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage
Chu, B.-J., Feng, Y., Liu, K., Shi, H., Nan, Z., Guo, Z., & Xu, B.-W. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00223 · Cited by 4
Agentic Much? Adoption of Coding Agents on GitHub
Robbes, R., Matricon, T., Degueule, T., Hora, A., & Zacchiroli, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3822180 · Cited by 56
AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development
Agarwal, S., He, H., & Vasilescu, B. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793589 · Cited by 20
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5
Re(gEx|DoS)Eval: Evaluating Generated Regular Expressions and their Proneness to DoS Attacks
Siddiq, M. L., Zhang, J., Roney, L., & Santos, J. C. S. (2024) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3639476.3639757 · Cited by 10
Ansible Lightspeed: A Code Generation Service for IT Automation
Sahoo, P., Pujar, S., Nalawade, G., Genhardt, R., Mandel, L., & Buratti, L. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695277 · Cited by 9
Challenges to Using Large Language Models in Code Generation and Repair
Pasquale, L., Sabetta, A., d’Amorim, M., Hegedűs, P., Mirakhorli, M. T., Okhravi, H., Payer, M., Rashid, A., Santos, J. C. S., Spring, J., et al. (2025) · IEEE Security & Privacy · doi:10.1109/msec.2025.3530488 · Cited by 5
Toward Effective AI Support for Developers
Khemka, M., & Houck, B. (2024) · Queue · doi:10.1145/3675416 · Cited by 7
Guiding ChatGPT for Better Code Generation: An Empirical Study
Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. L. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00018 · Cited by 10
Martínez, M. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728704 · Cited by 5
Blasquez, I. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729050 · Cited by 4
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Chen, V., Talwalkar, A., Brennan, R., & Neubig, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790850 · Cited by 36
EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translation
Wang, C., Yu, T., Shen, B., Wang, J., Chen, D., Zhang, W., Shi, Y., Xie, C., & Gu, X. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786856 · Cited by 4
Martin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 6
Code Generation of Smart Contracts with LLMs: A Case Study on Hyperledger Fabric
Olivieri, L., Beste, D., Negrini, L., Schönherr, L., Ciná, A. E., & Ferrara, P. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00034 · Cited by 3
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 6
Responsible AI in Agile Software Engineering - An Industry Perspective
Ulfsnes, R., Moe, N. B., Emmerhoff, J., Floryan, M., Griva, A., Gundelsby, J. H., Barbala, A., & Conboy, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_4 · Cited by 7
Cracking CodeWhisperer: Analyzing Developers’ Interactions and Patterns During Programming Tasks
Javahar, J., Budhrani, T., Basha, M., de Souza, R. B. C., Beschastnikh, I., & Rodríguez-Pérez, G. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00051 · Cited by 2
Bringing Context-Aware Completion Suggestions to Arbitrary Text Entry Interfaces
Aveni, T. J., Fox, A., & Hartmann, B. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586182.3615825 · Cited by 3
How much SPACE do metrics have in GenAI assisted software development?
Sikand, S., Phokela, K. K., Sharma, V. S., Singi, K., Kaulgud, V., Tung, T., Sharma, P., & Burden, A. P. (2024) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3641399.3641419 · Cited by 3
Assessing GitHub Copilot in Solidity Development: Capabilities, Testing, and Bug Fixing
Baralla, G., Ibba, G., & Tonelli, R. (2024) · IEEE Access · doi:10.1109/access.2024.3486365 · Cited by 4
Artificial Intelligence in Software Documentation: Embracing the Documentation as Code Paradigm
Nafz, F., Krajinovic, M., & Ley, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92196-4_14 · Cited by 3
Advancing Secure and Standard Source Code Generation Techniques
Siddiq, M. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00023 · Cited by 3
Organizational Ohm's Law in Practice: Measuring Engineering Productivity
Liu, F., & McKinnon, T. (2024) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3421892 · Cited by 3
Katzy, J., Huang, Y., Panchu, G.-R., Ziemlewski, M., Loizides, P., Vermeulen, S., Arie Van Deursen, A., & Izadi, M. (2025) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3727582.3728683 · Cited by 6
Brandebusemeyer, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00043 · Cited by 3
Exploring the Use and Misuse of Large Language Models
Valdez, H. P. D., Abri, F., Webb, J. B., & Austin, T. H. (2025) · Information · doi:10.3390/info16090758 · Cited by 5
Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
Tang, N., Smith, E., Huang, Y., Collin, M., & Li, T. J.-J. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00017 · Cited by 4
Exploring the Challenges and Opportunities of AI-assisted Codebase Generation
Eibl, P., Sabouri, S., & Chattopadhyay, S. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00034 · Cited by 2
The Emerging Artifacts of Centralized Open-Code
Choksi, M. Z., Mandel, I., Widder, D. G., & Shvartzshnaider, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659019 · Cited by 3
“No Free Lunch” when using Large Language Models to Verify Self-Generated Programs
Zilberman, S., & Cheng, H. C. B. (2024) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw60967.2024.00018 · Cited by 2
AI Copilot for the Modern Developer : Leveraging GenAI in Software Development
Nagpal, A. (2024) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit241051056 · Cited by 2
Refuting LLM-generated Code with Reactive Task Comprehension
Hebbar, S. V., Sasmita Harini, S., & Kumar, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729100 · Cited by 4
Wivestad, V. T., Barbala, A., & Stray, V. G. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_13 · Cited by 1
Improving LLM-based code completion using LR parsing
Atique, M. M. A. B., Moon, H., Sasano, I., & Choi, K. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101352 · Cited by 1
AI-Assisted SQL Authoring at Industry Scale
Maddila, C., Ghorbani, N., Jabre, K., Murali, V., Kim, E., Thakkar, P., Laptev, N., Harman, O., Hsu, D., Abreu, R., et al. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00019 · Cited by 3
A Systematic Approach to Measuring Developer Productivity: The Prismetrix Method
Kawalerowicz, M., & Pietranik, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09318-9_23 · Cited by 1
APILOT: Improving the Security and Usability of LLM Code Suggestions via Outdated API Mitigation
Bai, W., Ke, X., Huang, P., Wu, Q., Wen, J., Wu, J., & Lu, K. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00094 · Cited by 15
CodeVoyager: Integrating Interactive Visual Aids with LLMs for Code Comprehension
Kim, Y., Jang, K., Song, H., Shin, M., & Suh, B. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789057 · Cited by 1
Can we trust the source? A systematic review of watermarking and attribution for AI-generated code
Fahad, I. A., & Fuad, M. M. N. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108157 · Cited by 1
Human-AI Interaction, Moral Responsibility and Epistemic Credit
Largo, J. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01104-6 · Cited by 1
Bishop, G., Chakraborty, S., Zhou, H., Dehlinger, J., Deng, L., Lin, J., & Kist, B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00617-w · Cited by 1
Patel, T., Bayyavarapu, S. R. K. V., Soni, V. R., Purushothaman, R., Thokala, G. B., & Ranganathan, V. (2026) · International Conference on Advances in Artificial Intelligence and Machine Learning (AAIML) · doi:10.1109/aaiml67890.2026.11498111 · Cited by 1
LLM-Powered Copilot for Aspen Plus: Enabling Human–AI Collaborative Process Design
Kang, J., Yao, Y., & Wong, D. S. (2026) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.6c00963 · Cited by 1
Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance
Pinna, G., Gong, J., Williams, D., & Sarro, F. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793576 · Cited by 1
Alakmeh, T., D’Angelo, S., & Fritz, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787608 · Cited by 1
An Eye for AI: Eye-Tracking the Micro-Interruptions of GenAI Code Suggestions
Alakmeh, T., D’Angelo, S., & Fritz, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773132 · Cited by 1
Using AI-Based Code Completion for Domain-Specific Languages
Piereder, C., Fleck, G., Geist, V., Moser, M., & Pichler, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49266-2_16 · Cited by 1
Colaboração com Assistente de Codificação Baseado em IA: Benefícios e Desafios
Mendes, W., Souza, S., & de Souza, C. R. B. (2024) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2024.237964
AthenaLLM: Supporting Experiments with Large Language Models in Software Development
de Oliveira, B., & Castor, F. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644437 · Cited by 2
The Art of Programming: Challenges in Generating Code for Creative Applications
Cook, M. L. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664774
Lilholt, A., & Heverin, T. (2024) · Proceedings of the International Conference on AI Research. · doi:10.34190/icair.4.1.3034
Wivestad, V. T., & Ulfsnes, R. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-72781-8_5
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
Rachamala, N. R. (2024) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v12i1.11726 · Cited by 12
Determining Code Proficiency Levels from Python Textbooks
Rojpaisarnkit, R., Robles, G., Gonzalez-Barahona, J. M., Matsumoto, K., & Kula, R. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3769864 · Cited by 1
How Can Copilot Assist in Creating Accessible Websites? An Empirical Study
Teran, L. A., da Rocha, T. A., da Silva, I. M. M., de Salles, R. B., Melo, G. L. N., Mota, M. P., & de Souza, C. R. B. (2025) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais (IHC) · doi:10.5753/ihc.2025.10811
Automated Update of Android Deprecated API Usages With Large Language Models
Mahmud, T., Duan, B., Che, M., Yasmin, A., Ngu, A. H. H., & Yang, G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3627897
Nogueira, R. P., Vieira, M. P. A., & Campos, J. R. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00036
Olfati‐Saber, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5860839
Github Copilot's Impact on Developer Productivity : A Review of Early Evidence
Adepoju, S. A. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2221192 · Cited by 3
Software Vulnerability Management in the Era of Artificial Intelligence: An Industry Perspective
Kholoosi, M. M., Le, T. H. M., & Babar, M. A. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787781 · Cited by 1
The Automation Fallacy: Why AI Doesn’t Function as a Simple “Job Killer”
Yadav, P. S. (2026) · Future of business and finance · doi:10.1007/978-3-032-11748-9_1
Yapay Zeka Destekli Kod Üretimi İle İnsan Yazımı Kodların Karşılaştırılması: Python Örneği
EYGÜ, S., & Keskinkılıç, M. (2026) · Bilgisayar bilimleri ve mühendisliği dergisi. · doi:10.54525/bbmd.1781375
Matejić, T., Mihailović, Đ., & Dukić, A. (2025) · Limes plus. · doi:10.69899/limes-plus25222046m
Developers and generative AI: A study of self-admitted usage in open source projects
Tufano, R., Pepe, F., Zampetti, F., Mastropaolo, A., Dabić, O., Penta, M. D., & Bavota, G. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10848-w · Cited by 2
Examining Software Engineering Practices in the Pre-AI and Post-AI ERA
Ebiwonjumi, O., Ahakonye, L. A. C., & Kim, D. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454249
Garg, A., Ishaani, M., & DeLaPena, R. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798372
Engineering Students' Usage and Perceptions of GitHub Copilot in Open-Source Projects
Rani, N., Munnangi, J. R., Spangler, A. M., & Honeycutt, D. R. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59352
Adoption and Usability of AI Tools in Bangladesh: Opportunities and Challenges
Hossain, D. T., & Sakib, N. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491700
Inoue, S., Morisaki, S., Wada, T., & Hattori, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3702933
One More Prompt: LLM Coding is a Game You Can’t Quite Put Down - A Provocation
Beale, R. (2026) · Designing Interactive Systems Conference · doi:10.1145/3802974.3809412
GeoMind: An Autonomous Multi-Agent Framework for Verified Geospatial AnalysisPreprint
Habib, J., Srabon, M. I., Nahar, M. L., Lutfy, M., Khatun, M. A., Shammo, M. B. A. Z., & Hasan, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10034367/v1
Understanding and Predicting Accepted Code Suggestions in AI-Assisted Programming
Jiang, J., Li, L., Hou, J., Tan, X. L., & Zhang, L. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808125
Suryanarayanan, S. (2026) · International Conference on Inventive Computing and Informatics (ICICI) · doi:10.1109/icici68773.2026.11581171
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 6
Beyond Correctness: Measuring Hallucination Impact in AI-Assisted Development Workflows
Gupta, S., Vanteru, K., Reddy, P. K., & Sunkireddy, S. (2026) · International Conference on Innovations in Emerging Technologies for Sustainable Development (ICIETSD) · doi:10.1109/icietsd68684.2026.11584879
Schadow, F., Gerling, K., Kretzer, F., & Maedche, A. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3805029.3818286
Brandebusemeyer, C., Schimmer, T., & Bert Arnrich, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786870 · Cited by 4
Do Language Models Prefer Vulnerable Code? A Probabilistic Study of Insecure Code Preference
Melo, R., Reis, S., Catarino, A., & Abreu, R. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00075
PROBE: Benchmarking code generation in large language models
Nogueira, R. P., Vieira, M., & Campos, J. R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10904-5
The Watermark Paradox: How Provenance Verification Paves the Road to Camouflaged Backdoors
Zhang, H., Ran, H.-J., Kim, K., & Tang, X. (2026) · SIGSOFT FSE Companion · doi:10.1145/3803437.3805553
MMAIGCD — Multimodal multilingual AI-generated code detection
Boulibya, H., Ouedeghyry, L. E., Azami, Y., Lamrini, M., Tairi, H., & Riffi, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113038
Dey, R. K., Nath, S., Mukherjee, D., Ray, B., Mahapatra, K., & Swastik, B. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-3751-6.ch003
When AI Writes the Code, Where Is the Skill? Measuring Effective AI Use in Software DevelopmentPreprint
Mukherjee, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7118201
Test Coverage of Code Changes in AI-Generated Pull Requests
Alves, T., & Teixeira, L. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793618
AI Code Autocompletion and Optimization Method with Integrated Real-Time Error Diagnosis
Yao, S., Wenke, L., Tian, Y., Dong, R., & Wang, Y. (2026) · International Conference on Intelligent Communications and Computing (ICICC) · doi:10.1109/icicc71012.2026.11638044
Das, P., Reddy, Y. R., Potdar, S., & Warudkar, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_28
Software Engineering Reliability in the Era of Generic and Agentic AI
Nwodo, K., & Stavrou, A. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18585-3_9
Anderson, A. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00048
Abdelsalam, Y., Peitek, N., Maurer, A.-M., Toneva, M., & Apel, S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773219 · Cited by 3
Wang, C., Zhang, Z., Xu, X., Zheng, M., Wei, G., & Zhang, X. (2026) · Foundations and Trends® in Programming Languages · doi:10.1108/ftpgl-11-2025-0068
Dhingra, A., Menon, G., & Antony, A. (2026) · IEEE Access · doi:10.1109/access.2026.3741017
Wang, W., & Yue, S. (2026) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3785651.3831552 · Cited by 1
iBRAHIMOĞLU, N., & Demi̇rezen, M. U. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00695-w
Metadata
- Authors
- Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 21-29
- arXiv
- 2205.06537
- OpenAlex
- W4281669078
- Cited by
- 200
- References
- 12
- Citations by year
- 2026: 47 · 2025: 56 · 2024: 70 · 2023: 25 · 2022: 2
- FWCI
- 34.20
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- GitHub Copilot, developer productivity, code completion, developer perceptions, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
