The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Open accessAbstract
The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some recent studies confirm these symptoms: only 40% of CCT suggestions are accepted the first time they are proposed to developers, mainly because developers cannot control context or granularity (Liang et al., 2024).”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Liang et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GitHub Copilot was the most widely used tool (74.6%), whose users reported that a median of 30.5% of their code was now generated by the AI assistant.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Liang et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similarly, Liang et al. ( 2024) and Butler et al. (2025) found that developers generally feel productivity gains using the same assistant.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Liang et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Liang et al. [18] found that frustration occurs when the suggestions do not satisfy functional or non-functional requirements.”
From How do Copilot Suggestions Impact Developers' Frustration and Productivity? (2025) · cites this paper as Liang et al. [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Acceleration mode is usually preferred, as developers often use the tools to complete repetitive code that cannot be copy-pasted (Liang et al., 2024), and for recalling syntax they don't remember (Xu et al., 2022).”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Liang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“The literature also highlights the need for explanations (such as reporting the source or linking to documentation) for additional context on the generated code (Liang et al., 2024; Xu et al., 2022).”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Liang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“They also found that developers often do not use CCTs because these tools do not generate code that addresses certain functional or non-functional requirements, and because developers have trouble controlling the tool to generate the desired output.”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Liang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“The survey revealed that developers are most motivated to use AI programming assistants because they help developers reduce keystrokes, finish programming tasks quickly, and recall syntax.”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Liang et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“On the other hand, the most common reasons for not using such tools were 1) that the code output often is of subpar quality, and 2) the difficulty of controlling the AI assistants to produce the desired output.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Liang et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“The primary motivations for adopting AI assistants were 1) reducing keystrokes, 2) completing tasks faster, and 3) skipping web searches for code examples.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Liang et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Five misunderstandings about case-study research
Flyvbjerg, B. (2006) · Qualitative Inquiry · doi:10.1177/1077800405284363 · Cited by 14,266
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,936
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Herbsleb, J. D., & Moitra, D. (2001) · IEEE Software · doi:10.1109/52.914732 · Cited by 1,059
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Design lessons from the fastest q&a site in the west
Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G. M., & Hartmann, B. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979366 · Cited by 574
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Kitchenham, B. A., & Pfleeger, S. L. (2007) · Springer eBooks · doi:10.1007/978-1-84800-044-5_3 · Cited by 515
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Work practices and challenges in pull-based development
Gousios, G., Storey, M., & Bacchelli, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884826 · Cited by 356
Ross, S., Martinez, F., Houde, S., Muller, M. J., & Weisz, J. D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584037 · Cited by 345
Work Practices and Challenges in Pull-Based Development: The Integrator's Perspective
Gousios, G., Zaidman, A., Storey, M., & Arie Van Deursen, A. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.55 · Cited by 336
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
(2015) · doi:10.1007/s10664-013-9279-3 · Cited by 273
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257
Hammer, D., & Berland, L. K. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.802652 · Cited by 246
The (R) Evolution of social media in software engineering
Storey, M., Singer, L., Cleary, B., Filho, F. F., & Zagalsky, A. (2014) · Future of Software Engineering (FOSE) · doi:10.1145/2593882.2593887 · Cited by 207
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Assessing the quality of GitHub copilot’s code generation
Yetiştiren, B., Ozsoy, I., & Tüzün, E. (2022) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 187
The SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
What Makes a Great Software Engineer?
Li, P., Ko, A. J., & Zhu, J.-M. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.335 · Cited by 161
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 6×
“Other studies report that software developers face barriers in comprehending and debugging generated code to fit their use cases, because they need to have prior knowledge of the underlying programming principles, frameworks, or APIs [12, 60].”
BackgroundResultInfluentialProgrammers Are Users Too: Human-Centered Methods for Improving Programming Tools
Myers, B. A., Ko, A. J., LaToza, T. D., & Yoon, Y. (2016) · Computer · doi:10.1109/mc.2016.200 · Cited by 144
Self-citationImproving developer participation rates in surveys
Smith, E., Loftin, R., Murphy-Hill, E. R., Bird, C., & Zimmermann, T. (2013) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2013.6614738 · Cited by 132
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 3×
“The most popular strategy participants reported was providing very clear and explicit explanations of what the code should do in comments, which is a major activity while using AI programming assistants [44].”
BackgroundMethodTowards a theory of software development expertise
Baltes, S., & Diehl, S. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236061 · Cited by 109
On the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 100
Chasins, S., Mueller, M., & Bodík, R. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242661 · Cited by 92
Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79
Decision making in software architecture
van Vliet, H., & Tang, A. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.01.017 · Cited by 79
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
MacLeod, L., Storey, M., & Bergen, A. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.19 · Cited by 76
Small-Step Live Programming by Example
Ferdowsi, K., Ordookhanians, A., Peleg, H., Lerner, S., & Polikarpova, N. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415869 · Cited by 49
What distinguishes great software engineers
Li, P. L., Ko, A. J., & Begel, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09773-y · Cited by 48
HCI Guidelines for Gender Equity and Inclusivity
Scheuerman, M. K., Spiel, K., Haimson, O. L., Hamidi, F., & Branham, S. M. (2020) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2nw1f-p0jx · Cited by 48
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40
Leaving My Fingerprints: Motivations and Challenges of Contributing to OSS for Social Good
Huang, Y., Ford, D., & Zimmermann, T. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00096 · Cited by 40
Understanding skills for OSS communities on GitHub
Liang, J. T., Zimmermann, T., & Ford, D. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549082 · Cited by 38
Self-citationCharacterizing Software Engineering Work with Personas Based on Knowledge Worker Actions
Ford, D., Zimmermann, T., Bird, C., & Nagappan, N. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.54 · Cited by 37
Vaithilingam, P., Glassman, E. L., Groenwegen, P., Gulwani, S., Henley, A. Z., Malpani, R., Pugh, D., Radhakrishna, A., Soares, G., Wang, J., et al. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00022 · Cited by 37
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 11×
“This could assist the model in adapting to whether the developer was in acceleration mode–associated with short completions– or exploration mode–associated with long completions [12].”
BackgroundMethodResultInfluentialRobe, P., Kuttal, S. K., AuBuchon, J., & Hart, J. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549127 · Cited by 23
Analyzing Web Search Behavior for Software Engineering Tasks
Rao, N., Bansal, C., Zimmermann, T., Awadallah, A. H., & Nagappan, N. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378083 · Cited by 19
A Qualitative Study on the Implementation Design Decisions of Developers
Liang, J. T., Arab, M., Ko, M., Ko, A. J., & LaToza, T. D. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00047 · Cited by 12
Self-citationLiang, J. T., Myers, B., & Yang, C. (2023) · Figshare · doi:10.6084/m9.figshare.22355017 · Cited by 1
Self-citation
Research works printed but not linked: 7
- 2023. Code faster with AI code completions | Tabnine. Retrieved March 11, 2023 from https://www.tabnine.com/.
- 2023. Prolific • Quickly find research participants you can trust. Retrieved September 2, 2023 from https://prolific.co/.
- Ruijia Cheng, Ruotong Wang, Thomas Zimmermann, and Denae Ford. 2022. "It would work for me too": How online communities shape software developers' trust in AI-powered code generation tools. arXiv preprint arXiv:2212.03491 (2022). · arXiv
- Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2022. Aligning offline metrics and human judgments of value of AI-pair programmers. arXiv preprint arXiv:2210.16494 (2022). · arXiv
- 10.5555/3575618.3575622 · DOI
- Mohammad Tahaei and Kami Vaniea. 2022. Lessons Learned From Recruiting Participants With Programming Skills for Empirical Privacy and Security Studies. In International Workshop on Recruiting Participants for Empirical Software Engineering (RoPES).
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
Cited by
Sorted by citations
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Self-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead
Zhou, X., Cao, S., Sun, X., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708522 · Cited by 215
Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward
Sergeyuk, A., Golubev, Y., Bryksin, T., & Ahmed, I. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107610 · Cited by 101
Metadata
- Authors
- Liang, J. T., Yang, C., & Myers, B. A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-02-06
- Pages
- 1-13
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4391558520
- Cited by
- 203
- References
- 45
- Citations by year
- 2026: 68 · 2025: 92 · 2024: 42
- FWCI
- 76.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1745016, 1856641, 2140739)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- GitHub Copilot, usability, developer practices, code completion, developer motivation, non-functional requirements, cognitive effort, tool control, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
