CoChat
Sign Up
PreprintCited by 206Software Engineering ResearchSoftware Engineering Techniques and PracticesUsability and User Interface Design

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

Open access
Liang, Jenny T., Yang, Chenyang, Myers, Brad A.·2023·arXiv
DOI 10.48550/arxiv.2303.17125arXiv 2303.17125
Open this paper in CoChat — free accountOpen PDF

Abstract

The software engineering community recently has witnessed widespread deployment of AI programming assistants, such as GitHub Copilot. However, in practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. This leaves a number of open questions related to the usability of these tools. To understand developers' practices while using these tools and the important usability challenges they face, we administered a survey to a large population of developers and received responses from a diverse set of 410 developers. Through a mix of qualitative and quantitative analyses, we found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax, but resonate less with using them to help brainstorm potential solutions. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements and because developers have trouble controlling the tool to generate the desired output. Our findings have implications for both creators and users of AI programming assistants, such as designing minimal cognitive effort interactions with these tools to reduce distractions for users while they are programming.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.17125

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The in-IDE design itself may have been more helpful than access to LLMs, as it potentially reduced context-switching.

    “the in-IDE design itself may have been more helpful than access to LLMs, as it potentially reduced contextswitching.”

    From Using an LLM to Help With Code Understanding (2024) · cites this paper as [39]

    Section 4.3 (page unavailable) reports that participants used AI programming assistants to avoid going online for code snippets and to “stay in the flow,” which is consistent with reduced context-switching. The paper does not establish that the in-IDE design was more helpful than access to LLMs; that comparison remains unsupported. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • This transition reflects Liang et al.'s interactive steering paradigm of GenAI programming assistance, where the programmer's role shifts toward guiding and evaluating GenAI-generated solutions rather than implementing solutions from scratch.

    “This transition reflects what Liang et al. [19] call the”

    From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [19]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (2014), 105-120. https://doi.org/10.1609/aimag.v35i4.2513 · DOI · Paper page
  2. Sebastian Baltes and Stephan Diehl. 2018. Towards a theory of software de- velopment expertise. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 187-200. https://doi.org/10.1145/3236024.3236061 · DOI
  3. Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2022. Taking flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (2022), 35-57. https://doi.org/10.1145/3582083 · DOI · Paper page
  4. Sarah E Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scraping distributed hierarchical web data. In ACM Symposium on User Interface Software and Technology (UIST). 963-975. https://doi.org/10.1145/3242587.3242661 · DOI
  5. Ruijia Cheng, Ruotong Wang, Thomas Zimmermann, and Denae Ford. 2022. "It would work for me too": How online communities shape software developers' trust in AI-powered code generation tools. arXiv preprint arXiv:2212.03491 (2022). · arXiv
  6. Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In International Conference on Intelligent User Interfaces (IUI). 329-340.
  7. Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, Zhen Ming, et al. 2022. GitHub Copilot AI pair programmer: Asset or Liability? arXiv preprint arXiv:2206.15331 (2022). · arXiv
  8. Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In ACM Technical Symposium on Computer Science Education (SIGCSE). 1136-1142. https://doi.org/10.1145/3545945.3569823 · DOI · Paper page
  9. Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdeh, Han Liu, and Saleema Amershi. 2022. Aligning offline metrics and human judgments of value of AI-pair programmers. arXiv preprint arXiv:2210.16494 (2022). · arXiv
  10. Kasra Ferdowsifard, Allen Ordookhanians, Hila Peleg, Sorin Lerner, and Nadia Polikarpova. 2020. Small-step live programming by example. In ACM Symposium on User Interface Software and Technology (UIST). 614-626. https://doi.org/10. 1145/3379337.3415869 · DOI
  11. Bent Flyvbjerg. 2006. Five misunderstandings about case-study research. Quali- tative Inquiry 12, 2 (2006), 219-245. https://doi.org/10.1177/1077800405284363 · DOI · Paper page
  12. Denae Ford, Tom Zimmermann, Christian Bird, and Nachiappan Nagappan. 2017. Characterizing software engineering work with personas based on knowledge worker actions. In ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 394-403. https://doi.org/10.1109/ESEM. 2017.54 · DOI · Paper page
  13. Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of developer productivity: There's more to it than you think. Queue 19, 1 (2021), 20-48. https://doi.org/10. 1145/3454122.3454124 · DOI · Paper page
  14. Georgios Gousios, Margaret-Anne Storey, and Alberto Bacchelli. 2016. Work prac- tices and challenges in pull-based development: The contributor's perspective. In ACM/IEEE International Conference on Software Engineering (ICSE). 285-296. https://doi.org/10.1145/2884781.2884826 · DOI
  15. Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie Van Deursen. 2015. Work practices and challenges in pull-based development: The integrator's perspective. In IEEE/ACM International Conference on Software Engineering (ICSE), Vol. 1. 358-368. https://doi.org/10.1109/ICSE.2015.55 · DOI
  16. David Hammer and Leema K Berland. 2014. Confusing claims for data: A critique of common practices for presenting qualitative research on learning. Journal of the Learning Sciences 23, 1 (2014), 37-46. https://doi.org/10.1080/10508406.2013. 802652 · DOI
  17. James D Herbsleb and Deependra Moitra. 2001. Global software development. IEEE Software 18, 2 (2001), 16-20. https://doi.org/10.1109/52.914732 · DOI
  18. Yu Huang, Denae Ford, and Thomas Zimmermann. 2021. Leaving my fingerprints: Motivations and challenges of contributing to OSS for social good. In IEEE/ACM International Conference on Software Engineering (ICSE). 1020-1032. https://doi. org/10.1109/ICSE43902.2021.00096 · DOI
  19. Saki Imai. 2022. Is GitHub copilot a substitute for human pair-programming? An empirical study. In ACM/IEEE International Conference on Software Engineering (ICSE): Companion Proceedings. 319-321. https://doi.org/10.1145/3510454.3522684 · DOI · Paper page
  20. Dhanya Jayagopal, Justin Lubin, and Sarah E Chasins. 2022. Exploring the learnability of program synthesizers by novice programmers. In ACM Symposium on User Interface Software and Technology (UIST). 1-15. https://doi.org/10.1145/ 3526113.3545659 · DOI · Paper page
  21. Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the syntax and strategies of natural language programming with generative language models. In ACM CHI Conference on Human Factors in Computing Systems. 1-19. https: //doi.org/10.1145/3491102.3501870 · DOI · Paper page
  22. Barbara A. Kitchenham and Shari Lawrence Pfleeger. 2008. Personal opinion surveys. In Guide to advanced empirical software engineering, Forrest Shull, Janice Singer, and Dag I. K. Sjøberg (Eds.). Springer, 63-92. https://doi.org/10.1007/978- 1-84800-044-5_3 · DOI
  23. Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering 20, 1 (2015), 110-141. https://doi.org/10.1007/ s10664-013-9279-3 · DOI
  24. Paul Luo Li, Amy J Ko, and Andrew Begel. 2020. What distinguishes great software engineers? Empirical Software Engineering 25 (2020), 322-352. https: //doi.org/10.1007/s10664-019-09773-y · DOI · Paper page
  25. Paul Luo Li, Amy J Ko, and Jiamin Zhu. 2015. What makes a great software engineer?. In IEEE/ACM International Conference on Software Engineering (ICSE), Vol. 1. 700-710. https://doi.org/10.1109/ICSE.2015.335 · DOI
  26. Jenny T Liang, Maryam Arab, Minhyuk Ko, Amy J Ko, and Thomas D LaToza. 2023. A Qualitative Study on the Implementation Design Decisions of Developers. In IEEE/ACM International Conference on Software Engineering (ICSE). 435-447. https://doi.org/10.1109/ICSE48619.2023.00047 · DOI
  27. Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022. Understanding skills for OSS communities on GitHub. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 170-182. https://doi.org/10.1145/3540250.3549082 · DOI
  28. Laura MacLeod, Margaret-Anne Storey, and Andreas Bergen. 2015. Code, camera, action: How software developers document and share program knowledge using YouYube. In IEEE International Conference on Program Comprehension (ICPC). 104-114. https://doi.org/10.1109/ICPC.2015.19 · DOI
  29. Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann. 2011. Design lessons from the fastest q&a site in the west. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 2857-2866. https://doi.org/10.1145/1978942.1979366 · DOI
  30. Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on human-computer interaction 3, CSCW (2019), 1-23. https://doi.org/10.1145/3359174 · DOI · Paper page
  31. Andrew M McNutt, Chenglong Wang, Robert A DeLine, and Steven M Drucker. 2023. On the design of AI-powered code assistants for notebooks. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1-16. https: //doi.org/10.1145/3544548.3580940 · DOI
  32. Brad A Myers, Amy J Ko, Thomas D LaToza, and YoungSeok Yoon. 2016. Pro- grammers are users too: Human-centered methods for improving programming tools. Computer 49, 7 (2016), 44-52. https://doi.org/10.1109/MC.2016.200 · DOI · Paper page
  33. Ben Puryear and Gina Sprint. 2022. GitHub Copilot in the classroom: Learning to code with AI assistance. Journal of Computing Sciences in Colleges 38, 1 (2022), 37-47.
  34. Nikitha Rao, Chetan Bansal, Thomas Zimmermann, Ahmed Hassan Awadallah, and Nachiappan Nagappan. 2020. Analyzing web search behavior for software engineering tasks. In IEEE International Conference on Big Data (Big Data). 768- 777. https://doi.org/10.1109/BigData50022.2020.9378083 · DOI
  35. Peter Robe, Sandeep K Kuttal, Jake AuBuchon, and Jacob Hart. 2022. Pair pro- gramming conversations with agents vs. developers: challenges and opportuni- ties for SE community. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 319-331. https://doi.org/10.1145/3540250.3549127 · DOI
  36. Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023. The programmer's assistant: Conversational interaction with a large language model for software development. In ACM Conference on Intelligent User Interfaces (IUI). 491-514. https://doi.org/10.1145/3581641.3584037 · DOI
  37. Johnny Saldaña. 2009. The Coding Manual for Qualitative Researchers. SAGE Publications.
  38. Edward Smith, Robert Loftin, Emerson Murphy-Hill, Christian Bird, and Thomas Zimmermann. 2013. Improving developer participation rates in surveys. In International workshop on cooperative and human aspects of software engineering (CHASE). 89-92. https://doi.org/10.1109/CHASE.2013.6614738 · DOI
  39. Margaret-Anne Storey, Leif Singer, Brendan Cleary, Fernando Figueira Filho, and Alexey Zagalsky. 2014. The (r)evolution of social media in software engineering. Future of Software Engineering (2014), 100-116. https://doi.org/10.1145/2593882. 2593887 · DOI
  40. Mohammad Tahaei and Kami Vaniea. 2022. Lessons Learned From Recruiting Participants With Programming Skills for Empirical Privacy and Security Stud- ies. In International Workshop on Recruiting Participants for Empirical Software Engineering (RoPES).
  41. Priyan Vaithilingam, Elena L Glassman, Peter Groenwegen, Sumit Gulwani, Austin Z Henley, Rohan Malpani, David Pugh, Arjun Radhakrishna, Gustavo Soares, Joey Wang, and Aaron Yim. 2023. Towards more effective AI-assisted programming: A systematic design exploration to improve Visual Studio In- telliCode's user experience. In IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).
  42. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1-7. https://doi.org/10.1145/3491101.3519665 · DOI · Paper page
  43. Hans Van Vliet and Antony Tang. 2016. Decision making in software architecture. Journal of Systems and Software 117 (2016), 638-644. https://doi.org/10.1016/j.jss. 2016.01.017 · DOI · Paper page
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  45. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In ACM SIGPLAN International Symposium on Machine Programming (MAPS). 1-10. https://doi. org/10.1145/3520312.3534862 · DOI · Paper page
  46. Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2022. In-IDE code generation from natural language: Promise and challenges. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 2 (2022), 1-47. https://doi.org/10.1145/ 3487569 · DOI · Paper page
  47. Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of Github Copilot's code generation. In International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE). 62-71. https://doi.org/10. 1145/3558489.3559072 · DOI
  48. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assess- ment of neural code completion. In ACM SIGPLAN International Symposium on Machine Programming (MAPS). 21-29. https://doi.org/10.1145/3520312.3534864 · DOI · Paper page

Cited by

Sorted by citations

Authors
Liang, J. T., Yang, C., & Myers, B. A.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
1-13
ISSN
0270-5257 (ISSN-L)
DOI
10.48550/arxiv.2303.17125
arXiv
2303.17125
OpenAlex
W4391558520
Cited by
206
References
2
Citations by year
2026: 70 · 2025: 92 · 2024: 43 · 2023: 1
FWCI
77.24
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (1745016, 1856641, 2140739)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
GitHub Copilot, usability, developer practices, code completion, developer motivation, non-functional requirements, cognitive effort, tool control, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review