Significant Productivity Gains through Programming with Large Language Models
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Large language models like GPT and Codex drastically alter many daily tasks, including programming, where they can rapidly generate code from natural language or informal specifications. Thus, they will change what it means to be a programmer and how programmers act during software development. This work explores how AI assistance for code generation impacts productivity. In our user study (N=24), we asked programmers to complete Python programming tasks supported by a) an auto-complete interface using GitHub Copilot, b) a conversational system using GPT-3, and c) traditionally with just the web browser. Aside from significantly increasing productivity metrics, participants displayed distinctive usage patterns and strategies, highlighting that the form of presentation and interaction affects how users engage with these systems. Our findings emphasize the benefits of AI-assisted coding and highlight the different design challenges for these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174 · cited 1×
“While breaking down complex interaction patterns into individual interactions is a well-established pattern [8], prior work (e.g., Forsgren et al. [17] typically considers a higher number of interactions as indicative of high activity.”
Cited by
A Survey on LLM-Based Code Generation for Low-Resource and Domain-Specific Programming Languages
Joel, S., Wu, J. J., & Fard, F. H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770084 · Cited by 207
'I'm Categorizing LLM as a Productivity Tool': Examining Ethics of LLM Use in HCI Research Practices
Kapania, S., Wang, R., Li, T. J.-J., Li, T., & Shen, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711000 · Cited by 80
OntoGenix: Leveraging Large Language Models for enhanced ontology engineering from datasets
Val-Calvo, M., Aranguren, M. E., Mulero-Hernández, J., Almagro-Hernández, G., Deshmukh, P., Bernabé-Díaz, J. A., Espinoza-Arias, P., Sánchez-Fernández, J. L., Mueller, J., & Fernández‐Breis, J. T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104042 · Cited by 36
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Weinz, M., Zannone, N., Allodi, L., & Apruzzese, G. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736195 · Cited by 44
Kemell, K., Saarikallio, M., Nguyen‐Duc, A., & Abrahamsson, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107805 · Cited by 41
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“…and at a very specific point in the history of AI-enhanced developer tools and since questions about return on investment in AI and comparison across developer suites will continue to arise, it is critical for the field to keep researching and publishing these estimates and pull the field forward.”
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
How Scientists Use Large Language Models to Program
O’Brien, G. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713668 · Cited by 42
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Becker, J., Rush, N., Barnes, E., & Rein, D. (2025) · arXiv · doi:10.48550/arxiv.2507.09089 · Cited by 16
“Peng et al. [10] and Paradis et al. [11] find 56% and 21% speedups on coding tasks when using AI assistance, and Weber et al. [18] finds a 65% increase in the rate of task requirements satisfied with AI tools.”
Kudriavtseva, A., Hotak, N. A., & Gadyatskaya, O. (2025) · Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing · doi:10.1145/3672608.3707778 · Cited by 9
Responsible Vibe Coding: Architecture, Opportunities, and Research Agenda
Elgendy, I. A., Dwivedi, Y. K., Al-Sharafi, M. A., Hosny, M., Helal, M. Y. I., Crick, T., Hughes, L., Alwahaishi, S. S., Mahmud, M., Dutot, V., et al. (2026) · Journal of Computer Information Systems · doi:10.1080/08874417.2026.2621186 · Cited by 16
When Efficiency Meets Fulfillment: Understanding Long-Term LLM Integration in Knowledge Work
Kobiella, C., Mitrevska, T., Schmidt, A., & Draxler, F. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729180 · Cited by 19
Self-citationDesolda, G., Esposito, A., Greco, F., Tucci, C., Buono, P., & Piccinno, A. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103648 · Cited by 7
The role of firm AI capabilities in generative AI-pair coding
Bughin, J. (2024) · Journal of Decision Systems · doi:10.1080/12460125.2024.2428187 · Cited by 6
Chau, M., & Xu, J. (2025) · ACM Transactions on Management Information Systems · doi:10.1145/3713032 · Cited by 11
Durán, F., Martínez, M., Lago, P., & Martínez‐Fernández, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112574 · Cited by 18
“The resulting dataset closely emulates interactive IDE-style auto-complete code task and ensures uniformity in prompt length when comparing inference efficiency across different serving configurations [31].”
InfluentialBiazotto, J. P., Feitosa, D., Avgeriou, P., & Nakagawa, E. Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10691-5 · Cited by 6
“General development tools are usually adopted based on immediate productivity gains (Weber et al. 2024).”
Cornide-Reyes, H., Monsalves, D., Durán, E. P., Silva-Aravena, F., & Morales, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93536-7_16 · Cited by 11
Ansible Lightspeed: A Code Generation Service for IT Automation
Sahoo, P., Pujar, S., Nalawade, G., Genhardt, R., Mandel, L., & Buratti, L. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695277 · Cited by 18
Reaching new frontiers in nanoelectronics through artificial intelligence
Sivasubramani, S., & Prodromakis, T. (2025) · Frontiers in Nanotechnology · doi:10.3389/fnano.2025.1627210 · Cited by 8
Automated Code Review Using Large Language Models at Ericsson: An Experience Report
(2025) · doi:10.1109/icsme64153.2025.00061 · Cited by 4
“Moreover, the effectiveness of LLM-generated reviews is significantly dependent on model selection, prompt engineering, and integration with existing development workflows [6].”
Leveraging AI for Automated Code Generation from Systems Engineering Specifications
Padubrin, M., Reuss, H.-C., Brosi, F., Menz, L., & Guerocak, E. (2025) · SAE International Journal of Advances and Current Practices in Mobility · doi:10.4271/2025-01-0295 · Cited by 3
Working with Large Language Models: impacts on performance, efficiency and perceived responsibility
Kares, F., Hannig, L., & Langer, M. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2666279 · Cited by 3
Si, J. J., Li, D., Wang, Q., Liu, A. W., Lin, X., Zhu, Y., Zhou, X., Wang, A. Y., & Wang, K. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790366 · Cited by 7
Janabi, I. A., & Bland, T. (2026) · AI · doi:10.3390/ai7060223 · Cited by 3
Bouamor, H., Gongora-Svartzman, G., Heimann, L., et al. (2025) · Technical Symposium on Computer Science Education · doi:10.1145/3641555.3705175 · Cited by 2
“While these tools can boost programming productivity [6], they could also offer potential solutions to address disparate in students ’ technical preparedness.”
Kobiella, C., Breidenstein, D., & Schmidt, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790757 · Cited by 8
Self-citationJang, Y., Sakashita, M., Niinuma, K., & Gupta, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791260 · Cited by 3
Martin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 7
Li, Y., Li, R., & Wang, Q. (2026) · Sustainable Development · doi:10.1002/sd.71238 · Cited by 1
Enhancing Performance or Hijacking Attention? A Workshop on the Use of AI in Workplace Environments
Shahu, A., Wintersberger, P., Talypova, D., Vitali, A., Kobiella, C., & Dingler, T. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3707640.3729211 · Cited by 1
Using Generative AI in Software Design Education
Jackson, V., Liu, S.-S., & van der Hoek, A. (2026) · International Workshop on Designing Software (Designing) · doi:10.1145/3786152.3788589 · Cited by 2
Tian, W., & Zhou, Y. (2026) · Expert Systems · doi:10.1111/exsy.70341 · Cited by 3
Brochu, B. M., Cobler-Lichter, M. D., Arcieri, T. R., Shah, N. M., Delamater, J. M., Reyes, A. M., Sussman, M. S., Lineen, E. B., Sands, L. R., Hui, V. W., et al. (2025) · Global Surgical Education - Journal of the Association for Surgical Education · doi:10.1007/s44186-025-00413-2 · Cited by 1
Large language models in model-driven engineering: a systematic mapping study
Zhang, W., Jiang, B., Yuhong, F., Cheng, H., Hummel, M., Scotti, V., Hagel, N., Li, J., Großmann, G., Stumptner, M., et al. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10921-4 · Cited by 6
Generative Artificial Intelligence for AgileRE: A Systematic Mapping
Cornide-Reyes, H., Angulo, V., & Monsalves, D. (2025) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc67219.2025.11420613
Bug Detective and Quality Coach: Developers’ Mental Models of AI-Assisted IDE ToolsPreprint
Buono, P., Cerullo, M., Cirillo, S., Desolda, G., Greco, F., Guglielmi, E., Margarella, G., Polese, G., Scalabrino, S., & Tucci, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6411746
Seemann, M. (2026) · Springer eBooks · doi:10.1007/978-3-658-50335-2_11
The Efficacy of Exams in Assessing Student Learning in Capstones
Fazackerley, S., & Hui, B. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786580.3786989
The Impact of LLMs on Software Development
Vasanthakumar, S., & Wickramarachchi, R. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499962 · Cited by 1
“At the same time, LLMs aid learning programming languages and libraries, potentially replacing traditional code-search methods [8].”
Bug detective and quality coach: Developers’ mental models of AI-assisted IDE tools
Buono, P., Cerullo, M., Cirillo, S., Desolda, G., Greco, F., Guglielmi, E., Margarella, G., Polese, G., Scalabrino, S., & Tucci, C. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101157
A Diffractive Analysis of GenAI through Perspectives on Productivity
Swift, I. P., & Chattopadhyay, D. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3808045.3808074 · Cited by 1
Designing Human-Centered Code Completion Tools in the Age of Generative AI
Desolda, G., Esposito, A., Greco, F., Tucci, C., Buono, P., & Piccinno, A. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3755957
Large Language Models in the Business World: Usage Areas, Benefits, Impacts, and Future Perspectives
KARACA, M. F., & Aslan, K. (2026) · Acta Infologica · doi:10.26650/acin.1663665
One More Prompt: LLM Coding is a Game You Can’t Quite Put Down - A Provocation
Beale, R. (2026) · Designing Interactive Systems Conference · doi:10.1145/3802974.3809412
Flores, F. D. L. C., Flores, P. A. D. L. C., Garcia, T., & Frausto, H. R. (2026) · Journal of Integrative Science and Societal Impact · doi:10.29121/jissi.v2.i2.2026.35
Multi-Agent LLM Architecture for Systems Engineering Automation
Padubrin, M., Kulzer, A. C., & Guerocak, E. (2026) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2026-01-0777
Security Analysis of LLM-Generated Web API Backends
Khan, A. A., Rauti, S., & Mäkilä, T. (2026) · IEEE Access · doi:10.1109/access.2026.3727682
Informed Skepticism: Navigating The Code Hallucination Dilemma
Faker, O., & Toker, Ş. (2026) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec69317.2026.11418436
“This is changing how we do software engineering in practice [3][4].”
Nizamani, S. B., Chandrashekar, N. D., Ellis, M., & Ramakrishnan, N. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60967
Mining Type Constructs Using Patterns in AI-Generated Code
Lee, I., Hassan, T. U., & Hindle, A. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793578
The Longevity of Mental Health Apps: A Survival Analysis of the Largest Mental Health App Database
Herpertz, J., Dwyer, B., Flathers, M., Opel, N., Otte, C., Gold, S. M., Ebner‐Priemer, U., Schwarz, J., Linardon, J., Ryan, S., et al. (2026) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20260155 · Cited by 1
Mayer, S., Knierim, P., Villa, S., & Schmidt, A. (2026) · Springer eBooks · doi:10.1007/978-3-319-27648-9_119-1
Self-citationA Systematic Literature Review on Intelligent Tools and the Developer Experience
Galván, D. A. Á., Benítez–Guerrero, E., & Mezura-Godoy, C. (2025) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft66928.2025.00046
How students use generative AI for software testing: An observational study
Ardıç, B., Dilavrec, Q. L., & Zaidman, A. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10898-0 · Cited by 8
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4
LLM Firewall: Private LLM Program Repair Using Hybrid Intent Inference
Akyol, G., Latifoğlu, İ., Kaya, A., & Tüzün, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36590-3_25
Zhou, Y., Sun, C., Teo, H., Xu, S. X., & Zhang, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6515379 · Cited by 2
Aguilar-López, R., Sanchéz-García, Á. J., Pérez-Arriaga, J. C., & Limón, X. (2025) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft66928.2025.00047 · Cited by 2
“Many individuals prefer consulting GenAI over interacting [13], [14], [17]–[19], [26], [28], [29], [32], [33], [38], [45], [47], [52]–[55] 17 Time-saving through elimination of repetitive tasks and documentation generation [13], [18], [26], [28], [29], [31]–[33], [47], [53], [56] 11 Support in…”
InfluentialZare, A., & Assal, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29723-5_35
“Integrating LLMs into the software development process has been shown to increase developers’ productivity [34,43] and can aid different tasks such as code reviews, software testing, and helping developers understand a codebase [17,19,23,44,45].”
Schadow, F., Gerling, K., Kretzer, F., & Maedche, A. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3805029.3818286
Efecto de la IA sobre las habilidades técnicas de los desarrolladores de software
Jácome, A. J. M. (2026) · Revista Tecnológica - ESPOL · doi:10.37815/rte.v38n1.1417
TGIF: the evolution of developer commit times
Karyotakis, I., Talos, E., & Spinellis, D. D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10767-2 · Cited by 1
“Studies show that AI-assisted programming improves productivity (Weber et al. 2024 ) and can transform the development lifecycle by offloading boilerplate tasks, generating code snippets, and enabling shorter, task-focused sessions (Qiu et al. 2025; Barenkamp et al. 2020).”
Metadata
- Authors
- Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-06-17
- Volume
- 8
- Issue
- EICS
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4399723222
- Semantic Scholar
- 92ac5e2f527984494231459e3f7da4f61790d08f
- Cited by
- 64
- References
- 59
- Citations by year
- 2026: 37 · 2025: 24 · 2024: 3
- FWCI
- 24.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, AI-assisted programming, code generation, developer productivity, GitHub Copilot, GPT-3, conversational AI, Python programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.