The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Significant Productivity Gains through Programming with Large Language Models
Open accessAbstract
Large language models like GPT and Codex drastically alter many daily tasks, including programming, where they can rapidly generate code from natural language or informal specifications. Thus, they will change what it means to be a programmer and how programmers act during software development. This work explores how AI assistance for code generation impacts productivity. In our user study (N=24), we asked programmers to complete Python programming tasks supported by a) an auto-complete interface using GitHub Copilot, b) a conversational system using GPT-3, and c) traditionally with just the web browser. Aside from significantly increasing productivity metrics, participants displayed distinctive usage patterns and strategies, highlighting that the form of presentation and interaction affects how users engage with these systems. Our findings emphasize the benefits of AI-assisted coding and highlight the different design challenges for these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174 · cited 1×
“While breaking down complex interaction patterns into individual interactions is a well-established pattern [8], prior work (e.g., Forsgren et al. [17] typically considers a higher number of interactions as indicative of high activity.”
Cited by
A Survey on LLM-Based Code Generation for Low-Resource and Domain-Specific Programming Languages
Joel, S., Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770084 · Cited by 152
'I'm Categorizing LLM as a Productivity Tool': Examining Ethics of LLM Use in HCI Research Practices
Kapania, S., Wang, R., Li, T. J.-J., Li, T., & Shen, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711000 · Cited by 67
OntoGenix: Leveraging Large Language Models for enhanced ontology engineering from datasets
Val-Calvo, M., Aranguren, M. E., Mulero-Hernández, J., Almagro-Hernández, G., Deshmukh, P., Bernabé-Díaz, J. A., Espinoza-Arias, P., Sánchez-Fernández, J. L., Mueller, J., & Fernández‐Breis, J. T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104042 · Cited by 36
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Weinz, M., Zannone, N., Allodi, L., & Apruzzese, G. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736195 · Cited by 32
Metadata
- Authors
- Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-06-17
- Volume
- 8
- Issue
- EICS
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4399723222
- Semantic Scholar
- 92ac5e2f527984494231459e3f7da4f61790d08f
- Cited by
- 64
- References
- 59
- Citations by year
- 2026: 37 · 2025: 24 · 2024: 3
- FWCI
- 24.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, AI-assisted programming, code generation, developer productivity, GitHub Copilot, GPT-3, conversational AI, Python programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Significant Productivity Gains through Programming with Large Language Models. https://app.cochat.ai/papers/10.1145/3661145
- The paper
- Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S. (2024). Significant Productivity Gains through Programming with Large Language Models. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3661145
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.