CoChat

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Significant Productivity Gains through Programming with Large Language Models

Open access
Weber, Thomas, Brandmaier, Maximilian, Schmidt, Albrecht, Mayer, Sven · 2024 · Proceedings of the ACM on Human-Computer Interaction

Abstract

Large language models like GPT and Codex drastically alter many daily tasks, including programming, where they can rapidly generate code from natural language or informal specifications. Thus, they will change what it means to be a programmer and how programmers act during software development. This work explores how AI assistance for code generation impacts productivity. In our user study (N=24), we asked programmers to complete Python programming tasks supported by a) an auto-complete interface using GitHub Copilot, b) a conversational system using GPT-3, and c) traditionally with just the web browser. Aside from significantly increasing productivity metrics, participants displayed distinctive usage patterns and strategies, highlighting that the form of presentation and interaction affects how users engage with these systems. Our findings emphasize the benefits of AI-assisted coding and highlight the different design challenges for these systems.

Overview

Integrity signals

Registry

Crossref · 10.1145/3661145

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Cited by

See all 64

Metadata

Authors
Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S.
Affiliations
Ludwig-Maximilians-Universität München (DE)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2024-06-17
Volume
8
Issue
EICS
Pages
1-29
ISSN
2573-0142 (ISSN-L)
OpenAlex
W4399723222
Cited by
64
References
59
Citations by year
2026: 37 · 2025: 24 · 2024: 3
FWCI
24.00
Type
journal article
Language
en
Open access
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
large language models, AI-assisted programming, code generation, developer productivity, GitHub Copilot, GPT-3, conversational AI, Python programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.