Cited by 20Software Engineering ResearchSoftware Engineering Techniques and PracticesOpen Source Software Innovations

Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking

Tang, Ningzhi, Chen, Meng, Ning, Zheng, Bansal, Aakash, Huang, Yu, McMillan, Collin, Li, Toby Jia-Jun·2024·2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)

Abstract

The increasing use of large language model (LLM)-powered code generation tools, such as GitHub Copilot, is transforming software engineering practices. This paper investigates how developers validate and repair code generated by Copilot and examines the impact of code provenance awareness during these processes. We conducted a lab study with 28 participants tasked with validating and repairing Copilot-generated code in three software projects. Participants were randomly divided into two groups: one informed about the provenance of LLM-generated code and the other not. We collected data on IDE interactions, eye-tracking, cognitive workload assessments, and conducted semi-structured interviews. Our results indicate that, without explicit information, developers often fail to identify the LLM origin of the code. Developers exhibit LLM-specific behaviors such as frequent switching between code and comments, different attentional focus, and a tendency to delete and rewrite code. Being aware of the code’s provenance led to improved performance, increased search efforts, more frequent Copilot usage, and higher cognitive workload. These findings enhance our understanding of developer interactions with LLM-generated code and inform the design of tools for effective human-LLM collaboration in software development.

Integrity signals

Registry

Crossref · 10.1109/vl/hcc60511.2024.00015

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Tang et al. conducted a focused study on how developers validate and repair LLM-generated code, finding that awareness of AI authorship affected both behavior and cognitive load.

    “Recently, Tang et al. [15] conducted a focused study on how developers validate and repair LLM-generated code, finding that awareness of AI authorship affected both behavior and cognitive load.”
  • Tang et al. [15] observed that developers similarly shifted between prompt and code to resolve mismatches between intended and generated behaviour.

    “Which is in line with prior observations by Tang et al. [15], where developers similarly shifted between prompt and code to resolve mismatches between intended and generated behaviour.”
  • Tang et al. [15] observed similar behaviour when developers repaired code generated by GitHub Copilot.

    “A similar behaviour was observed by Tang et al. [15] when developers repaired code generated by GitHub Copilot.”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J.
Affiliations
  • University of Notre Dame (US)
  • Vanderbilt University (US)
Venue
2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)
Publisher
IEEE
Published
2024-09-02
Pages
40-46
ISSN
1943-6092 (ISSN-L)
OpenAlex
W4403447861
Cited by
18
References
49
Citations by year
2026: 7 · 2025: 10 · 2024: 1
FWCI
6.84
Type
conference paper
Language
en
Open access
closed
Locations
IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (publishedVersion)
Funders
Google
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
LLM-generated code, GitHub Copilot, code validation, code repair, developer behavior, human-LLM collaboration, mental workload, eye tracking, attentional focus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review