Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 47 of the 59 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.

LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

Open access
Fakhoury, Sarah, Naik, Aaditya, Sakkas, Georgios, Chakraborty, Saikat, Lahiri, Shuvendu K. · 2024 · IEEE Transactions on Software Engineering

Abstract

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflowTiCoderfor guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.

Overview

CoChat Summary

## TL;DR

This paper proposes TICODER, an interactive workflow that uses LLM-generated tests to clarify user intent and prune/rank code suggestions. In a controlled user study and benchmark evaluation, TICODER improved correctness and reduced cognitive load, and also boosted pass@1 code-generation accuracy across four LLMs on MBPP and HumanEval.

## Key findings

  • In the user study with 15 participants, Assistant 2 (TICODER-PASSFAIL) had mean correctness 0.84 vs 0.40 for the control (Assistant 1), with a significant difference reported as \(p = 0.001\); Assistant 3 (TICODER-OUTPUT) had mean correctness 0.64.
  • Mean cognitive load was lower with TICODER: 28.00 for Assistant 2 and 29.52 for Assistant 3 versus 45.46 for Assistant 1; the paper reports significant reductions in cognitive load, mental demand, effort, and some stress/pace measures.
  • Mean task time did not differ significantly: 327.7 seconds for Assistant 1, 284.15 seconds for Assistant 2, and 253.88 seconds for Assistant 3.
  • On benchmarks, TICODER improved pass@1 accuracy across all studied models and datasets; the paper reports an average absolute improvement of 45.73% within 5 user interactions, and gives examples such as text-davinci-003 on MBPP improving from 49.16% baseline pass@1 to 68.04% with TICODER-PASSFAIL after one interaction and 77.00% with TICODER-OUTPUT.

## Methods

  • User study: controlled within-subject study with 15 participants, 3 coding-evaluation tasks, and 3 assistants: control (no intent refinement), TICODER-PASSFAIL, and TICODER-OUTPUT.
  • Tasks/data: three Python tasks from MBPP (LOWERUNDERSCORE, FIRSTMISSING, MAXPRODUCT); participants evaluated 5 code suggestions per task, with tests surfaced and used for pruning in the TICODER conditions.
  • Analysis: mixed-effects regression models for correctness, time, and NASA-TLX cognitive-load measures, with participant ID and task as random effects; omnibus ANOVA and FDR correction for pairwise comparisons.
  • Benchmark evaluation: MBPP (427 examples) and HumanEval (164 examples), using pass@k and a ranked variant pass@k@m under simulated user responses based on the hidden reference implementation.

## Limitations

  • The paper states that the user study was highly controlled, so results may not scale to more complex, real-world code-generation scenarios; it also notes that specifying outputs in TICODER-OUTPUT may not always be straightforward.
  • The benchmark evaluation uses MBPP and HumanEval, which the paper says may not be representative of real-world software development or other languages/problem domains.
  • The paper notes that the offline benchmark setup uses an oracle/reference implementation as a proxy for user feedback, which represents an upper bound on improvement with real users.
  • Evident from the text: participants were prevented from editing prompts and code suggestions, and tasks were limited to 5 suggestions and 15 minutes each, so the study does not reflect unrestricted real-world use.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tse.2024.3428972

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 17 Not checked · 105 Own results and interpretations

51 more claims

Create a free account to see all 124 claims and to verify the 26 references still open.

See all claims — free account
  • It may state two separate claims as one. 46
  • Probably not a substantial claim. 35

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 59 references

Research works printed but not linked: 2

  1. Gulwani, 2011, Automating string processing in spreadsheets using input-output examples, 317
  2. Jain, 2022, Jigsaw: Large language models meet program synthesis, 1219

Cited by

See all 120

Metadata

Authors
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K.
Affiliations
  • Microsoft (United States) (US)
  • University of Pennsylvania (US)
  • University of California San Diego (US)
Venue
IEEE Transactions on Software Engineering
Published
2024-09
Volume
50
Issue
9
Pages
2254-2268
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
  • 2326-3881
OpenAlex
W4400878080
Cited by
120
References
59
Citations by year
2026: 56 · 2025: 59 · 2024: 5
FWCI
45.12
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
large language models, unit test generation, cognitive load, user feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. https://app.cochat.ai/papers/10.1109/tse.2024.3428972
The paper
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024). LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2024.3428972

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.