The DOI matches its Crossref record; 47 of the 59 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Open accessAbstract
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflowTiCoderfor guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.
Overview
CoChat Summary
## TL;DR
This paper proposes TICODER, an interactive workflow that uses LLM-generated tests to clarify user intent and prune/rank code suggestions. In a controlled user study and benchmark evaluation, TICODER improved correctness and reduced cognitive load, and also boosted pass@1 code-generation accuracy across four LLMs on MBPP and HumanEval.
## Key findings
- In the user study with 15 participants, Assistant 2 (TICODER-PASSFAIL) had mean correctness 0.84 vs 0.40 for the control (Assistant 1), with a significant difference reported as \(p = 0.001\); Assistant 3 (TICODER-OUTPUT) had mean correctness 0.64.
- Mean cognitive load was lower with TICODER: 28.00 for Assistant 2 and 29.52 for Assistant 3 versus 45.46 for Assistant 1; the paper reports significant reductions in cognitive load, mental demand, effort, and some stress/pace measures.
- Mean task time did not differ significantly: 327.7 seconds for Assistant 1, 284.15 seconds for Assistant 2, and 253.88 seconds for Assistant 3.
- On benchmarks, TICODER improved pass@1 accuracy across all studied models and datasets; the paper reports an average absolute improvement of 45.73% within 5 user interactions, and gives examples such as text-davinci-003 on MBPP improving from 49.16% baseline pass@1 to 68.04% with TICODER-PASSFAIL after one interaction and 77.00% with TICODER-OUTPUT.
## Methods
- User study: controlled within-subject study with 15 participants, 3 coding-evaluation tasks, and 3 assistants: control (no intent refinement), TICODER-PASSFAIL, and TICODER-OUTPUT.
- Tasks/data: three Python tasks from MBPP (LOWERUNDERSCORE, FIRSTMISSING, MAXPRODUCT); participants evaluated 5 code suggestions per task, with tests surfaced and used for pruning in the TICODER conditions.
- Analysis: mixed-effects regression models for correctness, time, and NASA-TLX cognitive-load measures, with participant ID and task as random effects; omnibus ANOVA and FDR correction for pairwise comparisons.
- Benchmark evaluation: MBPP (427 examples) and HumanEval (164 examples), using pass@k and a ranked variant pass@k@m under simulated user responses based on the hidden reference implementation.
## Limitations
- The paper states that the user study was highly controlled, so results may not scale to more complex, real-world code-generation scenarios; it also notes that specifying outputs in TICODER-OUTPUT may not always be straightforward.
- The benchmark evaluation uses MBPP and HumanEval, which the paper says may not be representative of real-world software development or other languages/problem domains.
- The paper notes that the offline benchmark setup uses an oracle/reference implementation as a proxy for user feedback, which represents an upper bound on improvement with real users.
- Evident from the text: participants were prevented from editing prompts and code suggestions, and tasks were limited to 5 suggestions and 15 minutes each, so the study does not reflect unrestricted real-world use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 17 Not checked · 105 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for Task 1, 3/4 participants that failed to identify the correct suggestion were using Assistant 1.”
plainstates an associationn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load.”
- Also stated in: § I. INTRODUCTION · plain
“We observe a significant reduction in cognitive effort reported by participants using either variant of TICODER over existing interaction mechanisms.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants shown this test correctly answered that this test should pass.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the mean is higher in Assistant 2 and 3, the effect is significant for Assistant 2 only with (p=0.001).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, we observe TICODER can boost smaller model pass@1 accuracy to levels comparable to much larger models, such as GPT-4-32k , within just one user interaction.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
51 more claims
Create a free account to see all 124 claims and to verify the 26 references still open.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 35
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ E. Results
“after just one user interaction code-davinci-002 on MBPP achieves 68.42% accuracy in the TICODER-PASSFAIL setting, which is in fact higher than the pass@1 accuracy of all three SOTA chat models: GPT-4-32k , GPT-4-turbo , and GPT-3.5-turbo .”
§ C. Models and Baselines
“| MBPP | GPT-4-turbo | 69.80 | 86.88 | 80.71”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ A. Impact on Task Correctness
“The mean correctness was 0.40 for participants using Assistant 1, 0.84 Assistant 2, and 0.64 Assistant 3.”
§ RQ1 How does TiCoder impact the performance of python developers evaluating AI generated code, in terms of task correctness, time, and cognitive load?
“We use a within subject design, such that each participant uses all three assistants , i.e. a different assistant for each task.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A. Impact on Task Correctness
“For Assistant 2, all participants were able to correctly identify that the test should fail.”
§ A. Impact on Task Correctness
“For participants using Assistant 2, 4/5 indicated that it should evaluate to 'False' while one participant indicated that it should (incorrectly) evaluate to 'True'.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ E. Results
“To answer RQ2, we evaluate the performance of four different models, with and without TiCoder in the TICODERPASSFAIL and TICODER-OUTPUT settings on MBPP and HumanEval datasets.”
§ E. Results
“All three chat models, GPT-3.5-turbo , GPT-4-turbo and GPT-4-32k demonstrate the highest accuracy for pass@1 with comparable performance across datasets. As expected, text-davinci-003 and code-davinci-002 , the two largest completion models, achieve the fourth and fifth baseline performance on both datasets.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LLM-based Test-driven Interactive Code Generation: User Study and Empirical Evaluation
“We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions”
§ E. Results
“TICODER-OUTPUT achieves an average absolute improvement of 45.73% in the code generation accuracy for both datasets and across all LLMs within 5 user interactions.”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“To correct for multiple comparisons and conduct False Discovery Rate (FDR) correction [41] for significant pairs of conditions.”
MethodDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 1×
“To test the impact of the workflow on cognitive effort, after each task we measure participants’ self-reported cognitive load via their responses to five NASA TLX questions [39], a standardized approach to measuring self-reported cognitive load used widely across disciplines [40].”
MethodNASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 1×
“To test the impact of the workflow on cognitive effort, after each task we measure participants’ self-reported cognitive load via their responses to five NASA TLX questions [39], a standardized approach to measuring self-reported cognitive load used widely across disciplines [40].”
MethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1×
“AlphaCode [21] and CodeT [22] both propose techniques to improve code generation accuracy by generating tests using LLMs, and then grouping code suggestions by the set of tests that they satisfy.”
MethodCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundA systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 1×
“There has been a surge in research around training LLMs over programming language artifacts in just the last couple of years [1], [2], [3], [4], [5].”
Background
Research works printed but not linked: 2
- Gulwani, 2011, Automating string processing in spreadsheets using input-output examples, 317
- Jain, 2022, Jigsaw: Large language models meet program synthesis, 1219
Cited by
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 123
“The Current Challenges of SE in the Era of Large Language Models 127:17 Some studies investigate the interactions between users and LLMs. Fakhoury et al. [41] propose a novel framework, TiCoder, to ensure the correctness of generated code.”
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00080 · Cited by 45
“Compared to the previous works that discover additional specification based on user intent [30], we do not rely on interactive user feedback.”
Nettur, S. B., Karpurapu, S., Nettur, U., & Gajja, L. S. (2024) · IEEE Access · doi:10.1109/access.2024.3521407 · Cited by 43
MetaIndux-PLC: A Control Logic-Guided LLM for PLC code generation in industrial control systems
Ren, L., Wang, H., Dong, J., Wang, H., Liu, S., Laili, Y., & Li, Z. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113673 · Cited by 19
Metadata
- Authors
- Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K.
- Affiliations
- Microsoft (United States) (US)
- University of Pennsylvania (US)
- University of California San Diego (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-09
- Volume
- 50
- Issue
- 9
- Pages
- 2254-2268
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4400878080
- Cited by
- 120
- References
- 59
- Citations by year
- 2026: 56 · 2025: 59 · 2024: 5
- FWCI
- 45.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- large language models, unit test generation, cognitive load, user feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. https://app.cochat.ai/papers/10.1109/tse.2024.3428972
- The paper
- Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024). LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2024.3428972
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.