LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Open accessThe DOI matches its Crossref record; 48 of the 59 references it cites were found in a registry or an index; none of the 26 sources its claims cite is checked for support yet.
Abstract
Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflowTiCoderfor guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.
Overview
CoChat Summary
## TL;DR
This paper proposes TICODER, an interactive workflow that uses LLM-generated tests to clarify user intent and prune/rank code suggestions. In a controlled user study and benchmark evaluation, TICODER improved correctness and reduced cognitive load, and also boosted pass@1 code-generation accuracy across four LLMs on MBPP and HumanEval.
## Key findings
- In the user study with 15 participants, Assistant 2 (TICODER-PASSFAIL) had mean correctness 0.84 vs 0.40 for the control (Assistant 1), with a significant difference reported as \(p = 0.001\); Assistant 3 (TICODER-OUTPUT) had mean correctness 0.64.
- Mean cognitive load was lower with TICODER: 28.00 for Assistant 2 and 29.52 for Assistant 3 versus 45.46 for Assistant 1; the paper reports significant reductions in cognitive load, mental demand, effort, and some stress/pace measures.
- Mean task time did not differ significantly: 327.7 seconds for Assistant 1, 284.15 seconds for Assistant 2, and 253.88 seconds for Assistant 3.
- On benchmarks, TICODER improved pass@1 accuracy across all studied models and datasets; the paper reports an average absolute improvement of 45.73% within 5 user interactions, and gives examples such as text-davinci-003 on MBPP improving from 49.16% baseline pass@1 to 68.04% with TICODER-PASSFAIL after one interaction and 77.00% with TICODER-OUTPUT.
## Methods
- User study: controlled within-subject study with 15 participants, 3 coding-evaluation tasks, and 3 assistants: control (no intent refinement), TICODER-PASSFAIL, and TICODER-OUTPUT.
- Tasks/data: three Python tasks from MBPP (LOWERUNDERSCORE, FIRSTMISSING, MAXPRODUCT); participants evaluated 5 code suggestions per task, with tests surfaced and used for pruning in the TICODER conditions.
- Analysis: mixed-effects regression models for correctness, time, and NASA-TLX cognitive-load measures, with participant ID and task as random effects; omnibus ANOVA and FDR correction for pairwise comparisons.
- Benchmark evaluation: MBPP (427 examples) and HumanEval (164 examples), using pass@k and a ranked variant pass@k@m under simulated user responses based on the hidden reference implementation.
## Limitations
- The paper states that the user study was highly controlled, so results may not scale to more complex, real-world code-generation scenarios; it also notes that specifying outputs in TICODER-OUTPUT may not always be straightforward.
- The benchmark evaluation uses MBPP and HumanEval, which the paper says may not be representative of real-world software development or other languages/problem domains.
- The paper notes that the offline benchmark setup uses an oracle/reference implementation as a proxy for user feedback, which represents an upper bound on improvement with real users.
- Evident from the text: participants were prevented from editing prompts and code suggestions, and tasks were limited to 5 suggestions and 15 minutes each, so the study does not reflect unrestricted real-world use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 17 Not checked · 105 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for Task 1, 3/4 participants that failed to identify the correct suggestion were using Assistant 1.”
plainstates an associationn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load.”
- Also stated in: § I. INTRODUCTION · plain
“We observe a significant reduction in cognitive effort reported by participants using either variant of TICODER over existing interaction mechanisms.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants shown this test correctly answered that this test should pass.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the mean is higher in Assistant 2 and 3, the effect is significant for Assistant 2 only with (p=0.001).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, we observe TICODER can boost smaller model pass@1 accuracy to levels comparable to much larger models, such as GPT-4-32k , within just one user interaction.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
51 more claims
Create a free account to see all 124 claims and to verify the 26 references still open.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 35
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ E. Results
“after just one user interaction code-davinci-002 on MBPP achieves 68.42% accuracy in the TICODER-PASSFAIL setting, which is in fact higher than the pass@1 accuracy of all three SOTA chat models: GPT-4-32k , GPT-4-turbo , and GPT-3.5-turbo .”
§ C. Models and Baselines
“| MBPP | GPT-4-turbo | 69.80 | 86.88 | 80.71”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ A. Impact on Task Correctness
“The mean correctness was 0.40 for participants using Assistant 1, 0.84 Assistant 2, and 0.64 Assistant 3.”
§ RQ1 How does TiCoder impact the performance of python developers evaluating AI generated code, in terms of task correctness, time, and cognitive load?
“We use a within subject design, such that each participant uses all three assistants , i.e. a different assistant for each task.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A. Impact on Task Correctness
“For Assistant 2, all participants were able to correctly identify that the test should fail.”
§ A. Impact on Task Correctness
“For participants using Assistant 2, 4/5 indicated that it should evaluate to 'False' while one participant indicated that it should (incorrectly) evaluate to 'True'.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ E. Results
“To answer RQ2, we evaluate the performance of four different models, with and without TiCoder in the TICODERPASSFAIL and TICODER-OUTPUT settings on MBPP and HumanEval datasets.”
§ E. Results
“All three chat models, GPT-3.5-turbo , GPT-4-turbo and GPT-4-32k demonstrate the highest accuracy for pass@1 with comparable performance across datasets. As expected, text-davinci-003 and code-davinci-002 , the two largest completion models, achieve the fourth and fifth baseline performance on both datasets.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LLM-based Test-driven Interactive Code Generation: User Study and Empirical Evaluation
“We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions”
§ E. Results
“TICODER-OUTPUT achieves an average absolute improvement of 45.73% in the code generation accuracy for both datasets and across all LLMs within 5 user interactions.”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“To correct for multiple comparisons and conduct False Discovery Rate (FDR) correction [41] for significant pairs of conditions.”
MethodDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 1×
“To test the impact of the workflow on cognitive effort, after each task we measure participants’ self-reported cognitive load via their responses to five NASA TLX questions [39], a standardized approach to measuring self-reported cognitive load used widely across disciplines [40].”
MethodNASA-Task Load Index (NASA-TLX); 20 Years Later
Hart, S. G. (2006) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120605000909 · Cited by 4,297 · cited 1×
“To test the impact of the workflow on cognitive effort, after each task we measure participants’ self-reported cognitive load via their responses to five NASA TLX questions [39], a standardized approach to measuring self-reported cognitive load used widely across disciplines [40].”
MethodTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1×
“AlphaCode [21] and CodeT [22] both propose techniques to improve code generation accuracy by generating tests using LLMs, and then grouping code suggestions by the set of tests that they satisfy.”
MethodCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundAutomating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 1×
“There has been a surge in research around training LLMs over programming language artifacts in just the last couple of years [1], [2], [3], [4], [5].”
BackgroundCode Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1×
“Following [1], [45], [46] for all models we settle on a temperature of 0.8, as it maximizes the number of examples for which at least one correct code is produced within 100 suggestions for k > 1 in pass @ k .”
MethodInfluentialOracle-guided component-based program synthesis
Jha, S., Gulwani, S., Seshia, S. A., & Tiwari, A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806833 · Cited by 510 · cited 4×
“To minimize the number of user interactions, it is desirable to prioritize tests that would result in the most number of incorrect code suggestions being pruned away [15], [29].”
BackgroundMethodInfluentialGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 3×
“Results suggest a need for interaction mechanisms that can support disambiguation; a critical feature of the usability of AI-Assistants [31], [32].”
BackgroundMethodProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends in Programming Languages · doi:10.1561/2500000010 · Cited by 319
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundCodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · cited 1×
“…example generation and disambiguation techniques in Programming By Examples (PBE) [17], and recent emerging ability of LLMs to generate tests [18], [19], [20] in this paper, we propose leveraging user-feedback through LLM-generated tests to improve the trust and correctness of LLM-generated…”
MethodSelf-citationWang, Q., Wen, C., Liu, D., Han, X., Yu, B., Chen, P., Qin, S., & Tian, C. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.esem.2026.25 · Cited by 282
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · cited 1×
“Following [1], [45], [46] for all models we settle on a temperature of 0.8, as it maximizes the number of examples for which at least one correct code is produced within 100 suggestions for k > 1 in pass @ k .”
MethodInfluentialDo Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259 · cited 1×
“In practice, this can often lead to users accepting code with subtle bugs while using LLMs [12], [13].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 2×
“In the case of LLM-based code generation, the generated tests not only help make natural language intent more precise and prune incorrect suggestions generated by the LLM, but can also serves as debugging aid for remaining suggestions and regression tests for future code edits [7].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?
Endres, M., Fakhoury, S., Chakraborty, S., & Lahiri, S. K. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660791 · Cited by 188
Self-citationInCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 2×
“2b: assert find_First_Missing([1,2,4,6]) ==0 .”
BackgroundThe Sketching Approach to Program Synthesis
Solar-Lezama, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10672-9_3 · Cited by 165 · cited 1×
“Similarly, work on program synthesis [23], [24] generates code that satisfies a formal specification either expressed as a logical specification or input-output tests [25].”
BackgroundTaking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 3×
“However, there are several challenges that arise when generating code from natural language specifications [8], [9].”
BackgroundCoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., & Xie, T. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623316 · Cited by 140
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128 · cited 1×
“In practice, this can often lead to users accepting code with subtle bugs while using LLMs [12], [13].”
BackgroundIs GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116 · cited 1×
“Several recent works [35], [36], [37] identify clear opportunities for improving the accuracy of LLM code generation techniques.”
BackgroundReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 2×
“Mozannar et al. [34] identify 12 core activities associated with using GitHub Copilot and find that programmers often iterate on their prompts until they obtain the suggestion they desire, and spend a significant amount of time verifying code suggestions.”
BackgroundMethodReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
A theory of formal synthesis via inductive learning
Jha, S., & Seshia, S. A. (2017) · Acta Informatica · doi:10.1007/s00236-017-0294-5 · Cited by 99 · cited 2×
“…user feedback using the code reference implementation as an idealized proxy, similar to prior works in oracle-guided inductive synthesis [15], [28] and interactive program synthesis [29], [30] where an an oracle (reference implementation or users) is queried to identify the output for a given…”
MethodUser Interaction Models for Disambiguation in Programming by Example
Mayer, M., Soares, G., Grechkin, M., Le, V., Marron, M., Polozov, O., Singh, R., Zorn, B., & Gulwani, S. (2015) · Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology · doi:10.1145/2807442.2807459 · Cited by 97 · cited 2×
“…code assistants, including how users should be able to disambiguate candidate programs or refine their initial specifications, echoing prior studies have indicated that disambiguation can be valuable in the context of assistants like GitHub Copilot [32] and traditional program synthesis tools [33].”
BackgroundMethodCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · cited 2×
“AlphaCode [21] and CodeT [22] both propose techniques to improve code generation accuracy by generating tests using LLMs, and then grouping code suggestions by the set of tests that they satisfy.”
MethodOn the Design of AI-powered Code Assistants for Notebooks
Mcnutt, A. M., Wang, C., Deline, R. A., & Drucker, S. M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 77 · cited 3×
“Results suggest a need for interaction mechanisms that can support disambiguation; a critical feature of the usability of AI-Assistants [31], [32].”
BackgroundMethodInteractive Program Synthesis by Augmented Examples
Zhang, T., Lowmanstone, L., Wang, X., & Glassman, E. L. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415900 · Cited by 57 · cited 2×
“However, prior works in this area appeal to an automatic symbolic engine (such as a constraint solver [30] or automata construction [17]) to generate distinguishing example inputs for a pair of programs, which is inconceivable for general purpose imperative programming languages such as Python.”
BackgroundMethodWhy Programming by Demonstration Systems Fail: Lessons Learned for Usable AI
Lau, T. (2009) · AI Magazine · doi:10.1609/aimag.v30i4.2262 · Cited by 48
Question selection for interactive program synthesis
Ji, R., Liang, J., Xiong, Y., Zhang, L., & Hu, Z. (2020) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3385412.3386025 · Cited by 47 · cited 2×
“However, prior works in this area appeal to an automatic symbolic engine (such as a constraint solver [30] or automata construction [17]) to generate distinguishing example inputs for a pair of programs, which is inconceivable for general purpose imperative programming languages such as Python.”
BackgroundMethodCodeGen2: Lessons for Training LLMs on Programming and Natural Languages
Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., & Zhou, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02309 · Cited by 35 · cited 1×
“5-7B [44] is an improvement on CodeGen-6B and is slightly larger, with 7B parameters.”
BackgroundRahmani, K., Raza, M., Gulwani, S., Le, V., Morris, D., Radhakrishna, A., Soares, G., & Tiwari, A. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485535 · Cited by 33 · cited 1×
“However, it would be interesting for future work to leverage user-provided tests to improve the quality of code generation, as explored in recent works [26], [27].”
BackgroundTowards Generating Functionally Correct Code Edits from Natural Language Issue DescriptionsPreprint
Fakhoury, S., Chakraborty, S., Musuvathi, M., & Lahiri, S. K. (2023) · arXiv.org · doi:10.48550/arxiv.2304.03816 · Cited by 27 · cited 2×
“In future work, we plan to extend and evaluate our implementation reflecting real-world scenarios including: more complex programs, an in-situ user study for various software development tasks, and an empirical evaluation on realistic benchmarks such as CoderEval [47] and NL2Fix [43].”
BackgroundMethodSelf-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 3×
“Results suggest a need for interaction mechanisms that can support disambiguation; a critical feature of the usability of AI-Assistants [31], [32].”
BackgroundMethodFormalizing Natural Language Intent into Program Specifications via Large Language ModelsPreprint
Endres, M., Fakhoury, S., Chakraborty, S., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.01831 · Cited by 6
Self-citationTOGA: A Neural Method for Test Oracle GenerationPreprint
Dinella, E., Ryan, G., Mytkowicz, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09262 · Cited by 5 · cited 2×
“For example, tests for stateful APIs comprises of a test-prefix as input and the output oracle consists of a non-trivial predicate (e.g., checking functional correctness of a stack object using the predicate s.pop() == a on a stack object s and element a ) [19].”
MethodSelf-citationMeaningful Human-in-the-Loop Checking of GenAI Synthesis for Restricted Languages
Prasad, S., Austen, S., Fisler, K., & Krishnamurthi, S. (2026) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecoop.2026.22 · Cited by 1
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 1×
“In practice, this can often lead to users accepting code with subtle bugs while using LLMs [12], [13].”
Background
Research works printed but not linked: 1
- Jain, 2022, Jigsaw: Large language models meet program synthesis, 1219
Cited by
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 54
“The Current Challenges of SE in the Era of Large Language Models 127:17 Some studies investigate the interactions between users and LLMs. Fakhoury et al. [41] propose a novel framework, TiCoder, to ensure the correctness of generated code.”
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
SpecRover: Code Intent Extraction via LLMs
Ruan, H.-F., Zhang, Y.-T., & Roychoudhury, A. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00080 · Cited by 152
“Compared to the previous works that discover additional specification based on user intent [30], we do not rely on interactive user feedback.”
Nettur, S. B., Karpurapu, S., Nettur, U., & Gajja, L. S. (2024) · IEEE Access · doi:10.1109/access.2024.3521407 · Cited by 43
Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G. (2025) · arXiv · doi:10.48550/arxiv.2508.20863 · Cited by 22
Metadata
- Authors
- Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K.
- Affiliations
- Microsoft (United States) (US)
- University of Pennsylvania (US)
- University of California San Diego (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-09
- Volume
- 50
- Issue
- 9
- Pages
- 2254-2268
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4400878080
- Cited by
- 120
- References
- 59
- Citations by year
- 2026: 56 · 2025: 59 · 2024: 5
- FWCI
- 45.12
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- large language models, unit test generation, cognitive load, user feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.