CoChat
Sign Up
Cited by 154Software Engineering ResearchSoftware Engineering Techniques and PracticesTeaching and Learning Programming

In-IDE Code Generation from Natural Language: Promise and Challenges

Open access
Xu, Frank F., Vasilescu, Bogdan, Neubig, Graham·2022·ACM Transactions on Software Engineering and Methodology
DOI 10.1145/3487569
Open this paper in CoChat — free accountOpen PDF

Abstract

A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code , especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries , but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this article, we perform the first comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking, “At the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?” To facilitate the study, we first develop a plugin for the PyCharm IDE that implements a hybrid of code generation and code retrieval functionality, and we orchestrate virtual environments to enable collection of many user events (e.g., web browsing, keystrokes, fine-grained code edits). We ask developers with various backgrounds to complete 7 varieties of 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Further analysis identifies several pain points that could improve the effectiveness of future machine learning-based code generation/retrieval developer assistants and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies on this topic, as well as development of better code generation models.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper reports a controlled PyCharm study of an in-IDE natural-language-to-code assistant that combines code generation with Stack Overflow retrieval. Participants generally liked it, but the quantitative evidence did not show faster or more correct task completion, and query quality, missing context, and post-insertion edits emerged as key obstacles.

Key findings

  • Across 31 participants and 14 Python tasks in 7 categories, using the plugin produced no statistically significant difference in task completion time or correctness; median correctness was 7/10 with the plugin versus 6/10 without, but the regression was not significant. Code written with the plugin averaged about 4 more source lines, while cyclomatic complexity showed no significant difference.
  • A logistic regression predicting whether users chose generated versus retrieved snippets reached 65.9% accuracy against a 55.6% retrieval-only baseline. Generation was favored for basic Python, file, and data-structure queries, while retrieval was favored for terms tied to datetime, matplotlib, sklearn, and cross-validation.
  • In a human-oracle check of 50 sampled queries, only 20 (40%) were sufficient as written, rising to 34 (68%) when the source file context was provided. Query quality was associated with generation success (phi = 0.37, p = 0.008 as-is; phi = 0.45, p = 0.001 with context), and generation failed on 31/50 queries (62%), mostly under-specified ones (23/31 failures).
  • Generated snippets showed no significant token-length change after edits (p = 0.345; mean edit distance 5.2 tokens), whereas retrieved snippets changed significantly (p = 1.195e-7; median difference 18 tokens; mean edit distance 13.2 tokens). Users added identifiers and keywords and removed numeric literals and boilerplate; 15/31 rated the plugin experience neutral, 15/31 somewhat positive, and 1 somewhat negative. Using the plugin was associated with 2.8 fewer in-browser searches, though this was smaller than the random user-intercept standard deviation of about 4.

Methods

  • The authors built a PyCharm plugin that takes English queries and returns ranked code candidates from a neural semantic-parsing generation model and a custom Bing/Stack Overflow retrieval engine. They ran a controlled study with 31 participants, each assigned 8 tasks (4 with plugin, 4 without) drawn from 14 tasks across 7 categories, inside an instrumented virtual machine that logged IDE activity, browser use, keystrokes, queries, selections, and edits.
  • Quantitative analyses used mixed-effects regressions for completion time, correctness, source lines of code, and cyclomatic complexity; logistic regression with bag-of-words features for generation-versus-retrieval choice; a human expert oracle on 50 sampled queries; and Wilcoxon signed-rank tests plus edit-distance measures for snippet edits. Qualitative data came from post-task and final surveys coded for themes.

Limitations

  • Stated by the paper: the task set may not have offered enough opportunities to benefit from the plugin; 31 participants may be too few to detect small effects; generation results were always shown before retrieval results and source icons were visible, which could bias choices; unsuccessful queries were not logged; the oracle analysis used one expert and a 50-query sample with a 13% margin of error; only 303 of 397 successful queries had uploaded edits; experience-based task assignment limits generalization to novices; and qualitative coding involved only one researcher.
  • Evident from the text: the study used a controlled virtual environment and relatively short tasks rather than real project work; it covered only Python and PyCharm with specific generation and retrieval systems; the retrieval comparison was a custom Bing wrapper rather than the proprietary Bing Developer Assistant; and ordering effects were not experimentally tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3487569

Cites

156 of 157 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 19 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    The BLEU accuracy score commonly used in natural language machine translation has only weak correlation with the semantic correctness of translated source code.

    “the BLEU [89] accuracy score commonly used in natural language machine translation has only weak correlation with the semantic correctness of the translated source code [110].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Tran et al. [110] 110. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migra- tion? In IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165-176. source
    • cites [110] 110. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migra- tion? In IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165-176. source
  • Premisecitation

    BLEU is an imperfect metric of correctness of a source code snippet.

    “However, many code snippets are equivalent in functionality but differ quite largely in terms of token sequences, or differ only slightly in token sequence but greatly in functionality, and thus BLEU is an imperfect metric of correctness of a source code snippet [110].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [110] 110. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migra- tion? In IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165-176. source
  • Findingown data

    The retrieval results sometimes contain spurious code not part of the natural language intent.

    “Note how the retrieval results sometimes contain spurious code, not part of the natural language intent”
    Quote in the paperNot a substantial claim
  • Findingown data

    The analysis demonstrates when developers prefer code generation over code retrieval and vice versa.

    “demonstrates when developers prefer code generation over code retrieval and vice versa.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Quantitative results regarding increased productivity, code quality, or program correctness when using the NL2Code plugin were inconclusive.

    “quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 59
  • May combine two claims. 33
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The code generation model by Xu et al. [117] reports state-of-the-art accuracy on the CoNaLa benchmark dataset.

    “This model reports state-of-the-art accuracy on the CoNaLa benchmark dataset [122]”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [117] 117. Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020. Incorporating external knowledge through pre-training for natural language to code generation. In Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 6045-6052.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [122] 122. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Repositories (MSR). ACM, 476-486. DOI:https://doi.org/10.1145/3196398.3196408 source

      The Results and Discussion (pp. 7–8) evaluate extraction of natural-language/code pairs from Stack Overflow, not code-generation accuracy on CoNaLa. Searches for CoNaLa and Xu’s code-generation model found no passage in our copy reporting the attributed benchmark result. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisereasoning

    The default model may suffer from heterogeneity bias because task category experience varies within and across participants, so experience and user may be correlated, potentially biasing estimates.

    “As specified, our default model may suffer from heterogeneity bias [13].”
    • Also stated in: § Methodology. · hedged
      “This means that experience (a fixed effect) and user (a random effect) may be”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [13] 13. Andrew Bell, Malcolm Fairbrother, and Kelvyn Jones. 2019. Fixed and random effects models: Making an informed choice. Qual. Quant. 53, 2 (2019), 1051-1074. source

  • May combine two claims.

    Premisecitation

    Splitting experience into between-subject and within-subject components (de-meaning) addresses potential heterogeneity bias and allows mixed-effects models to model both within- and between-subject effects.

    “Therefore, to address potential heterogeneity bias, we split our fixed effect (experience) into two variables, each representing a different source of variation: a participant's average experience across all task categories (experience_btw) and the deviation for each task from the participants's overall mean experience (experience_wi).This process is known as de-meaning or person-mean centering [”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [34] 34. Andrew Gelman and Jennifer Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cam- bridge University Press. source
    • cites [13] 13. Andrew Bell, Malcolm Fairbrother, and Kelvyn Jones. 2019. Fixed and random effects models: Making an informed choice. Qual. Quant. 53, 2 (2019), 1051-1074. source
    • cites [79] 79. Yair Mundlak. 1978. On the pooling of time series and cross section data. Economet.: J. Economet. Societ. (1978), 69-85. source

  • May combine two claims.

    Premisecitation

    Many code snippets are equivalent in functionality but differ quite largely in terms of token sequences, or differ only slightly in token sequence but greatly in functionality.

    “However, many code snippets are equivalent in functionality but differ quite largely in terms of token sequences, or differ only slightly in token sequence but greatly in functionality, and thus BLEU is an imperfect metric of correctness of a source code snippet [110].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [110] 110. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migra- tion? In IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165-176. source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The two measures (SLOC and CC) tend to be correlated, but not strongly enough to conclude that CC is redundant with SLOC.

    “The two measures tend to be correlated, but not strongly enough to conclude that CC is redundant with SLOC [61].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [61] 61. Davy Landman, Alexander Serebrenik, Eric Bouwers, and Jurgen J. Vinju. 2016. Empirical analysis of the relationship between CC and SLOC in a large corpus of Java methods and C functions. J. Softw. Evolut. Process 28, 7 (2016), 589- 618. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The code generation model can copy important information such as string literals and variable names from the input natural language intent.

    “has the ability to copy important information such as string literals and variable names from the input natural language intent”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Tasks with code fragments automatically generated or retrieved using the plugin were, on average, neither completed faster nor more correctly than tasks without any NL2Code assistant.

    “tasks with code fragments automatically generated or retrieved using our plugin were, on average, neither completed faster nor more correctly than tasks where participants did not use any NL2Code assistant.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The code generation model can generate syntactically correct code snippets by construction.

    “Note how the model can generate syntactically correct code snippets by construction”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The user-written full intent resulted in a sufficient answer 34/59 times, and the simplified intent without variable names returned a sufficient answer 36/59 times.

    “the user-written full intent resulted in a sufficient answer 34/59 times, and the simplified intent without variable names returned a sufficient answer 36/59 times”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The exact-match overlap between the top-1 results was 22/59, and overlap between the top-7 result lists was 182/(59*7).

    “We also measured the exact-match overlap between the top-1 results and found it to be 22/59, and overlap between the top-7 result lists was 182/(59*7).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most of the variance in all response variables is explained by the two random effects (task and user), indicating significant user-to-user and task-to-task variability.

    “with most of the variance explained attributable to the two random effects (task and user)-there is significant user-to-user and task-to-task variability in all response variables.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Findingown data

    Using the plugin assumes one has a 'good background understanding of the underlying principles/modules/frameworks', and they primarily need help with 'look[ing] up little syntax bits that you have forgotten'.

    “A closely related theme is that using the plugin assumes one has a”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some respondents mention that it would be useful to include additional (links to) explanations and documentation alongside the returned code snippets.

    “Some respondents mention that it would be useful to include additional (links to) explanations and documentation alongside the returned code snippets so the user could understand what the snippet is supposed to do, or even”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingown data

    The quality of code generation and retrieval results could be higher.

    “The quality of code generation and retrieval results could be higher.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Code retrieval snippets tend to be longer than code generation ones both before (p < 2.2e-16; median difference 18 tokens, 95% CI 14 to Infinity) and after edits (p = 2.657e-14; median difference 10 tokens, 95% CI 7 to Infinity).

    “We also note that code retrieval snippets tend to be longer than code generation ones both before (p < 2.2e-16; median difference 18 tokens, with a 95% confidence interval from 14 to Infinity) and after edits (p = 2.657e-14; median difference 10 tokens, with a 95% confidence interval from 7 to Infinity).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingown data

    Users could benefit from additional context.

    “Users could benefit from additional context.”
    • Also stated in: § Results. · hedged
      “The plugin could benefit from additional context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S2 (checkable), S3 (load-bearing).

    Findingown data

    Querying the plugin could be interactive.

    “Querying the plugin could be interactive.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In some cases, results from the plugin were 'not related to the search', and users 'didn't find what [they were] searching for'.

    “In some cases, results from the plugin were”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingown data

    Some participants suggest to make querying interactive, dialogue-based, rather than unidirectional.

    “Finally, some participants suggest to make querying interactive, dialogue-based, rather than unidirectional.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Respondents expressed appreciation for both the generation and retrieval results, and there was little expression of preference for one method over the other.

    “Respondents expressed appreciation for both the generation and retrieval results, and there was little expression of preference for one method over the other”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participants comment on how the plugin's query syntax takes some getting used to, referring in particular to the way the code generation model expects queries to include variables, while the web search code retrieval engine allows users to only use keywords.

    “Participants also comment on how the plugin's query syntax takes some getting used to (P2, P12, P15), referring in particular to the way the code generation model expects queries to include variables, while the web search code retrieval engine allows users to only use keywords.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some code snippets retrieved from Stack Overflow are in the format of IPython REPL, which uses 'In' and 'Out' to separate Python source code and execution outputs.

    “We find that it is mostly due to some of the code snippets retrieved from Stack Overflow being in the format of IPython REPL, which uses”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    Some participants suggest that the plugin could be 'smarter' if it becomes more aware of the local context in the developer's IDE.

    “Some participants suggest that the plugin could be”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Even when writing code manually, developers still consult the Web and Stack Overflow and copy-paste code therein.

    “Among the possible explanations for why we do not find supporting evidence for the”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Overall, study participants report having a neutral (15/31; 48.4%) or at least somewhat positive (15/31; 48.4%) experience using the NL2Code plugin, with only one participant rating their experience as somewhat negative.

    “Overall, study participants report having a neutral (15/31; 48.4%) or at least somewhat positive (15/31; 48.4%) experience using the NL2Code plugin, with only one participant rating their experience as somewhat negative.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingreasoning

    We observed that users are not always formulating queries in the way that we would expect, perhaps because they are used to traditional search engines that are more robust to noisy inputs and designed for keyword-based search.

    “We observed that users are not always formulating queries in the way that we would expect, perhaps because they are used to traditional search engines that are more robust to noisy inputs and designed for keyword-based search.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Across our sample the median score per task was 7 out of 10 when using the plugin and 6 when not using the plugin, but the multivariate regression analysis did not find the difference to be statistically significant.

    “Even though across our sample the median score per task was 7 out of 10 when using the plugin and 6 when not using the plugin, the multivariate regression analysis did not find the difference to be statistically significant.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    On average, code written by users in the plugin condition is approximately 4 source lines of code longer than code written by users without using the plugin.

    “On average, the code written by users in the plugin condition is ~4 source lines of code longer than the code written by users without using the plugin.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Our analysis of input queries and resulting code snippets for RQ 2 shows that the code generation model produces fundamentally different results than the (simple) code retrieval engine we used for comparison, and that study participants choose snippets returned by the code generation model almost as frequently as they do snippets from the code retrieval engine.

    “Our analysis of input queries and resulting code snippets for RQ 2 shows that the code generation model produces fundamentally different results than the (simple) code retrieval engine we used for comparison, and that study participants choose snippets returned by the code generation model almost as frequently as they do snippets from the code retrieval engine.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using the NL2Code plugin does not seem to help study participants significantly improve their scores either, despite there being room for improvement.

    “However, using the NL2Code plugin does not seem to help our study participants significantly improve their scores either, despite there being room for improvement.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    As our respondents commented in the post-test survey, getting used to formulating queries this way takes some practice.

    “As our respondents commented in the post-test survey, getting used to formulating queries this way takes some practice.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The mixed-effects regression models fit the data reasonably well, with R2c = 50% for SLOC and R2c = 27% for CC.

    “The models fit the data reasonably well (R 2 c = 50% for SLOC, R 2 c = 27% for CC).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    A dedicated search engine already incorporates advanced indexing and ranking mechanisms driven by user interaction data, therefore it is preferable to using the internal Stack Overflow search engine directly.

    “A dedicated search engine already incorporates advanced indexing and ranking mechanisms in its algorithms, driven by user interaction data, therefore it is preferable to using the internal Stack Overflow search engine directly [115].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [115] 115. Yi Wei, Nirupama Chandrasekaran, Sumit Gulwani, and Youssef Hamadi. 2015. Building Bing Developer Assistant. Technical Report. MSR-TR-2015-36, Microsoft Research.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Only 31 out of the 64 eligible applicants after the pre-test survey actually completed their assigned tasks.

    “Only 31 out of the 64 eligible applicants after the pre-test survey actually completed their assigned tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The top 1% most extreme values were removed as outliers.

    “First, we removed as outliers the top 1% most extreme values.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The Human Subjects research protocol underwent review and was approved by the Carnegie Mellon University Institutional Review Board.

    “This Human Subjects research protocol underwent review and was approved by the Carnegie Mellon University Institutional Review Board.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductreasoning

    The time spent for uploading edits in the plugin setting is minimal because the plugin automatically helps the user remove auto-generated comments with a keyboard shortcut.

    “However the time spent for uploading is minimal as the plugin automatically helps the user to remove the auto-generated comments with only a press of a keyboard shortcut.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    The study design imposes a hierarchical structure: the same participant completed multiple tasks, so individual observations are not independent.

    “First, our study design imposes a hierarchical structure during data collection, therefore the individual observations are not independent-by construction, the same participant will have completed multiple tasks over the course of the study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Within each task category, one of the two tasks was randomly assigned to the NL2Code plugin (treatment) condition and the other to the no plugin (control) condition.

    “in each category, we randomly assign one of the 2 tasks to the NL2Code plugin (treatment) condition and the other task to the no plugin (control) condition.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Variance inflation factor (VIF) for all predictors was below 3, indicating multicollinearity is not an issue.

    “we checked for collinearity among the predictors we use the variance inflation factor (VIF) [22]; all were below 3, i.e., multicollinearity is not an issue [58].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [22] 22. J. Cohen. 2003. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Lawrence Erlbaum.

      Not checked in the free run.

    • cites [58] 58. Ned Kock and Gary Lynn. 2012. Lateral collinearity and misleading results in variance-based SEM: An illustration and recommendations. J. Assoc. Inf. Syst. 13, 7 (2012).

  • Classified as probably not a substantial claim.

    Premisereasoning

    Tasks vary in difficulty by construction, so their response variables (e.g., task completion times) can be correlated with the tasks themselves.

    “Moreover, tasks vary in difficulty, again by construction, therefore it is expected that their corresponding response variables, e.g., task completion times, can be correlated with the tasks themselves; e.g., on average, more complex tasks will take longer to complete.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationown data

    These queries (the 8 failure cases) are targets where further improved code generation techniques could improve the utility of the plugin.

    “These queries are targets where further improved code generation techniques could improve the utility of the plugin.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To account for possible breaks and obtain more accurate estimates of time spent on task, the study subtracted time intervals when participants used explicit pause/resume functionality, as well as all intervals of idle time with no mouse or keyboard activity for two minutes or more.

    “To account for possible breaks and obtain more accurate estimates of time spent on task, we further subtract the time intervals when participants used our explicit pause/resume functionality, as well as all intervals of idle time in which participants had no mouse or keyboard activity for two minutes or more”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The experience-based filtering of participants means that different conclusions might be reached if novice programmers or programmers without domain expertise used the plugin instead.

    “However, at the same time it also means that different conclusions might be reached if novice programmers or programmers without domain expertise used the plugin instead.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The 31 participants had relatively diverse backgrounds: 12 (39%) were software engineers, 11 (35%) were computer science students, 2 (6%) were researchers, and 6 (19%) were other occupations.

    “Their backgrounds were relatively diverse; of the 31 participants, 12 (39%) were software engineers and 11 (35%) were computer science students, with the rest being researchers (2, 6%), and other occupations (6, 19%).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Each submission was graded blindly, i.e., graders did not know whether it came from the control or treatment condition.

    “To reduce subjectivity, we graded each submission blindly (i.e., not knowing whether it came from the control or treatment condition)”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Participants vary in self-reported Python and task category experience, so experience-related effects should be separated from effects of using the plugin.

    “Finally, the participants vary in their self reported levels of Python and individual task category experience; we should separate experience-related effects from effects of using the plugin, if any.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Rubric items were automated when possible, e.g., using input-output test cases for deterministic tasks and checking abstract syntax tree nodes for required types.

    “we automated rubric items when possible, e.g., using input-output test cases for the deterministic tasks and checking if the abstract syntax tree contains nodes corresponding to required types (data structures) such as dictionaries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The two metrics (SLOC and CC) are only crude approximations of the complex, multifaceted concept of code quality.

    “Among the possible explanations for why we do not find supporting evidence for the”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The two metrics (SLOC and CC) are only crude approximations of the complex, multifaceted concept of code quality.

  • Classified as probably not a substantial claim.

    Premisenone given

    The tool used retrieved code snippets as they appeared on Stack Overflow.

    “This may be explained by the fact that the tool used retrieved code snippets as they appeared on Stack Oveflow, and thus many retrieved code snippets contain additional boilerplate code required for initialization or setup and hard-coded example inputs and outputs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    When integrating these snippets, users will have to remove this superfluous text.

    “When integrating these snippets, the users will have to remove this superfluous text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    In these cases, the code snippets seem to miss substantive parts and relevant functionality, which may be partly due to the lack of specificity described in the previous section.

    “Stated differently, in these cases the code snippets seem to miss substantive parts and relevant functionality, which also may be partly due to the lack of specificity described in the previous section.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    When integrated into the development workflow, e.g., through an IDE plugin, such systems can help developers focus by reducing the need for context switching, further improving their productivity.

    “When integrated into the development workflow, e.g., through an IDE plugin, such systems can help developers focus by reducing the need for context switching, further improving their productivity.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    The NL2Code generation model we experimented with in this study was trained on natural language queries that are not only complete English sentences, but also include references to variables or literals involved with an intent, specially delimited by dedicated syntax (grave accents).

    “The NL2Code generation model we experimented with in this study was trained on natural language queries that are not only complete English sentences, but also include references to variables or literals involved with an intent, specially delimited by dedicated syntax (grave accents).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The evidence so far suggests that the code generation model does add some level of sophistication, and customization of results to the developers' intent (e.g., composing function calls), compared to what any code retrieval engine could.

    “Therefore, the evidence so far suggests that the code generation model does add some level of sophistication, and customization of results to the developers' intent (e.g., composing function calls), compared to what any code retrieval engine could.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The study controlled the development environment for different users by using a complete virtual machine-based environment.

    “To control the user study's development environment for different users as much as possible, and to enable data collection and activity recording outside the IDE (e.g., web browsing activity during the development), we design a complete virtual machine-based environment for users to access remotely and perform the user study on.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Four of the 31 participants did not complete all 8 of their assigned tasks.

    “Note that 4 of the 31 participants did not complete all 8 of their assigned tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    For the four participants who did not complete all assigned tasks, data from completed tasks were included and unfinished tasks were not considered.

    “We include their data from the tasks they completed and do not consider the tasks they did not finish.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Study design and conductnone given

    Our tasks, despite their diversity and, we believe, representativeness of real-world Python use, may not lend themselves sufficiently well to NL2Code queries and, therefore, study participants may not have sufficient opportunities to use, and benefit from, the plugin.

    “In particular, our tasks, despite their diversity and, we believe, representativeness of real-world Python use, may not lend themselves sufficiently well to NL2Code queries and, therefore, study participants may not have sufficient opportunities to use, and benefit from, the plugin.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The study tasks are representative of real-world Python use.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    Our results suggest that some queries may be better answered through code retrieval techniques, and others through code generation.

    “Our results suggest that some queries may be better answered through code retrieval techniques, and others through code generation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Our quantitative task performance results for the current version of this NL2Code plugin, while negative, do not imply that future, better performing such systems will also not be markedly useful in practice; the qualitative data from our study participants already suggests otherwise, as does quantitative data from prior research on the usefulness of in-IDE code search plugins [92].

    “Our quantitative task performance results for the current version of this NL2Code plugin, while negative, do not imply that future, better performing such systems will also not be markedly useful in practice; the qualitative data from our our study participants already suggests otherwise, as does quantitative data from prior research on the usefulness of in-IDE code search plugins [92].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [92] 92. Luca Ponzanelli, G. Bavota, M. D. Penta, R. Oliveto, and M. Lanza. 2014. Mining Stack Overflow to turn the IDE into a self-confident programming prompter. In International Conference on Mining Software Repositories (MSR). source

  • May combine two claims.

    Premisenone given

    The drawback of retrieval relevance annotation is that the cost is high and often requires experts in the specific area.

    “The drawback of this evaluation method is that the cost of retrieval relevance annotation is high and often requires experts in the specific area.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Our study population (31 participants) may not be large enough for us to detect effects with small sizes, should they exist.

    “Moreover, our study population (31 participants) may not be large enough for us to detect effects with small sizes, should they exist.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Eight participants who did not use the plugin at all were excluded from the final sample of 31 participants.

    “We discovered this while analyzing the data collected from the study and filtered out 8 participants that did not use the plugin at all.They do not count towards the final sample of 31 participants we analyze data from, even though they completed tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The plugin handles various data collection processes inside the IDE.

    “The plugin also handles various data collection processes inside the IDE.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Three of the retrieved snippets cannot be parsed and were omitted.

    “Three of the retrieved snippets cannot be parsed and thus are omitted.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    If retrieval results were placed first, users would likely rarely scroll past them to view generation results due to screen real-estate issues.

    “If we put the retrieval results first, then it is likely that the users would rarely scroll past the retrieval results and view the generation results due to issues of screen real-estate.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The study measured code complexity using two standard proxies: source lines of code (SLOC) and McCabe's cyclomatic complexity (CC).

    “We measure two standard proxies for code complexity of the Python programs produced by our study participants in each of their assigned tasks, i.e., the number of source lines of code (SLOC) and McCabe's cyclomatic complexity (CC), a measure of the number of linearly independent paths through a program's source code [75]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [75] 75. T. McCabe. 1976. A complexity measure. IEEE Trans. Softw. Eng. SE-2 (1976), 308-320. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The man-in-the-middle proxy infrastructure enables interception and data collection of both HTTP and secured HTTPS requests.

    “This infrastructure enables interception and data collection of both HTTP and secured HTTPS requests.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The deployed model's queries are fast enough for the plugin to be usable in real time.

    “We deployed the model on an internal research server and exposed a HTTP API that the plugin can access; queries are fast enough for the plugin to be usable in real time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    With the keylogger, the study can collect data outside the IDE about the users' activities.

    “With the keylogger, we can collect data outside the IDE about the users' activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The user appears to have issued the query ahead of setting up the rest of the context.

    “the user appears to have issued the query ahead of setting up the rest of the context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Examining alternate orderings was not feasible within the scope of the current study.

    “although examining alternate orderings was not feasible within the scope of the current study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The keylogger logs all the user's keystrokes along with timestamps to the server.

    “We develop a program that runs in the background during the user study and logs all the user's keystrokes along with the timestamps to our server.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Some participants did not use the plugin even when it was available and when instructed.

    “Despite these instructions, some participants did not use the plugin even when it was available and when instructed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The analysis only considers exact substring matches.

    “Note that this only considers exact substring matches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All the user's task start, pause, resume, and submission events are logged so the completion time of each task for the user could be calculated.

    “All the user's task start, pause, resume, and submission events are logged so the completion time of each task for the user could be calculated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    With the proxy, the study can collect users' complete web browsing activities during the user study.

    “With this, we can collect users' complete web browsing activities during the user study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Alternative orderings may result in different experimental results.

    “It is important to consider that alternative orderings may result in different experimental results, although examining alternate orderings was not feasible within the scope of the current study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Some token types (ENCODING, ENDMARKER, NL) were left out of the analysis.

    “We also left out some uninteresting token types, such as ENCODING, ENDMARKER, NL.”
    Quote in the paperNot a substantial claim
80 of 80

  • The authors state this as an assumption.

    If NL2Code systems are useful in practice, developers would be able to complete programming tasks faster without compromising on output quality.

    “One would expect that if such systems are useful in practice, then developers would be able to complete programming tasks faster without compromising on output quality.”

  • The authors state this as an assumption.

    After a user makes a meaningful edit, they usually pause for a short time before the next edit.

    “The intuition behind this heuristic is that after a user makes some type of meaningful edit, such as adding or modifying an argument, variable, or function, they usually pause for a short time before the next edit.”

  • The paper relies on this without support.

    The designed tasks on average take about 15-40 minutes to complete.

    “The tasks have varying difficulties, and on average each task would take about 15-40 minutes to complete.”

  • The authors state this as an assumption.

    Study participants know the correct code snippet for a given natural language query and can recognize it among the candidates provided by the NL2Code plugin, so that the snippet source variable captures actual quality differences between code snippets produced by the generation and retrieval models.

    “There is an implicit assumption here that study participants know what the right answer (code snippet) should be given a natural language query and are able to recognize it among the candidates provided by the NL2Code plugin; therefore, we assume that the snippet source variable captures actual quality differences between code snippets produced by the generation and retrieval models, respectively.”

  • The paper relies on this without support.

    Choosing and inserting a candidate code snippet returned by the NL2Code plugin indicates that the code snippet was generally useful.

    “Choosing (and inserting into the IDE source file) one of the candidate code snippets returned by the NL2Code plugin indicates that the code snippet was generally useful.”

  • The authors state this as an assumption.

    The study tasks are representative of real-world Python use.

    “In particular, our tasks, despite their diversity and, we believe, representativeness of real-world Python use, may not lend themselves sufficiently well to NL2Code queries and, therefore, study participants may not have sufficient opportunities to use, and benefit from, the plugin.”

  • The paper relies on this without support.

    The NL2Code plugin's underlying code generation model has state-of-the-art BLEU-score performance on a benchmark dataset.

    “Our NL2Code plugin and its main underlying code generation technology, despite state-of-the-art (BLEU-score) performance on a benchmark dataset, is not developed enough to be markedly useful in practice just yet.”

  • The paper relies on this without support.

    The two metrics (SLOC and CC) are only crude approximations of the complex, multifaceted concept of code quality.

    “Among the possible explanations for why we do not find supporting evidence for the”

  • The paper relies on this without support.

    Natural language can be a useful medium to turn ideas into code, even for experienced programmers.

    “In contemporary software development, which involves countless and constantly changing programming languages and APIs, natural language can be a useful medium to turn ideas into code, even for experienced programmers.”

  • The paper relies on this without support.

    Usage log data is much less readily available than automated metrics such as BLEU.

    “However, compared to automated metrics such as BLEU, such data is much less readily available.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Chaining increased users' self-awareness of their goals: the ability to decompose tasks led some participants to create more generalizable outputs better suited to their broader goals.

    “Chaining also increased users' self-awareness of their goals: the ability to decompose tasks led some participants to create more generalizable outputs better suited to their broader goals [203].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [203]

    The Results section reports that some participants suggested an interactive plugin that could help decompose complex functionality into smaller steps. It does not report chaining increasing users’ self-awareness or producing more generalizable outputs suited to broader goals; searches for those terms found no such finding in our copy. Page numbers were not available in the indexed passages. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Recent works exploring developer-AI interaction have highlighted the need for mechanisms to facilitate verification of AI-generated code.

    “In fact, several recent works exploring developer-AI interaction have highlighted the need for mechanisms to facilitate verification of AI-generated code[14], [9], such as those that allow users to use tests that disambiguate between the different code suggestions [15].”

    From LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation (2024) · cites this paper as [9]

Cites

Flagged first, then by citations

14 of the 171 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. Sarah E. Chasins, Maria Mueller, and Rastislav Bodík. 2018. Rousillon: Scraping distributed hierarchical web data. In 31st Annual ACM Symposium on User Interface Software and Technology (UIST).
  2. Mary Beth Kery, Amber Horvath, and B. Myers. 2017. Variolite: Supporting exploratory programming by data scientists. CHI Conference on Human Factors in Computing Systems (CHI).
  3. A. Ko and B. Myers. 2008. Debugging reinvented. In ACM/IEEE 30th International Conference on Software Engineering (ICSE). 301–310.
  4. Nate Kushman and R. Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL).
  5. Toby Jia-Jun Li, Amos Azaria, and B. Myers. 2017. SUGILITE: Creating multimodal smartphone automation by demonstration. In CHI Conference on Human Factors in Computing Systems (CHI).
  6. 10.5555/1394399 · DOI
  7. 10.1145/2896587 · DOI · Paper page
  8. Mukund Raghothaman, Y. Wei, and Y. Hamadi. 2016. SWIM: Synthesizing what I mean—Code search and idiomatic snippet synthesis. In IEEE/ACM 38th International Conference on Software Engineering (ICSE). 357–367.
  9. 10.5555/2832249.2832359 · DOI
  10. 10.5555/1324786 · DOI
  11. Ziyu Yao, Daniel S. Weld, W. Chen, and Huan Sun. 2018. StaQC: A systematically mined question-code dataset from stack overflow. In World Wide Web Conference (WWW).

Cited by

Sorted by citations

Related

Authors
Xu, F. F., Vasilescu, B., & Neubig, G.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
ACM Transactions on Software Engineering and Methodology
Publisher
Association for Computing Machinery (ACM)
Published
2022-03-04
Volume
31
Issue
2
Pages
1-47
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
DOI
10.1145/3487569
OpenAlex
W3123221944
Semantic Scholar
4a160efbe80c38cd5eb2f92c7c095b49b113397d
Cited by
154
References
171
Citations by year
2026: 15 · 2025: 44 · 2024: 54 · 2023: 24 · 2022: 9 · 2021: 8
FWCI
32.36
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
NSF (1815287)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
natural language to code generation, code search, developer productivity, developer experience, developer workflows, code quality, program correctness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review