CoChat
Sign Up
PreprintCited by 47Software Engineering ResearchTeaching and Learning ProgrammingSpreadsheets and End-User Computing

What is it like to program with artificial intelligence?

Open access
Sarkar, Advait, Gordon, Andrew D., Negreanu, Carina, Poelitz, Christian, Ragavan, Sruti Srinivasa, Zorn, Ben·2022·arXiv
DOI 10.48550/arxiv.2208.06213arXiv 2208.06213
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models, such as OpenAI's codex and Deepmind's AlphaCode, can generate code to solve a variety of problems expressed in natural language. This technology has already been commercialised in at least one widely-used programming editor extension: GitHub Copilot. In this paper, we explore how programming with large language models (LLM-assisted programming) is similar to, and differs from, prior conceptualisations of programmer assistance. We draw upon publicly available experience reports of LLM-assisted programming, as well as prior usability and design studies. We find that while LLM-assisted programming shares some properties of compilation, pair programming, and programming via search and reuse, there are fundamental differences both in the technical possibilities as well as the practical experience. Thus, LLM-assisted programming ought to be viewed as a new way of programming with its own distinct properties and challenges. Finally, we draw upon observations from a user study in which non-expert end user programmers use LLM-assisted tools for solving data tasks in spreadsheets. We discuss the issues that might arise, and open research challenges, in applying large language models to end-user programming, particularly with users who have little or no programming expertise.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.06213

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work has shown that parsing and debugging unfamiliar code is a challenge for professional developers.

    “Prior work has shown that this is a challenge for professional developers [Sarkar et al. 2022; Vaithilingam et al. 2022].”

    From How Do Programming Students Use Generative AI? (2025) · cites this paper as Sarkar et al. 2022

    Section 7.2 (pp. 11–12) describes the burden of inspecting AI-generated code and debugging subtle errors that differ from those developers normally encounter. This supports the challenge, but the paper’s discussion is specifically about AI-generated code; it does not establish the claim for unfamiliar code generally or explicitly identify the developers as professionals. (LLM, conf=0.81)

    Experimental — accuracy not yet measured

  • A misalignment between users' trust and AI ability could lead to overreliance and misuse of AI systems, resulting in undesirable consequences.

    “A misalignment between the trust of users and the ability of AI could lead to user overreliance and misuse of AI systems, resulting in undesirable consequences [ 21 , 43 , 51 ].”

    From "It would work for me too": How Online Communities Shape Software Developers' Trust in AI-Powered Code Generation Tools (2024) · cites this paper as [51]

    Section 5 (p. 7) says users may overestimate the correctness of AI-generated code, leading to over-reliance and potentially harmful outcomes such as insecure code. It discusses code-generating AI, however, rather than misuse of AI systems generally; it also uses “misalignment” to mean a mismatch between a prompt and the user’s intent, not between trust and ability. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • GenAI systems present usability challenges around prompting, evaluating and relying on outputs, and optimizing workflows.

    “Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows.”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [154]

    The paper identifies difficulty writing effective prompts (Section 7.1, p. 10), evaluating generated code (Section 5, p. 7), and avoiding over-reliance on it (Section 5, p. 7). These findings concern AI-assisted programming; the passages do not establish the same challenges for GenAI systems generally, and the discussion of workflows (Section 7.3, p. 13) describes their use rather than a challenge of optimizing them. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The generality of GenAI poses a higher-level metacognitive demand: the challenge of knowing whether and how to incorporate GenAI into workflows.

    “Alongside the local interactions with GenAI systems, the generality of GenAI poses another, higherlevel metacognitive demand: the challenge of knowing whether and how to incorporate GenAI into workflows-i.e., one's 'automation strategy' (see also [154]).”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [154]

    Section 7.3 (p. 13) reports that some programmers developed AI-dependent workflows for particular tasks, and Section 8.2 (p. 15) describes the wide range of tasks a prompt can specify. These passages support the idea that programmers make choices about how to use AI assistance, but they do not establish the broader claim that GenAI’s generality creates a higher-level metacognitive demand to decide whether and how to incorporate it into workflows. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Sarkar [154] describes this change as a shift from production to 'critical integration', where the output of AI systems will need to be integrated into a wider workflow involving human action, a process requiring critical evaluation of outputs.

    “Sarkar [154] describes this change as a shift from production to 'critical integration', where”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [154]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 47

Research works printed but not linked: 77

  1. Allamanis, M., Barr, E. T., Devanbu, P. T., & Sutton, C. (2018). A survey of machine learning for big code and naturalness. ACM Comput. Surv. , 51 (4), 81:1-81:37. Retrieved from https://doi .org/10.1145/3212695 doi: 10.1145/3212695 · DOI · Paper page
  2. Allamanis, M., & Brockschmidt, M. (2017). Smartpaste: Learning to adapt source code. arXiv preprint arXiv:1705.07867 . · arXiv
  3. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., . . . Sutton, C. (2021). Program synthesis with large language models. arXiv. Retrieved from https://arxiv.org/abs/2108 .07732 doi: 10.48550/ARXIV.2108.07732 · DOI · Paper page
  4. Barik, T., Ford, D., Murphy-Hill, E., & Parnin, C. (2018). How should compilers explain problems to developers? In Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering (pp. 633-643).
  5. Barik, T., Johnson, B., & Murphy-Hill, E. (2015). I heart hacker news: expanding qualitative research findings by analyzing social news websites. In Proceedings of the 2015 10th joint meeting on foundations of software engineering (pp. 882-885).
  6. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In M. C. Elish, W. Isaac, & R. S. Zemel (Eds.), Facct '21: 2021 ACM conference on fairness, accountability, and transparency, virtual event / toronto, canada, march 3-10, 2021 (pp. 610-623). ACM. Retrieved from https://doi.org/10.1145/ 3442188.3445922 doi: 10.1145/3442188.3445922 · DOI · Paper page
  7. Bergström, I., & Blackwell, A. F. (2016). The practices of programming. In 2016 ieee symposium on visual languages and human-centric computing (vl/hcc) (pp. 190-198).
  8. Blackwell, A. F. (2002a). First steps in programming: A rationale for attention investment models. In Proceedings ieee 2002 symposia on human centric computing languages and environments (pp. 2-10).
  9. Bødker, S. (2015). Third-wave hci, 10 years later - participation and sharing. Interactions , 22 (5), 24-31. Retrieved from https://doi.org/10.1145/2804405 doi: 10.1145/2804405 · DOI
  10. Cao, J., Fleming, S. D., Burnett, M., & Scaffidi, C. (2015). Idea garden: Situated support for problem solving by end-user programmers. Interacting with Computers , 27 (6), 640-660.
  11. Chalhoub, G., & Sarkar, A. (2022). 'It's Freedom to Put Things Where My Mind Wants': Understanding and Improving the User Experience of Structuring Data in Spreadsheets. In CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3491102.3501833 doi: 10.1145/3491102 .3501833 · DOI
  12. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., . . . Zaremba, W. (2021). Evaluating large language models trained on code. CoRR , abs/2107.03374 . Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
  13. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., . . . Zaremba, W. (2021). Evaluating large language models trained on code. ArXiv , abs/2107.03374 . · arXiv
  14. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., . . . Fiedel, N. (2022). Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311 . · arXiv
  15. Colmerauer, A., & Roussel, P. (1996). The birth of prolog. In History of programming languages-ii (pp. 331-367).
  16. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171-4186). Minneapolis, Minnesota: Association for Computational Linguistics. Retrieved from https://aclanthology.org/N19-1423 doi: 10.18653/v1/N19-1423 · DOI · Paper page
  17. Green, T., & Blackwell, A. (1998). Cognitive dimensions of information artefacts: a tutorial. In Bcs hci conference (Vol. 98, pp. 1-75).
  18. Green, T. R., & Petre, M. (1992). When visual programs are harder to read than textual programs. In Human-computer interaction: Tasks and organisation, proceedings of ecce-6 (6th european conference on cognitive ergonomics). gc van der veer, mj tauber, s. bagnarola and m. antavolits. rome, cud (pp. 167-180).
  19. Gulwani, S. (2011). Automating string processing in spreadsheets using input-output examples. In T. Ball & M. Sagiv (Eds.), Proceedings of the 38th ACM SIGPLAN-SIGACT symposium on principles of programming languages, POPL 2011, austin, tx, usa, january 26-28, 2011 (pp. 317-330). ACM. Retrieved from https://doi.org/10.1145/1926385.1926423 doi: 10.1145/1926385 .1926423 · DOI
  20. Hannay, J. E., Dybå, T., Arisholm, E., & Sjøberg, D. I. (2009). The effectiveness of pair programming: A meta-analysis. Information and software technology , 51 (7), 1110-1122.
  21. Henley, A. Z., & Fleming, S. D. (2014). The patchworks code editor: Toward faster navigation with less code arranging and fewer navigation mistakes. In Proceedings of the sigchi conference on human factors in computing systems (pp. 2511-2520).
  22. Hermans, F., Pinzger, M., & van Deursen, A. (2015). Detecting and refactoring code smells in spreadsheet formulas. Empirical Software Engineering , 20 (2), 549-575.
  23. Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. T. (2016). On the naturalness of software. Commun. ACM , 59 (5), 122-131. Retrieved from https://doi.org/10.1145/2902362 doi: 10.1145/2902362 · DOI · Paper page
  24. Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. T. (2012). On the naturalness of software. In M. Glinz, G. C. Murphy, & M. Pezzè (Eds.), 34th international conference on software engineering, ICSE 2012, june 2-9, 2012, zurich, switzerland (pp. 837-847). IEEE Computer Society. Retrieved from https://doi.org/10.1109/ICSE.2012.6227135 doi: 10.1109/ICSE.2012.6227135 · DOI · Paper page
  25. Hoare, C. A. R. (1969). An axiomatic basis for computer programming. Commun. ACM , 12 (10), 576580. Retrieved from https://doi.org/10.1145/363235.363259 doi: 10.1145/363235 .363259 · DOI
  26. Hochreiter, S., & Schmidhuber, J. (1997, nov). Long short-term memory. Neural Comput. , 9 (8), 1735-1780. Retrieved from https://doi.org/10.1162/neco.1997.9.8.1735 doi: 10 .1162/neco.1997.9.8.1735 · DOI · Paper page
  27. Horvitz, E. (1999). Principles of mixed-initiative user interfaces. In Proceedings of the sigchi conference on human factors in computing systems (pp. 159-166).
  28. Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct manipulation interfaces. Hum. Comput. Interact. , 1 (4), 311-338. Retrieved from https://doi.org/10.1207/s15327051hci0104 \_2 doi: 10.1207/s15327051hci0104\_2 · DOI
  29. Imai, S. (2022). Is github copilot a substitute for human pair-programming? an empirical study. In 2022 ieee/acm 44th international conference on software engineering: Companion proceedings (icsecompanion) (pp. 319-321).
  30. Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., . . . Terry, M. (2022). Discovering the syntax and strategies of natural language programming with generative language models. In Chi conference on human factors in computing systems (pp. 1-19).
  31. Kery, M. B., & Myers, B. A. (2017). Exploring exploratory programming. In 2017 ieee symposium on visual languages and human-centric computing (vl/hcc) (pp. 25-29).
  32. Ko, A. J., & Myers, B. A. (2004). Designing the whyline: a debugging interface for asking questions about program behavior. In Proceedings of the sigchi conference on human factors in computing systems (pp. 151-158).
  33. Kulesza, T., Amershi, S., Caruana, R., Fisher, D., & Charles, D. (2014). Structured labeling for facilitating concept evolution in machine learning. In Proceedings of the sigchi conference on human factors in computing systems (pp. 3075-3084).
  34. Kurlander, D., Cypher, A., & Halbert, D. C. (1993). Watch what i do: programming by demonstration . MIT press.
  35. Lau, S., Srinivasa Ragavan, S. S., Milne, K., Barik, T., & Sarkar, A. (2021). Tweakit: Supporting enduser programmers who transmogrify code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-12).
  36. Li, J., Tang, T., Zhao, W. X., & Wen, J.-R. (2021, 8). Pretrained language model for text generation: Asurvey. In Z.-H. Zhou (Ed.), Proceedings of the thirtieth international joint conference on artificial intelligence, IJCAI-21 (pp. 4492-4499). International Joint Conferences on Artificial Intelligence Organization. Retrieved from https://doi.org/10.24963/ijcai.2021/612 (Survey Track) doi: 10.24963/ijcai.2021/612 · DOI
  37. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., . . . Vinyals, O. (2022b). Competition-level code generation with alphacode. arXiv. Retrieved from https://arxiv.org/ abs/2203.07814 doi: 10.48550/ARXIV.2203.07814 · DOI · Paper page
  38. Li, Y., Choi, D. H., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., . . . Vinyals, O. (2022a). Competition-level code generation with alphacode. ArXiv , abs/2203.07814 . · arXiv
  39. Lieberman, H. (2001). Your wish is my command: Programming by example . Morgan Kaufmann.
  40. Lieberman, H., & Liu, H. (2006). Feasibility studies for programming in natural language. In End user development (pp. 459-473). Springer.
  41. Liu, S., Chen, Y., Xie, X., Siow, J. K., & Liu, Y. (2021). Retrieval-augmented generation for code summarization via hybrid GNN. In International conference on learning representations. Retrieved from https://openreview.net/forum?id=zv-typ1gPxA
  42. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., . . . Liu, S. (2021). Codexglue: A machine learning benchmark dataset for code understanding and generation. ArXiv , abs/2102.04664 . · arXiv
  43. Luger, E., & Sellen, A. (2016). "like having a really bad pa" the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 chi conference on human factors in computing systems (pp. 5286-5297).
  44. Madi, N. A. (2022). How readable is model-generated code? examining readability and visual inspection of github copilot. arXiv preprint arXiv:2208.14613 . · arXiv
  45. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Weinberger (Eds.), Advances in neural information processing systems (Vol. 26). Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper/2013/file/ 9aa42b31882ec039965f3c4923ce901b-Paper.pdf
  46. Miller, L. A. (1981). Natural language programming: Styles, strategies, and contrasts. IBM Systems Journal , 20 (2), 184-215.
  47. Mu, J., & Sarkar, A. (2019). Do we need natural language? Exploring restricted language interfaces for complex domains. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).
  48. Myers, B. A. (1992). Demonstrational interfaces: A step beyond direct manipulation. Computer , 25 (8), 61-73.
  49. Myers, B. A., & Stylos, J. (2016). Improving api usability. Communications of the ACM , 59 (6), 62-69.
  50. Nardi, B. A. (1993). A small matter of programming: perspectives on end user computing . MIT press.
  51. Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2015). Divide-and-conquer approach for multi-phase statistical migration for source code (t). 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 585-596.
  52. Pandita, R., Parnin, C., Hermans, F., & Murphy-Hill, E. (2018). No half-measures: A study of manual and tool-assisted end-user programming tasks in excel. In 2018 ieee symposium on visual languages and human-centric computing (vl/hcc) (pp. 95-103).
  53. Panko, R. R. (2008). Reducing overconfidence in spreadsheet development. arXiv preprint arXiv:0804.0941 . · arXiv
  54. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2021). Asleep at the keyboard? assessing the security of github copilot's code contributions. arXiv. Retrieved from https://arxiv.org/ abs/2108.09293 doi: 10.48550/ARXIV.2108.09293 · DOI
  55. Piccioni, M., Furia, C. A., & Meyer, B. (2013). An empirical study of api usability. In 2013 acm/ieee international symposium on empirical software engineering and measurement (pp. 5-14).
  56. Potthast, M., Hagen, M., & Stein, B. (2021). The dilemma of the direct answer. In Acm sigir forum (Vol. 54, pp. 1-12).
  57. Raychev, V., Vechev, M. T., & Krause, A. (2015). Predicting program properties from "big code". In S. K. Rajamani & D. Walker (Eds.), Proceedings of the 42nd annual ACM SIGPLAN-SIGACT symposium on principles of programming languages, POPL 2015, mumbai, india, january 15-17, 2015 (pp. 111-124). ACM. Retrieved from https://doi.org/10.1145/2676726.2677009 doi: 10.1145/2676726.2677009 · DOI
  58. Salge, C. A. D. L., & Berente, N. (2016). Pair programming vs. solo programming: What do we know after 15 years of research? In 2016 49th hawaii international conference on system sciences (hicss) (pp. 5398-5406).
  59. Sarkar, A. (2016). Interactive analytical modelling (Tech. Rep. No. UCAM-CL-TR-920). University of Cambridge, Computer Laboratory. Retrieved from https://www.cl.cam.ac.uk/ techreports/UCAM-CL-TR-920.pdf doi: 10.48456/tr-920 · DOI
  60. Sarkar, A. (2022, March). Is explainable AI a race against model complexity? In Workshop on Transparency and Explanations in Smart Systems (TeXSS), in conjunction with ACM Intelligent User Interfaces (IUI 2022) (pp. 192-199). Retrieved from http://ceur-ws.org/Vol-3124/ paper22.pdf
  61. Sarkar, A., & Gordon, A. D. (2018, September). How do people learn to use spreadsheets? (work in progress). In Proceedings of the 29th Annual Conference of the Psychology of Programming Interest Group (PPIG 2018) (pp. 28-35).
  62. Sarkar, A., Jamnik, M., Blackwell, A. F., & Spott, M. (2015). Interactive visual machine learning in spreadsheets. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (pp. 159-163).
  63. Sarkar, A., Srinivasa Ragavan, S., Williams, J., & Gordon, A. D. (2022). End-user encounters with lambda abstraction in spreadsheets: Apollo's bow or Achilles' heel? In 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
  64. Srinivasa Ragavan, S., Hou, Z., Wang, Y., Gordon, A. D., Zhang, H., & Zhang, D. (2022). Gridbook: Natural language formulas for the spreadsheet grid. In 27th international conference on intelligent user interfaces (p. 345-368). New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3490099.3511161 doi: 10.1145/3490099.3511161 · DOI · Paper page
  65. Srinivasa Ragavan, S., Kuttal, S. K., Hill, C., Sarma, A., Piorkowski, D., & Burnett, M. (2016). Foraging among an overabundance of similar variants. In Proceedings of the 2016 chi conference on human factors in computing systems (pp. 3509-3521).
  66. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Proceedings of the 27th international conference on neural information processing systems - volume 2 (p. 3104-3112). Cambridge, MA, USA: MIT Press.
  67. Tanimoto, S. L. (2013). A perspective on the evolution of live programming. In 2013 1st international workshop on live programming (live) (pp. 31-34).
  68. Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022). Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts (pp. 1-7).
  69. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st international conference on neural information processing systems (p. 6000-6010). Red Hook, NY, USA: Curran Associates Inc.
  70. Wei, J., Goyal, M., Durrett, G., & Dillig, I. (2020). Lambdanet: Probabilistic type inference using graph neural networks. ArXiv , abs/2005.02161 . · arXiv
  71. Williams, J., Negreanu, C., Gordon, A. D., & Sarkar, A. (2020). Understanding and inferring units in spreadsheets. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (pp. 1-9).
  72. Williams, L. A., & Kessler, R. R. (2000). All i really need to know about pair programming i learned in kindergarten. Communications of the ACM , 43 (5), 108-114.
  73. Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022). A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming .
  74. Xu, F. F., Vasilescu, B., & Neubig, G. (2022). In-IDE Code Generation from Natural Language: Promise and Challenges. ACM Transactions on Software Engineering and Methodology (TOSEM) , 31 (2), 1-47.
  75. Yoon, Y., & Myers, B. A. (2015). Supporting selective undo in a code editor. In 2015 ieee/acm 37th ieee international conference on software engineering (Vol. 1, pp. 223-233).
  76. Zhang, H., Jain, A., Khandelwal, G., Kaushik, C., Ge, S., & Hu, W. (2016). Bing developer assistant: improving developer productivity by recommending sample code. In Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering (pp. 956-961).
  77. Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., . . . Aftandilian, E. (2022). Productivity assessment of neural code completion. arXiv preprint arXiv:2205.06537 . · arXiv

Authors
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B.
Affiliations
  • University of Cambridge (GB)
  • University of Edinburgh (GB)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.06213
arXiv
2208.06213
OpenAlex
W4302797994
Cited by
47
References
0
Citations by year
2026: 3 · 2025: 9 · 2024: 16 · 2023: 19
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
Keywords
LLM-assisted programming, large language models, GitHub Copilot, end-user programming, non-expert programmers, spreadsheet software, code generation, pair programming, compilation, user experience

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review