The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Significant Productivity Gains through Programming with Large Language Models
Open accessAbstract
Large language models like GPT and Codex drastically alter many daily tasks, including programming, where they can rapidly generate code from natural language or informal specifications. Thus, they will change what it means to be a programmer and how programmers act during software development. This work explores how AI assistance for code generation impacts productivity. In our user study (N=24), we asked programmers to complete Python programming tasks supported by a) an auto-complete interface using GitHub Copilot, b) a conversational system using GPT-3, and c) traditionally with just the web browser. Aside from significantly increasing productivity metrics, participants displayed distinctive usage patterns and strategies, highlighting that the form of presentation and interaction affects how users engage with these systems. Our findings emphasize the benefits of AI-assisted coding and highlight the different design challenges for these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“These systems rely on the Transformer architecture [61] and resulting Large Language Models like Generative Pre-Trained (GPT) Transformers [6] or Bidirectional Encoder Representations from Transformers (BERT) systems [12].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174 · cited 1×
“While breaking down complex interaction patterns into individual interactions is a well-established pattern [8], prior work (e.g., Forsgren et al. [17] typically considers a higher number of interactions as indicative of high activity.”
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 2×
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität
Hassenzahl, M., Burmester, M., & Koller, F. (2003) · Berichte des German Chapter of the ACM · doi:10.1007/978-3-322-80058-9_19 · Cited by 955 · cited 2×
“This survey contains ten questions resulting in measures for pragmatic quality , hedonic quality , and attractiveness , which are closely related to user satisfaction [18].”
BackgroundMethodVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452 · cited 2×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · cited 1×
“One quite ubiquitous feature in many modern editors is simple code completion [34], which suggests what code the developer might type next.”
BackgroundLiu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390 · cited 1×
“We have found several evaluation efforts to assess the quality of the generated software (e.g. [2, 9, 30, 68]).”
BackgroundUsing structural context to recommend source code examples
Holmes, R., & Murphy, G. C. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062491 · Cited by 379 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
Learning from examples to improve code completion systems
Bruch, M., Monperrus, M., & Mezini, M. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595728 · Cited by 378 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 346 · cited 1×
“Additionally, LLMs can also process existing code and take over tedious tasks like debugging and bug fixing [39, 42, 54] or generating explanations, documentation, and test cases [58, 59].”
BackgroundAn Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338
The Inference of Perceived Usability From Beauty
Hassenzahl, M., & Monk, A. (2010) · Human-Computer Interaction · doi:10.1080/07370024.2010.500139 · Cited by 330 · cited 1×
“Participants completed the short AttrakDiff [16, 18, 19] after each test condition to quantify satisfaction.”
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287 · cited 2×
“In the domain of auto-complete systems, Peng et al. [40] conducted a between-groups experiment where participants had to create an HTTP server using JavaScript.”
MethodResultMetrics for assessing a software system's maintainability
Oman, P., & Hagemeister, J. (2003) · Proceedings Conference on Software Maintenance 1992 · doi:10.1109/icsm.1992.242525 · Cited by 272 · cited 1×
“Furthermore, we include the Maintainability Index [37, 38] as a metric, which combines multiple static code metrics, like Cyclomatic Complexity and Halstead Volume [37, 63].”
MethodRoss, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258 · cited 2×
“Ross et al. [51] emphasize that the different forms of AI assistants will each have different trade-offs for productivity, though.”
BackgroundCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · cited 2×
“While it is possible with more general LLMs like the family of GPT models [6], there are dedicated models specifically tailored for this use case [6, 9–11, 15, 29].”
BackgroundMethodSoftware developers' perceptions of productivity
Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635892 · Cited by 224 · cited 2×
“For this reason, we include the correctness of the code that participants created during the task, which corresponds to the number of work items (bugs, tasks) closed [31] and the absence of bugs [17].”
ChatGPT in Computer Science Curriculum Assessment: An analysis of Its Successes and Shortcomings
Qureshi, B. (2023) · International Conference e-Society, e-Learning and e-Technologies · doi:10.1145/3613944.3613946 · Cited by 218
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 4×
“Thus, the presentation and interaction are essential components to ensure that developers can use the LLMs productively in their work [4, 25, 70].”
BackgroundMethodInfluentialProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Assessing the quality of GitHub copilot’s code generation
Yetiştiren, B., Ozsoy, I., & Tüzün, E. (2022) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 193 · cited 1×
“Using the Codex system, GitHub Copilot can directly integrate into coding editors like Visual Studio Code or IDEs [67].”
MethodPyMT5: multi-mode translation of natural language and Python code with transformers
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., & Sundaresan, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.728 · Cited by 184
Pythia: AI-assisted Code Completion System
Svyatkovskiy, A., Zhao, Y., Fu, S., & Sundaresan, N. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330699 · Cited by 169 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
The SPACE of Developer Productivity: There's more to it than you think.
Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., & Butler, J. (2021) · Queue · doi:10.1145/3454122.3454124 · Cited by 164
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
How Program History Can Improve Code Completion
Robbes, R., & Lanza, M. (2008) · 2008 23rd IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2008.42 · Cited by 155 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
BackgroundSobania, D., Briesch, M., & Rothlauf, F. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528700 · Cited by 155
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
How do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C., & Inoue, K. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 150 · cited 1×
“This allowed, for instance, looking up information on Stack Overflow, a typical development behavior [1, 65].”
Natural Language to Code Generation in Interactive Data Science NotebooksPreprint
Yin, P., Li, W.-D., Xiao, K.-F., Rao, A., Wen, Y., Shi, K.-S., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09248 · Cited by 132 · cited 1×
“We have found several evaluation efforts to assess the quality of the generated software (e.g. [2, 9, 30, 68]).”
BackgroundMeasuring and predicting software productivity: A systematic map and review
Petersen, K. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.001 · Cited by 128 · cited 5×
“Afairlysimple but common way is to use the Lines of Code (LoC), which a developer creates, deletes, or modifies, as a proxy for productivity [41].”
InfluentialReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 2×
“Developers can access these models through a wide variety of modalities, from a conversational format like ChatGPT [45, 51], integrated into their familiar coding environment like GitHub’s Copilot [4, 25, 33, 70] or even with new tools, built around these models, like the Cursor Code Editor 1 .”
BackgroundMethodReading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Intelligent Code Completion with Bayesian Networks
Proksch, S., Lerch, J., & Mezini, M. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2744200 · Cited by 105 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
Improving code completion with program history
Robbes, R., & Lanza, M. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0064-x · Cited by 95 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
Agashe, R., Iyer, S., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1546 · Cited by 92 · cited 1×
“We have found several evaluation efforts to assess the quality of the generated software (e.g. [2, 9, 30, 68]).”
BackgroundTowards a Theory of Software Developer Job Satisfaction and Perceived Productivity
Storey, M.-A., Zimmermann, T., Bird, C., Czerwonka, J., Murphy, B., & Kalliamvakou, E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2944354 · Cited by 90 · cited 1×
“Beyond these code-centric metrics, Storey et al. [56] emphasizes the importance of developer satisfaction, and Meyer et al. [31] emphasize the focus on developers’ subjective perception.”
Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79
Software Maintainability Index Revisited
Welker, K. D. (2001) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · Cited by 76
What Do Developers Use the Crowd For? A Study Using Stack Overflow
Abdalkareem, R., Shihab, E., & Rilling, J. (2017) · IEEE Software · doi:10.1109/ms.2017.31 · Cited by 74 · cited 1×
“This allowed, for instance, looking up information on Stack Overflow, a typical development behavior [1, 65].”
Code and commit metrics of developer productivity: a study on team leaders perceptions
de Oliveira, E. C. C., Fernandes, E., Steinmacher, I., de Cristo, M. A. P., Conte, T. U., & Garcia, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09820-z · Cited by 62 · cited 2×
“Recently, Oliveira et al. [36] reviewed the use of additional metrics.”
A systematic review of code generation proposals from state machine specifications
Domínguez, E., Pérez, B., Rubio, Á. L., & Zapata, M. A. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.008 · Cited by 58 · cited 1×
“Frequently, these relied on a clear specification in a format like state machines [13].”
BackgroundCSCC: Simple, Efficient, Context Sensitive Code Completion
Asaduzzaman, M., Roy, C. K., Schneider, K. A., & Hou, D. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.29 · Cited by 55 · cited 1×
“…and industry have explored different mechanisms how these suggestions and code completions can be generated, from semantic analysis of the source code [3, 22, 28, 48, 49] to machine learning approaches [43, 57] Programming with Large Language Models 256:3 and different combinations [7, 47].”
BackgroundLost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52
Artificial Intelligence in Software Test Automation: A Systematic Literature Review
(2020) · doi:10.5220/0009417801810192 · Cited by 51 · cited 1×
“Additionally, LLMs can also process existing code and take over tedious tasks like debugging and bug fixing [39, 42, 54] or generating explanations, documentation, and test cases [58, 59].”
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
(2023) · doi:10.48550/arxiv.2301.08653 · Cited by 49 · cited 1×
“Additionally, LLMs can also process existing code and take over tedious tasks like debugging and bug fixing [39, 42, 54] or generating explanations, documentation, and test cases [58, 59].”
BackgroundWhat is it like to program with artificial intelligence?
Sarkar, A., Gordon, A. D., Negreanu, C., Poelitz, C., Ragavan, S. S., & Zorn, B. (2022) · arXiv · doi:10.48550/arxiv.2208.06213 · Cited by 47 · cited 2×
“These systems are pre-trained with a large volume of data, which may include natural language text, code, etc., which shapes for which tasks these systems can be used [53].”
BackgroundSOFTWARE ENGINEERING JOB PRODUCTIVITY — A SYSTEMATIC REVIEW
Hernández-López, A., Colomo‐Palacios, R., & García-Crespo, Á. (2013) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194013500125 · Cited by 35 · cited 4×
“The importance of time-relative metrics matches with the findings of a literature review by Hernández-López et al. [20] who found that Lines of Code by Time or Functions Points by Time , or Tasks Completed by Time in long term projects, are commonly used metrics.”
InfluentialAutomatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
Prenner, J. A., & Robbes, R. (2021) · arXiv · doi:10.48550/arxiv.2111.03922 · Cited by 25 · cited 1×
“Additionally, LLMs can also process existing code and take over tedious tasks like debugging and bug fixing [39, 42, 54] or generating explanations, documentation, and test cases [58, 59].”
BackgroundNatural language-guided programming
Heyman, G., Huysegems, R., Justen, P., & Cutsem, T. V. (2021) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3486607.3486749 · Cited by 24 · cited 1×
“Instead, we must also consider human factors, e.g., how productively a software developer can work with these tools [21].”
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 5×
“To assess these three levels of support, we selected three tasks considering literature in this domain [4, 32, 33, 55, 60, 66] and popular software projects (e.g. [44, 62]): Task A Text Analysis: For the text analysis task, participants had to perform text analysis of a scraped website, extracting…”
BackgroundMethodResultInfluentialJin, X., & Servant, F. (2018) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3196398.3196474 · Cited by 20 · cited 1×
“While these systems can play an important role in productive development [28], there is still limited research regarding their impact on developer behavior [26].”
BackgroundExamining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2021) · arXiv · doi:10.48550/arxiv.2112.02125 · Cited by 17 · cited 1×
“Additionally, LLMs can also process existing code and take over tedious tasks like debugging and bug fixing [39, 42, 54] or generating explanations, documentation, and test cases [58, 59].”
BackgroundFischer, P. T., Kuliga, S., Eisenberg, M., & Amin, I. (2018) · ACM International Symposium on Pervasive Displays (PerDis) · doi:10.1145/3205873.3205875 · Cited by 9 · cited 1×
“Participants completed the short AttrakDiff [16, 18, 19] after each test condition to quantify satisfaction.”
Method
Cited by
A Survey on LLM-Based Code Generation for Low-Resource and Domain-Specific Programming Languages
Joel, S., Wu, J. J., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3770084 · Cited by 204
'I'm Categorizing LLM as a Productivity Tool': Examining Ethics of LLM Use in HCI Research Practices
Kapania, S., Wang, R., Li, T. J.-J., Li, T., & Shen, H. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711000 · Cited by 79
OntoGenix: Leveraging Large Language Models for enhanced ontology engineering from datasets
Val-Calvo, M., Aranguren, M. E., Mulero-Hernández, J., Almagro-Hernández, G., Deshmukh, P., Bernabé-Díaz, J. A., Espinoza-Arias, P., Sánchez-Fernández, J. L., Mueller, J., & Fernández‐Breis, J. T. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104042 · Cited by 36
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
Weinz, M., Zannone, N., Allodi, L., & Apruzzese, G. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736195 · Cited by 44
Metadata
- Authors
- Weber, T., Brandmaier, M., Schmidt, A., & Mayer, S.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2024-06-17
- Volume
- 8
- Issue
- EICS
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4399723222
- Semantic Scholar
- 92ac5e2f527984494231459e3f7da4f61790d08f
- Cited by
- 64
- References
- 59
- Citations by year
- 2026: 37 · 2025: 24 · 2024: 3
- FWCI
- 24.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, AI-assisted programming, code generation, developer productivity, GitHub Copilot, GPT-3, conversational AI, Python programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.