How Beginning Programmers and Code LLMs (Mis)read Each Other
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 2×
“We adapted validated scales from previous work to understand student perceptions of the usability, trustworthiness, and friendliness of Charlie [7, 20, 51, 99] and the mental workload of the task [36].”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Beginning students are particularly likely to err in this way: they may struggle to understand generated code, and their lack of confidence in their own abilities may make them trust the automated system over their own judgment (an example of automation bias [18, 30, 32, 91]).”
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487 · cited 1×
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2×
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 6×
“After Codex responds, Charlie shows students the Codex-generated code and displays whether it works on the given input/output examples (Figure 3b).”
BackgroundInfluentialAn Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · cited 2×
“Our participant pool represents students who mostly completed CS1 before Code LLMs became commonplace.”
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394 · cited 1×
“5-turbo and code-davinci-002 score 48% and 46% respectively on the Hu-manEval Python programming benchmark [11, 74], the most commonly used Python benchmark for Code LLMs.”
MethodSelf-citationStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
Relationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Accountability and automation bias
SKITKA, L. J., MOSIER, K., & BURDICK, M. D. (2000) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0349 · Cited by 324 · cited 1×
“Beginning students are particularly likely to err in this way: they may struggle to understand generated code, and their lack of confidence in their own abilities may make them trust the automated system over their own judgment (an example of automation bias [18, 30, 32, 91]).”
BackgroundCodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · cited 2×
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
BackgroundMethodMirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 1×
“In programming, along with fields like digital art [79, 81, 83], creative writing [2, 45, 68], and digital music [1, 63], generative AI has reduced the technical skills that users need by allowing them to prompt a model with a natural language description of their desired output.”
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221 · cited 1×
“Beginning students are particularly likely to err in this way: they may struggle to understand generated code, and their lack of confidence in their own abilities may make them trust the automated system over their own judgment (an example of automation bias [18, 30, 32, 91]).”
BackgroundInvestigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219 · cited 1×
“This is an example of an ongoing equity concern for researchers and educators.”
BackgroundAI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Wortman Vaughan, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.8036d03b · Cited by 210
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2306.01941 · Cited by 209
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206 · cited 1×
“We discuss the logistics of implementing the study in Section 4.”
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools
Myers, B. A., Ko, A. J., LaToza, T. D., & Yoon, Y. (2016) · Computer · doi:10.1109/mc.2016.200 · Cited by 144
Repair Is Nearly Generation: Multilingual Program Repair with LLMs
Joshi, H., Cambronero Sanchez, J., Gulwani, S., Le, V., Verbruggen, G., & Radiček, I. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25642 · Cited by 126
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125
Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445645 · Cited by 120 · cited 2×
“We adapted validated scales from previous work to understand student perceptions of the usability, trustworthiness, and friendliness of Charlie [7, 20, 51, 99] and the mental workload of the task [36].”
MethodWhere to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119 · cited 1×
“Our post-study interview and survey was inspired by their exploration of participant’s perceptions of AI.”
MethodLiu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119 · cited 1×
“Liu et al. [62] build a new interaction model, grounded abstraction matching, which targets spreadsheets and data frames, constraining the generated code to support grounding.”
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94
AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 6×
“After Codex responds, Charlie shows students the Codex-generated code and displays whether it works on the given input/output examples (Figure 3b).”
BackgroundInfluentialA Hazard Analysis Framework for Code Synthesis Large Language Models
Khlaaf, H., Mishkin, P., Achiam, J., Krueger, G., & Brundage, M. (2022) · arXiv · doi:10.48550/arxiv.2207.14157 · Cited by 23
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Self-citationGenerating High-Precision Feedback for Programming Syntax Errors using Large Language Models
Phung, T., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G. (2023) · arXiv · doi:10.48550/arxiv.2302.04662 · Cited by 16
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
Research works printed but not linked: 54
- Andrea Agostinelli Timo I. Denk Zalán Borsos Jesse Engel Mauro Verzetti Antoine Caillon Qingqing Huang Aren Jansen Adam Roberts Marco Tagliasacchi Matt Sharifi Neil Zeghidour and Christian Frank. 2023. MusicLM: Generating Music From Text. http://arxiv.org/abs/2301.11325 · arXiv
- Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. http://arxiv.org/abs/2010.01717 arXiv:2010.01717 [cs]. · arXiv
- Hannah McLean Babe Sydney Nguyen Yangtian Zi Arjun Guha Molly Q. Feldman and Carolyn Jane Anderson. 2023. StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code. http://arxiv.org/abs/2306.04556 · arXiv
- 10.1145/800177.810072 · DOI
- Patrick Bareiß Beatriz Souza Marcelo d’Amorim and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot Pre-Trained Language Models on Code. http://arxiv.org/abs/2206.01335 arXiv:2206.01335 [cs]. · arXiv
- 10.18637/jss.v067.i01 · DOI · Paper page
- 10.1145/3624062.3624088 · DOI
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. http://arxiv.org/abs/2107.03374 arXiv:2107.03374 [cs]. · arXiv
- 10.1145/2538862.2538911 · DOI
- Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators. http://arxiv.org/abs/2307.16364 arXiv:2307.16364 [cs]. · arXiv
- Stefania Druga and Amy J Ko. 2021. How do children’s perceptions of machine intelligence change when training and coding smart programs?. In Interaction Design and Children. ACM Athens Greece 49–61. https://doi.org/10.1145/3459990.3460712 · DOI · Paper page
- 10.1145/971300.971312 · DOI · Paper page
- 10.1145/3173574.3173838 · DOI
- Kasra Ferdowsi Ruanqianqian Huang Michael B. James Nadia Polikarpova and Sorin Lerner. 2023. Live Exploration of AI-Generated Programs. http://arxiv.org/abs/2306.09541 arXiv:2306.09541 [cs]. · arXiv
- 10.1145/1089786.1089797 · DOI
- Matthew Flatt, Matthias Felleisen, Robert Bruce Findler, and Shriram Krishnamurthi. 2001. How To Design Programs. MIT Press.
- 10.1145/3408877.3432539 · DOI
- Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. http://arxiv.org/abs/2204.05999 arXiv:2204.05999 [cs]. · arXiv
- Marwa Gadala. 2017. Automation bias: exploring causal mechanisms and potential mitigation strategies. https://api.semanticscholar.org/CorpusID:41123263
- Chuqin Geng Haolin Ye Yixuan Li Tianyu Han Brigitte Pientka and Xujie Si. 2022. Novice Type Error Diagnosis with Natural Language Models. http://arxiv.org/abs/2210.03682 arXiv:2210.03682 [cs]. · DOI
- 10.1145/3372782.3406273 · DOI
- Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, and others. 2017. Program synthesis. Foundations and Trends® in Programming Languages 4, 1-2 (2017), 1–119. Publisher: Now Publishers, Inc..
- 10.1145/3051457.3051467 · DOI · Paper page
- 10.1145/800233.807050 · DOI
- 10.1145/367415.367422 · DOI · Paper page
- Kenneth Holstein and Shayan Doroudi. 2021. Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?http://arxiv.org/abs/2104.12920 arXiv:2104.12920 [cs]. · arXiv
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH
- 10.1145/3411764.3445424 · DOI · Paper page
- Daphne Ippolito Ann Yuan Andy Coenen and Sehmon Burnam. 2022. Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers. http://arxiv.org/abs/2211.05030 arXiv:2211.05030 [cs]. · arXiv
- 10.1109/TSE.2010.62 · DOI · Paper page
- 10.1109/VLHCC.2004.47 · DOI · Paper page
- 10.31234/osf.io/nfc45 · DOI · Paper page
- 10.1609/aimag.v30i4.2262 · DOI · Paper page
- Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim Qian Liu Evgenii Zheltonozhskii Terry Yue Zhuo Thomas Wang Olivier Dehaene Mishig Davaadorj Joel Lamy-Poirier João Monteiro Oleh Shliazhko Nicolas Gontier Nicholas Meade Armel Zebaze Ming-Ho Yee Logesh Kumar Umapathi Jian Zhu Benjamin Lipkin Muhtasham Oblokulov Zhiruo Wang Rudra Murthy Jason Stillerman Siva Sankalp Patel Dmitry Abulkhanov Marco Zocca Manan Dey Zhihan Zhang Nour Fahmy Urvashi Bhattacharyya Wenhao Yu Swayam Singh Sasha Luccioni Paulo Villegas Maxim Kunakov Fedor Zhdanov Manuel Romero Tony Lee Nadav Timor Jennifer Ding Claire Schlesinger Hailey Schoelkopf Jan Ebert Tri Dao Mayank Mishra Alex Gu Jennifer Robinson Carolyn Jane Anderson Brendan Dolan-Gavitt Danish Contractor Siva Reddy Daniel Fried Dzmitry Bahdanau Yacine Jernite Carlos Muñoz Ferrandis Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2023. StarCoder: may the source be with you!http://arxiv.org/abs/2305.06161 arXiv:2305.06161 [cs]. · arXiv
- Vivian Liu, Tao Long, Nathan Raw, and Lydia Chilton. 2023. Generative Disco: Text-to-Video Generation for Music Visualization. http://arxiv.org/abs/2304.08551 · arXiv
- 10.1147/sj.202.0184 · DOI · Paper page
- 10.1016/0167-6423(95)96871-J · DOI · Paper page
- Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. http://arxiv.org/abs/2302.06590 arXiv:2302.06590 [cs]. · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. 2019. Language models are unsupervised multitask learners. OpenAI 1, 8 (2019), 9. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- 10.1145/3290605.3300774 · DOI
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 8821–8831. https://proceedings.mlr.press/v139/ramesh21a.html ISSN: 2640-3498.
- Hemilis Joyse Barbosa Rocha, Patrícia Cabral De Azevedo Restelli Tedesco, and Evandro De Barros Costa. 2023. On the use of feedback in learning computer programming by novices: a systematic literature mapping. Informatics in Education 22, 2 (2023), 209. Publisher: Institute of Mathematics and Informatics.
- 10.1145/3581641.3584037 · DOI · Paper page
- Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. http://arxiv.org/abs/2308.12950 arXiv:2308.12950 [cs]. · arXiv
- 10.1145/365230.365274 · DOI
- Max Schäfer Sarah Nadi Aryaz Eghbali and Frank Tip. 2023. Adaptive Test Generation Using a Large Language Model. https://doi.org/10.48550/arXiv.2302.06527 Issue: arXiv:2302.06527 _eprint: 2302.06527. · DOI
- David E Shaw William R Swartout and C Cordell Green. 1975. Inferring LISP Programs From Examples. In IJCAI Vol. 75. 260–267.
- 10.1145/2491956.2462195 · DOI · Paper page
- 10.1145/3338283 · DOI · Paper page
- 10.1109/VLHCC.2017.8103457 · DOI
- 10.1109/ICSE-SEIP58684.2023.00022 · DOI · Paper page
- 10.1145/3569897 · DOI · Paper page
- 10.1145/3570220 · DOI · Paper page
- Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. http://arxiv.org/abs/2308.04748 arXiv:2308.04748 [cs]. · arXiv
Cited by
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
International AI Safety Report
Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.17805 · Cited by 16
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“Recent work has explored the potential of these tools to assist students and novice programmers [8], [23], [24], as well as program-mers’ perceptions of [25], [26], [27] and trust in [28], [29], [30] AI tools.”
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Metadata
- Authors
- Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-26
- OpenAlex
- W4391417542
- Cited by
- 66
- References
- 51
- FWCI
- 29.93
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- Funders
- National Science Foundation (SES-2326173, SES-2326174, SES-2326175)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.