How Beginning Programmers and Code LLMs (Mis)read Each Other
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 2×
“We adapted validated scales from previous work to understand student perceptions of the usability, trustworthiness, and friendliness of Charlie [7, 20, 51, 99] and the mental workload of the task [36].”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Beginning students are particularly likely to err in this way: they may struggle to understand generated code, and their lack of confidence in their own abilities may make them trust the automated system over their own judgment (an example of automation bias [18, 30, 32, 91]).”
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Research works printed but not linked: 55
- Andrea Agostinelli Timo I. Denk Zalán Borsos Jesse Engel Mauro Verzetti Antoine Caillon Qingqing Huang Aren Jansen Adam Roberts Marco Tagliasacchi Matt Sharifi Neil Zeghidour and Christian Frank. 2023. MusicLM: Generating Music From Text. http://arxiv.org/abs/2301.11325 · arXiv
- Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. http://arxiv.org/abs/2010.01717 arXiv:2010.01717 [cs]. · arXiv
- Hannah McLean Babe Sydney Nguyen Yangtian Zi Arjun Guha Molly Q. Feldman and Carolyn Jane Anderson. 2023. StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code. http://arxiv.org/abs/2306.04556 · arXiv
- 10.1145/800177.810072 · DOI
- Patrick Bareiß Beatriz Souza Marcelo d’Amorim and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot Pre-Trained Language Models on Code. http://arxiv.org/abs/2206.01335 arXiv:2206.01335 [cs]. · arXiv
- 10.18637/jss.v067.i01 · DOI · Paper page
- 10.1145/3624062.3624088 · DOI
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. http://arxiv.org/abs/2107.03374 arXiv:2107.03374 [cs]. · arXiv
- 10.1145/2538862.2538911 · DOI
- Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators. http://arxiv.org/abs/2307.16364 arXiv:2307.16364 [cs]. · arXiv
- Stefania Druga and Amy J Ko. 2021. How do children’s perceptions of machine intelligence change when training and coding smart programs?. In Interaction Design and Children. ACM Athens Greece 49–61. https://doi.org/10.1145/3459990.3460712 · DOI · Paper page
- 10.1145/971300.971312 · DOI · Paper page
- 10.1145/3173574.3173838 · DOI
- Kasra Ferdowsi Ruanqianqian Huang Michael B. James Nadia Polikarpova and Sorin Lerner. 2023. Live Exploration of AI-Generated Programs. http://arxiv.org/abs/2306.09541 arXiv:2306.09541 [cs]. · arXiv
- 10.1145/1089786.1089797 · DOI
- Matthew Flatt, Matthias Felleisen, Robert Bruce Findler, and Shriram Krishnamurthi. 2001. How To Design Programs. MIT Press.
- 10.1145/3408877.3432539 · DOI
- Daniel Fried Armen Aghajanyan Jessy Lin Sida Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. http://arxiv.org/abs/2204.05999 arXiv:2204.05999 [cs]. · arXiv
- Marwa Gadala. 2017. Automation bias: exploring causal mechanisms and potential mitigation strategies. https://api.semanticscholar.org/CorpusID:41123263
- Chuqin Geng Haolin Ye Yixuan Li Tianyu Han Brigitte Pientka and Xujie Si. 2022. Novice Type Error Diagnosis with Natural Language Models. http://arxiv.org/abs/2210.03682 arXiv:2210.03682 [cs]. · DOI
- 10.1145/3372782.3406273 · DOI
- Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, and others. 2017. Program synthesis. Foundations and Trends® in Programming Languages 4, 1-2 (2017), 1–119. Publisher: Now Publishers, Inc..
- 10.1145/3051457.3051467 · DOI · Paper page
- 10.1145/800233.807050 · DOI
- 10.1145/367415.367422 · DOI · Paper page
- Kenneth Holstein and Shayan Doroudi. 2021. Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?http://arxiv.org/abs/2104.12920 arXiv:2104.12920 [cs]. · arXiv
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH
- 10.1145/3411764.3445424 · DOI · Paper page
- Daphne Ippolito Ann Yuan Andy Coenen and Sehmon Burnam. 2022. Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers. http://arxiv.org/abs/2211.05030 arXiv:2211.05030 [cs]. · arXiv
- 10.1109/TSE.2010.62 · DOI · Paper page
- 10.1109/VLHCC.2004.47 · DOI · Paper page
- 10.31234/osf.io/nfc45 · DOI · Paper page
- 10.1609/aimag.v30i4.2262 · DOI · Paper page
- 10.1145/3587102.3588785 · DOI · Paper page
- Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov Chenghao Mou Marc Marone Christopher Akiki Jia Li Jenny Chim Qian Liu Evgenii Zheltonozhskii Terry Yue Zhuo Thomas Wang Olivier Dehaene Mishig Davaadorj Joel Lamy-Poirier João Monteiro Oleh Shliazhko Nicolas Gontier Nicholas Meade Armel Zebaze Ming-Ho Yee Logesh Kumar Umapathi Jian Zhu Benjamin Lipkin Muhtasham Oblokulov Zhiruo Wang Rudra Murthy Jason Stillerman Siva Sankalp Patel Dmitry Abulkhanov Marco Zocca Manan Dey Zhihan Zhang Nour Fahmy Urvashi Bhattacharyya Wenhao Yu Swayam Singh Sasha Luccioni Paulo Villegas Maxim Kunakov Fedor Zhdanov Manuel Romero Tony Lee Nadav Timor Jennifer Ding Claire Schlesinger Hailey Schoelkopf Jan Ebert Tri Dao Mayank Mishra Alex Gu Jennifer Robinson Carolyn Jane Anderson Brendan Dolan-Gavitt Danish Contractor Siva Reddy Daniel Fried Dzmitry Bahdanau Yacine Jernite Carlos Muñoz Ferrandis Sean Hughes Thomas Wolf Arjun Guha Leandro von Werra and Harm de Vries. 2023. StarCoder: may the source be with you!http://arxiv.org/abs/2305.06161 arXiv:2305.06161 [cs]. · arXiv
- Vivian Liu, Tao Long, Nathan Raw, and Lydia Chilton. 2023. Generative Disco: Text-to-Video Generation for Music Visualization. http://arxiv.org/abs/2304.08551 · arXiv
- 10.1147/sj.202.0184 · DOI · Paper page
- 10.1016/0167-6423(95)96871-J · DOI · Paper page
- Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. http://arxiv.org/abs/2302.06590 arXiv:2302.06590 [cs]. · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. 2019. Language models are unsupervised multitask learners. OpenAI 1, 8 (2019), 9. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- 10.1145/3290605.3300774 · DOI
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 8821–8831. https://proceedings.mlr.press/v139/ramesh21a.html ISSN: 2640-3498.
- Hemilis Joyse Barbosa Rocha, Patrícia Cabral De Azevedo Restelli Tedesco, and Evandro De Barros Costa. 2023. On the use of feedback in learning computer programming by novices: a systematic literature mapping. Informatics in Education 22, 2 (2023), 209. Publisher: Institute of Mathematics and Informatics.
- 10.1145/3581641.3584037 · DOI · Paper page
- Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. http://arxiv.org/abs/2308.12950 arXiv:2308.12950 [cs]. · arXiv
- 10.1145/365230.365274 · DOI
- Max Schäfer Sarah Nadi Aryaz Eghbali and Frank Tip. 2023. Adaptive Test Generation Using a Large Language Model. https://doi.org/10.48550/arXiv.2302.06527 Issue: arXiv:2302.06527 _eprint: 2302.06527. · DOI
- David E Shaw William R Swartout and C Cordell Green. 1975. Inferring LISP Programs From Examples. In IJCAI Vol. 75. 260–267.
- 10.1145/2491956.2462195 · DOI · Paper page
- 10.1145/3338283 · DOI · Paper page
- 10.1109/VLHCC.2017.8103457 · DOI
- 10.1109/ICSE-SEIP58684.2023.00022 · DOI · Paper page
- 10.1145/3569897 · DOI · Paper page
- 10.1145/3570220 · DOI · Paper page
- Chunqiu Steven Xia Matteo Paltenghi Jia Le Tian Michael Pradel and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. http://arxiv.org/abs/2308.04748 arXiv:2308.04748 [cs]. · arXiv
Cited by
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
How much does AI impact development speed? An enterprise-based randomized controlled trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2024) · arXiv · doi:10.48550/arxiv.2410.12944 · Cited by 20
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“Recent work has explored the potential of these tools to assist students and novice programmers [8], [23], [24], as well as program-mers’ perceptions of [25], [26], [27] and trust in [28], [29], [30] AI tools.”
How Scientists Use Large Language Models to Program
O'Brien, G. (2025) · arXiv · doi:10.48550/arxiv.2502.17348 · Cited by 14
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Metadata
- Authors
- Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-26
- OpenAlex
- W4391417542
- Cited by
- 66
- References
- 50
- FWCI
- 29.93
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- Funders
- National Science Foundation (SES-2326173, SES-2326174, SES-2326175)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Beginning Programmers and Code LLMs (Mis)read Each Other. https://app.cochat.ai/papers/10.1145/3613904.3642706
- The paper
- Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024). How Beginning Programmers and Code LLMs (Mis)read Each Other. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642706
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.