“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbot
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Students submitted more Codegen prompts than Support prompts, with a ratio of 62% to 38%.
“Overall, we found that students submitted more Codegen prompts than Support , with a ratio of 62% to 38% 1 , confirming observations by Mailach et al. [2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Mailach et al. [2025]
This supports prior findings that some students do not understand what information precisely needs to be shared with the chatbot.
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Mailach et al. [2025]
The effect of difficulty reviewing ChatGPT output is expected to be exacerbated if students depend on ChatGPT for knowledge and conceptual questions.
“We expect this effect to be exacerbated if students also depend on ChatGPT for knowledge and conceptual questions, which has been found to be negatively associated with performance in recent work [Mailach et al. 2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Mailach et al. 2025
The propensity of learners to use GenAI outputs varies significantly by exercise topic.
“The propensity of learners to use GenAI outputs has been shown to vary significantly by exercise topic [Kazemitabaar et al. 2023; Mailach et al. 2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Mailach et al. 2025
Cites
Interacting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841 · see citing works
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · see citing works
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · see citing works
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 345 · cited 1× · see citing works
“Students, by contrast, did not experience significant benefits from using the LLM-based tool (Nam et al. 2024).”
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
Ross, S., Martinez, F., Houde, S., Muller, M. J., & Weisz, J. D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584037 · Cited by 272 · see citing works
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259 · see citing works
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · cited 2× · see citing works
“Closest to our work are studies conducted by Prather and others, who observe how students interact with LLMs in a CS1 course (Prather et al. 2023), and Kazemitabaar and others, who extract interaction patterns from students using Codex in a self-paced learning environment (Kazemitabaar et al. 2023).”
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210 · see citing works
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203 · see citing works
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · see citing works
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · see citing works
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · see citing works
Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185 · see citing works
Novice Programmers and Introductory Programming
Robins, A. V. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.013 · Cited by 183 · see citing works
Measuring and modeling programming experience
Siegmund, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9286-4 · Cited by 141 · see citing works
Self-citationKazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 132 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114 · see citing works
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 2× · see citing works
“For instance, studies show that the use of such tools may increase security risks (Perry et al. 2023) or could even degrade developers’ productivity despite the hype around them (Mozannar et al. 2023).”
Ouh, E. L., Gan, B. K. S., Shim, K. J., & Wlodkowski, S. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588794 · Cited by 101 · see citing works
A Pilot Study Integrating an AI-driven Chatbot in an Introductory Programming Course
Verleger, M., & Pembridge, J. (2018) · 2018 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8659282 · Cited by 86 · see citing works
AI-Tutoring in Software Engineering Education
Frankford, E., Sauerwein, C., Bassner, P., Krusche, S., & Breu, R. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640061 · Cited by 78 · see citing works
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69 · see citing works
How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639201 · Cited by 68 · see citing works
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59 · see citing works
The Implications of Large Language Models for CS Teachers and Students
MacNeil, S., Kim, J., Leinonen, J., Denny, P. C., Bernstein, S., Becker, B. A., Michel Wermelinger, B., Hellas, A., Tran, A., Sarsa, S., et al. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545947.3573358 · Cited by 37 · see citing works
Chat Overflow: Artificially Intelligent Models for Computing Education - renAIssance or apocAIypse?
Denny, P. C., Becker, B. A., Leinonen, J., & Prather, J. E. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588773 · Cited by 33 · see citing works
Prather, J. E., Denny, P. C., Leinonen, J., Becker, B. A., Al-Bluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594206 · Cited by 26 · see citing works
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 2× · see citing works
“This observation is included in a grounded theory developed by Barke and others, and is called exploration mode, in which the assistant is used primarily to find the next steps in the implementation [1].”
BackgroundMethodDo Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 4×
“For instance, studies show that the use of such tools may increase security risks (Perry et al. 2023) or could even degrade developers’ productivity despite the hype around them (Mozannar et al. 2023).”
Influential"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv · cited 2×
“Closest to our work are studies conducted by Prather and others, who observe how students interact with LLMs in a CS1 course (Prather et al. 2023), and Kazemitabaar and others, who extract interaction patterns from students using Codex in a self-paced learning environment (Kazemitabaar et al. 2023).”
Flagged first, then by citations
Cited by
Sorted by citations
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45 · see citing works
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
Brown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 8 · see citing works
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12 · see citing works
Yang, S., Chen, M., & Schneider, B. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770009 · Cited by 4 · see citing works
Shang, S., Yi, T., & You, X. (2025) · Internet Research · doi:10.1108/intr-10-2023-0853 · Cited by 3 · see citing works
Qi, J., Zhao, H., Xiang, R., Zhang, Y., Zhang, X., Wang, C., Jian, L., & Ji, F. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2601067 · Cited by 7 · see citing works
Uncovering Behavioral Patterns in Student–LLM Conversations during Code Refactoring Tasks
de Oliveira, E. C., Keuning, H., & Jeuring, J. T. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770000 · Cited by 2 · see citing works
Hasan, M. T., Jin, H. H., Mahmood, M., Hasan, M. T., Sarkar, J. B., Salleh, N. M., & Chowdhury, A. Y. (2025) · Multidisciplinary Reviews · doi:10.31893/multirev.2025338 · Cited by 3 · see citing works
Concepts for Teaching Software Development in the Age of AI-Tools
Böttcher, A., Thurner, V., & Zönnchen, B. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016505 · Cited by 2 · see citing works
Unpacking Vibe Coding: Help-Seeking Processes in Student-AI Interactions While Programming
Rinja, D., de Oliveira, E. A., López‐Pernas, S., Saqr, M., Specht, M. M., & Misiejuk, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29763-1_35 · Cited by 4 · see citing works
The Impact of Artificial Intelligence on Programmer Jobs: Threat or Opportunity?
Hananto, A., Hasibuan, N. S., Olivia, S., Tukino, T., & Novalia, E. (2025) · JUSIFO (JURNAL SISTEM INFORMASI) · doi:10.19109/jusifo.v11i1.25082
Yuan, Z., Jia, C., Lan, M., Zhao, L., Chen, Z., Yang, M., Liu, X., Ni, N., & Wu, S. (2026) · PLoS ONE · doi:10.1371/journal.pone.0345117
He, Z., Pang, P. C., Siu, K.-M., & Ou, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29763-1_4
Camilo, H. T., & Hauck, J. C. R. (2026) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2026.20394
- Authors
- Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J.
- Affiliations
- Center for Scalable Data Analytics and Artificial Intelligence (DE)
- Chemnitz University of Technology (DE)
- Corresponding author
- Mailach, Alina
- Venue
- Empirical Software Engineering
- Published
- 2024-11-30
- Volume
- 30
- Issue
- 1
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W4404874258
- Cited by
- 13
- References
- 31
- Citations by year
- 2026: 5 · 2025: 8
- FWCI
- 4.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Empirical Software Engineering (publishedVersion)
- Funders
- Bundesministerium für Bildung und Forschung (ScaDS.AI)
- Deutsche Forschungsgemeinschaft (SI 2171/2-2)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- chatbots, large language models, code generation, prompt engineering, code quality, debugging, software testing, CS2 course, programming exercises
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
