The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index.
Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Open accessAbstract
Artificial Intelligence-driven Development Environments (AIDEs) offer developers revolutionary computer programming assistance. There is great potential in incorporating AIDEs into Computer Science education; however, the effects of these tools should be fully examined before doing so. Here, a within-subjects study was conducted to compare the programming performance, workload, emotion, and self-efficacy of seventeen novices coding with and without use of the GitHub Copilot AIDE under time pressure. Results showed that using the AIDE significantly increased programming efficiency and reduced effort and mental workload but did not significantly impact emotion or self-efficacy. However, participants' performance improved with more experience using the AI, and their self-efficacy followed. The results suggest that students who try AIDEs will likely be tempted to use them for time-sensitive work. There is no evidence that providing AIDEs will aid struggling students, but there is a clear need for students to practice with AI to become competent and confident using it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 3×
“C+ to B) and a self-grade of eight, for example, corresponds to a B. Figure 2: Exact questions asked to participants about their emotional affect alongside a complimentary visualization based on the Russell [33] circumplex of affect model.”
InfluentialDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Psychology of Learning and Motivation
(2021) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(21)x0002-3 · Cited by 5,027
Sweller, J. (2011) · Psychology of Learning and Motivation · doi:10.1016/b978-0-12-387691-1.00002-8 · Cited by 4,856
The influence of academic self-efficacy on academic performance: a systematic review.
Honicke, T., & Broadbent, J. (2016) · Educational Research Review · doi:10.1016/j.edurev.2015.11.002 · Cited by 1,554 · cited 2×
“Self-efficacy is another key predictor for academic success [17,36].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Contingency Tables Involving Small Numbers and the χ2 Test
Yates, F. (1934) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.2307/2983604 · Cited by 1,281 · cited 1×
“The sign test was calculated by normal approximation using the Yates [38] continuity corrected equal proportions test with prop.test(p=0.5,correct=T) from the standard stats library.”
InfluentialVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626 · cited 1×
“Due to lack of skills and experience, programming tasks can be very challenging for novices [26] .”
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2×
“It performed well enough to earn exam scores that placed it above 75% of the students in the course [12] .”
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 1×
“This raises concerns that students will overuse AIDEs, fail to learn from educational programming assignments, and be under-prepared for their careers [5] .”
BackgroundStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · cited 8×
“The AI-assisted students who performed best on their entrance exam also showed significant improvements in learning based on the summative exams, indicating that AIDEs can potentially provide learning benefits to more advanced learners [21] .”
InfluentialAn empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 2×
“While many studies have evaluated AIDEs in isolation (e.g., [8,27,39] ), it is more useful to consider how people use them.”
Assessing the quality of GitHub copilot’s code generation
Yetiştiren, B., Ozsoy, I., & Tüzün, E. (2022) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 193 · cited 2×
“While many studies have evaluated AIDEs in isolation (e.g., [8,27,39] ), it is more useful to consider how people use them.”
Gerjets, P., Scheiter, K., & Catrambone, R. (2006) · Learning and Instruction · doi:10.1016/j.learninstruc.2006.02.007 · Cited by 178 · cited 3×
“In the case of [13], increased learning performance was associated with lower effort and “task demand,” a combination of the mental and physical dimensions together.”
Wilson, K. S., & Narayan, A. (2014) · Educational Psychology · doi:10.1080/01443410.2014.926312 · Cited by 160 · cited 2×
“Self-efficacy is another key predictor for academic success [17,36].”
Barker, L. J., McDowell, C., & Kalahar, K. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1539024.1508923 · Cited by 158
Experiencing programming assignments in CS1: the emotional toll
Kinnunen, P., & Simon, B. (2010) · Proceedings of the Sixth international workshop on Computing education research · doi:10.1145/1839594.1839609 · Cited by 156 · cited 1×
“Programmers’ emotions are often overlooked, but can have significant impacts on performance [11] and student experience [22].”
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
Asleep at the keyboard? Assessing the security of GitHub copilot’s code contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2025) · Communications of the ACM · doi:10.1145/3610721 · Cited by 140
The Affective Experience of Novice Computer Programmers
Bosch, N., & D’Mello, S. K. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0069-5 · Cited by 134
Revisiting why students drop CS1
Petersen, A., Craig, M., Campbell, J., & Tafliovich, A. (2016) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999552 · Cited by 115 · cited 1×
“Mismatches between students’ abilities and the demands of a CS1 course can lead them to drop the course [3,31] or the CS major altogether [6].”
Basawapatna, A. R., Repenning, A., Koh, K. H., & Nickerson, H. (2013) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2493394.2493404 · Cited by 103 · cited 1×
“The Zone of Proximal Flow Model considers a similar phenomenon, warning that excessively easy coursework can also harm learning via boredom [4].”
Student perceptions of computer science
(2008) · doi:10.1145/1352322.1352274 · Cited by 83
(2020) · IEEE Standards · doi:10.1109/ieeestd.2020.9084219 · Cited by 77 · cited 2×
“IEEE 7010-2020 is an active standard for assessing the benefits and harms of autonomous and intelligent systems on human well-being [19] .”
BackgroundChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70 · cited 1×
“Popovici reported a similar finding for assignments in a functional programming course [32] .”
Lishinski, A., & Rosenberg, J. M. (2021) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3446871.3469740 · Cited by 55 · cited 1×
“It is also an outcome of the learning process and correlates with positive emotions [25].”
How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot
Al Madi, N. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3560438 · Cited by 50 · cited 2×
“Previous studies either allowed internet access [35,37] or did not specify [1,21], so this choice is important to consider.”
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Dehbozorgi, N., Maher, M. L., & Dorodchi, M. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274119 · Cited by 25 · cited 1×
“Programmers’ emotions are often overlooked, but can have significant impacts on performance [11] and student experience [22].”
BackgroundAssessing Workload Perception in Introductory Computer Science Projects using NASA-TLX
Madi, N. A., Peng, S., & Rogers, T. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499406 · Cited by 14 · cited 5×
“This discrepancy may be attributable to the counterintuitive layout of this scale or the fact that we used raw TLX ratings, while [2] reported ratings scaled according to contribution to overall workload.”
InfluentialDistribution Needed for Hypothesis Testing
(2013) · Cited by 1
Research works printed but not linked: 6
- 10.1007/s40593-015-0069--5 · doi:10.1007/s40593-015-0069--5
- Torsten Hothorn, Kurt Hornik, Mark A. van de Wiel, Henric Winell, and Achim Zeileis. 2015. Conditional inference procedures in a permutation test framework. Coin Package R Version 31 3 Retrieved April 6, (2015), 2016.
- Boris Iglewicz and David C. Hoaglin. 1993. Volume 16: how to detect and handle outliers. Quality Press.
- Laerd Statistics. 2015. Tutorials for SPSS Statistics. Retrieved December 12, 2023 from https://statistics.laerd.com/
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. ArXiv Prepr. ArXiv220313474 (2022).
- Julie Pallant. 2007. SPSS Survival Manual. Open University Press New York, NY, USA.
Cited by
Lepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 90
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 48
AI, Jobs, and the Automation Trap: Where Is HCI?
Constantinides, M., & Quercia, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729191 · Cited by 19
“Some evidence suggests these tools may hinder learning by bypassing foundational understanding [25].”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Metadata
- Authors
- Gardella, N., Pettit, R., & Riggs, S. L.
- Affiliations
- University of Virginia (US)
- Venue
- Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2024-07-03
- Pages
- 290-296
- OpenAlex
- W4400268466
- Cited by
- 15
- References
- 28
- Citations by year
- 2026: 9 · 2025: 6
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1842490 and 1829004)
- Commonwealth Cyber Initiative
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- AI-assisted code generation, GitHub Copilot, novice programmers, programming education, mental workload, emotions, self-efficacy, programming performance, time pressure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.