The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index.
Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Open accessAbstract
Artificial Intelligence-driven Development Environments (AIDEs) offer developers revolutionary computer programming assistance. There is great potential in incorporating AIDEs into Computer Science education; however, the effects of these tools should be fully examined before doing so. Here, a within-subjects study was conducted to compare the programming performance, workload, emotion, and self-efficacy of seventeen novices coding with and without use of the GitHub Copilot AIDE under time pressure. Results showed that using the AIDE significantly increased programming efficiency and reduced effort and mental workload but did not significantly impact emotion or self-efficacy. However, participants' performance improved with more experience using the AI, and their self-efficacy followed. The results suggest that students who try AIDEs will likely be tempted to use them for time-sensitive work. There is no evidence that providing AIDEs will aid struggling students, but there is a clear need for students to practice with AI to become competent and confident using it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 3×
“C+ to B) and a self-grade of eight, for example, corresponds to a B. Figure 2: Exact questions asked to participants about their emotional affect alongside a complimentary visualization based on the Russell [33] circumplex of affect model.”
InfluentialDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
The Psychology of Learning and Motivation
(2021) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(21)x0002-3 · Cited by 5,027
Sweller, J. (2011) · Psychology of Learning and Motivation · doi:10.1016/b978-0-12-387691-1.00002-8 · Cited by 4,856
The influence of academic self-efficacy on academic performance: a systematic review.
Honicke, T., & Broadbent, J. (2016) · Educational Research Review · doi:10.1016/j.edurev.2015.11.002 · Cited by 1,554 · cited 2×
“Self-efficacy is another key predictor for academic success [17,36].”
BackgroundEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Contingency Tables Involving Small Numbers and the χ2 Test
Yates, F. (1934) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.2307/2983604 · Cited by 1,281 · cited 1×
“The sign test was calculated by normal approximation using the Yates [38] continuity corrected equal proportions test with prop.test(p=0.5,correct=T) from the standard stats library.”
InfluentialVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Research works printed but not linked: 6
- 10.1007/s40593-015-0069--5 · doi:10.1007/s40593-015-0069--5
- Torsten Hothorn, Kurt Hornik, Mark A. van de Wiel, Henric Winell, and Achim Zeileis. 2015. Conditional inference procedures in a permutation test framework. Coin Package R Version 31 3 Retrieved April 6, (2015), 2016.
- Boris Iglewicz and David C. Hoaglin. 1993. Volume 16: how to detect and handle outliers. Quality Press.
- Laerd Statistics. 2015. Tutorials for SPSS Statistics. Retrieved December 12, 2023 from https://statistics.laerd.com/
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. ArXiv Prepr. ArXiv220313474 (2022).
- Julie Pallant. 2007. SPSS Survival Manual. Open University Press New York, NY, USA.
Cited by
Lepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 90
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 48
AI, Jobs, and the Automation Trap: Where Is HCI?
Constantinides, M., & Quercia, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729191 · Cited by 19
“Some evidence suggests these tools may hinder learning by bypassing foundational understanding [25].”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Metadata
- Authors
- Gardella, N., Pettit, R., & Riggs, S. L.
- Affiliations
- University of Virginia (US)
- Venue
- Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2024-07-03
- Pages
- 290-296
- OpenAlex
- W4400268466
- Cited by
- 15
- References
- 28
- Citations by year
- 2026: 9 · 2025: 6
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1842490 and 1829004)
- Commonwealth Cyber Initiative
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- AI-assisted code generation, GitHub Copilot, novice programmers, programming education, mental workload, emotions, self-efficacy, programming performance, time pressure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation. https://app.cochat.ai/papers/10.1145/3649217.3653615
- The paper
- Gardella, N., Pettit, R., & Riggs, S. L. (2024). Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation. Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. https://doi.org/10.1145/3649217.3653615
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.