The DOI matches its Crossref record; 105 of the 106 references it cites were found in a registry or an index; 15 of the 30 sources its claims cite were checked for support.
“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
Open accessAbstract
Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.
Overview
CoChat Summary
TL;DR
This paper reports the first observational and interview study of novice CS1 students using GitHub Copilot on a typical introductory programming assignment. It finds that novices often saw Copilot as fast and helpful, but also struggled to understand its suggestions, worried about over-reliance, and exhibited two new interaction patterns: shepherding and drifting.
Key findings
- Most students perceived Copilot as helping them write code faster and avoid syntax errors, but many also worried about not understanding generated code and becoming dependent on it; “Outsourcing” concerns appeared in 46 interview excerpts from 17 unique participants, and “Speed” in 37 excerpts from 17 unique participants.
- The study identifies two novel interaction patterns: *shepherding*, where students spent most of their time coercing Copilot toward a solution, and *drifting*, where students moved from one incorrect suggestion to the next, backtracking and deleting code without direction.
- Students showed cognitive and metacognitive difficulties: “Confused” was observed 65 times across 13 unique participants, and “Metacognitive” appeared 24 times in observations and 17 times in interviews, each across 10 unique participants.
- Some students attributed intelligence or agency to Copilot, e.g., “It like read my mind!” and “I thought it was weird that it knows what I want,” while others raised fears about future job impacts, privacy, plagiarism, and over-reliance.
Methods
- Nineteen novice programmers (5 women, 14 men, ages 18–22) enrolled in a CS1 course at a Midwestern private research university in the USA were observed one at a time in April 2022 while solving a Minesweeper-style C++ assignment in Visual Studio Code with Copilot enabled.
- Each participant had 30 minutes, followed a think-aloud protocol, and then completed a short interview about whether Copilot helped them understand the problem, whether they would use it, its advantages, and their fears or worries.
- Data from observations and interviews were analyzed together using reflexive thematic analysis following Braun and Clarke’s six phases; themes included Interactions, Cognitive, Purpose, and Speculation, with saturation treated as the point where additional data produced no new emergent themes.
Limitations
- The paper states that screens and audio were not recorded due to IRB considerations, that study conditions were somewhat lab-like rather than normal in-class assignment conditions, that only Copilot was studied, and that data collection occurred in April 2022 shortly after Copilot’s release, so novelty effects may have shaped results.
- Evident from the text: the sample is small (19 students), from a single institution and single CS1 course, with only a 30-minute exposure to Copilot, so findings may not generalize to other populations, courses, or longer-term use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 3 Not valid · 11 Not checked · 109 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Code generators like Codex have been shown to perform better than the average student on introductory-level programming problems, thus they provide an effective tool for students who might be contemplating cheating [34].”
What the sources say
- Supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - source
Section 4.1.1 (page not available) reports that Codex ranked 17th alongside 71 students on two introductory programming tests, placing it in the top quartile—above the average student. Section 5.3 also describes Codex as a way for students to obtain solutions with low risk of similarity detection, supporting the sentence’s academic-integrity context. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our participants acknowledged that such over-reliance could hinder their own learning, a concern that has also been expressed by computing educators [32].”
What the sources say
- Supportedcites [32] [32] Neil A Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2 (2022), 106-110. - source
The cited paper’s “Learning to Program” discussion (Possible Challenges with AIDE, p. 3) expresses the concern attributed to it: learners may accept AI-generated recommendations without fully understanding them. “Our participants” refers to the citing study’s participants, not to this source. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The primary benefit that novices saw in using Copilot was that it accelerated their progress on the programming task, mirroring results with experienced programmers as reported by Barke et al. [8] and Vaithilingam et al. [80].”
What the sources say
- Supportedcites [8] [8] Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022. Grounded Copilot: How Programmers Interact with Code-Generating Models. arXiv preprint arXiv:2206.15000 (2022). https://doi.org/10.48550/ARXIV.2206.15000 - source
Barke et al. are cited for the parallel finding among experienced programmers, not for the citing paper’s finding about novices. The Introduction says Copilot helps programmers reach a known next step more quickly, and §4.1 describes its use to type out intended code faster. The Related Work section notes that their participants were mostly experienced developers. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [80] [80] Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts . Association for Computing Machinery, New York, NY, USA, 1-7. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On introductory level problems like the one in our study, Copilot generates correct solutions most - but not all - of the time [34].”
What the sources say
- Partly supportedcites [34] [34] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22) . Association for Computing Machinery, New York, NY, USA, 10-19. https://doi.org/10.1145/3511861.3511863 - sourceExperimental — accuracy not yet measured
The Introduction says Codex powers GitHub Copilot, and §4.1 reports that Codex solved 19 of 23 introductory programming questions within ten attempts. But only about half were solved on the first attempt, so the paper does not establish that a single Copilot-generated solution is correct most of the time. Page numbers are unavailable in our copy. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Still others utilized the system like a colleague who can help them when stuck, a behavior previously documented between students co-regulating their learning in study groups [69], and which matches the idea of Copilot being 'your AI pair programmer.'”
What the sources say
- Supportedcites [69] [69] James Prather, Lauren Margulieux, Jacqueline Whalley, Paul Denny, Brent N. Reeves, Brett A. Becker, Paramvir Singh, Garrett Powell, and Nigel Bosch. 2022. Getting By With Help From My Friends: Group Study in Introductory Programming Understood as Socially Shared Regulation. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22) . Association for Computing Machinery, New York, NY, USA, 164-176. https://doi.org/10.1145/3501385.3543970 - source
The cited part of the sentence is the peer-helping-when-stuck behaviour documented in study groups, not the Copilot framing. Prather et al. (2022) report 'help seeking -social' as the most common co-regulation theme in their study-group data (§4.2.2 Co-regulation Themes), illustrated by student statements such as 'When we try hard but still can not figure out how to solve the problem, we tend to ask each other' and 'I work with my friends when I become stuck ... and we help each other to understand the problem better and tackle it' (§5.3). Table 3 quantifies the theme (35 responses at Lab 7, 35 at Lab 12 for co-regulation), and §5.2 discusses the shift from resources to social help-seeking within groups. Thus the source does document students treating peers as a helping colleague when stuck while co-regulating in study groups. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
56 more claims · 1 with their cited sources checked
Create a free account to see all 134 claims and to verify the 2 references still open.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 38
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our findings affirm that AI-generated outputs can boost productivity; they often require rigorous review and contextual interpretation to ensure software safety and integrity [14].”
From Redefining the Programmer: Human-AI Collaboration, LLMs, and Security in Modern Software Engineering (2025) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the dependency on GenAI to complete academic tasks could reduce students' critical thinking and independent problem-solving skills [16,36].”
From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [159]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“From this study, two main modalities emerged (Prather et al., 2024): in 'acceleration' mode, the programmer is aware of their next steps and utilizes the tool to speed up the coding process. In contrast, in the 'exploration' mode, the programmer is uncertain about how to proceed and employs the tool to investigate his options.”
From Understanding user mental models in AI-driven code completion tools: Insights from an elicitation study (2025) · cites this paper as Prather et al., 2024
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Reflecting on reflexive thematic analysis
Braun, V., & Clarke, V. (2019) · Qualitative Research in Sport, Exercise and Health · doi:10.1080/2159676x.2019.1628806 · Cited by 19,677 · cited 1×
“Coding is fluid and involves combining and splitting codes in an iterative process as the researchers become more familiar with the data, and throughout the analysis [19].”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Extreme Programming Explained: Embrace Change
Beck, K. L., & Andres, C. (2000) · Cited by 5,902
Conceptual and design thinking for thematic analysis.
Braun, V., & Clarke, V. (2022) · Qualitative Psychology · doi:10.1037/qup0000196 · Cited by 4,502 · cited 1×
“To analyze the data we used relexive thematic analysis [18, 20] which aims to understand pattern and meaning present in qualitative data.”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“The internal architecture of the models is opaque for most users, which has led to multiple approaches for constructing functional prompts, often called “prompt engineering” [51].”
BackgroundCognitive load theory, learning difficulty, and instructional design
Sweller, J. (1994) · Learning and Instruction · doi:10.1016/0959-4752(94)90003-5 · Cited by 3,801 · cited 1×
“We hypothesize that lengthy auto-generated code suggestions may lead to an increased cognitive load [98] among novices and call on researchers to explore this in future work.”
BackgroundProtocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 1×
“These include Deepmind AlphaCode [61], Amazon CodeWhisperer [5], CodeBert [41], Code4Me [27], FauxPilot [40], and Tabnine [99].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Unlocking the clubhouse: women in computing
(2002) · Choice Reviews Online · doi:10.5860/choice.39-5237 · Cited by 1,958
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Cheating in Academic Institutions: A Decade of Research
McCabe, D. L., Trevino, L. K., & Butterfield, K. D. (2001) · Ethics & Behavior · doi:10.1207/s15327019eb1103_2 · Cited by 1,379 · cited 1×
“Academic misconduct is a widespread problem in many disciplinary areas [59, 76].”
BackgroundEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 3×
“DeepMind purports that AlphaCode can perform similar to the median competitor in programming competitions [49].”
BackgroundCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 1×
“Although XAI has been studied by researchers for nearly 40 years [26], it is increasing in importance as modern machine learning becomes more frequent in our daily lives [62].”
BackgroundDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 1×
“, the way humans reason informs XAI techniques [82].”
BackgroundAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · cited 1×
“While most work in this area focuses on the algorithms, HCI researchers have called for more studies on how it impacts the humans using or benefiting from these AI systems [1].”
BackgroundThe epistemology of a rule-based expert system —a framework for explanation
Clancey, W. J. (1983) · Artificial Intelligence · doi:10.1016/0004-3702(83)90008-5 · Cited by 712 · cited 1×
“Although XAI has been studied by researchers for nearly 40 years [23], it is increasing in importance as modern machine learning becomes more frequent in our daily lives [50].”
BackgroundVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6592.6594 · Cited by 567 · cited 1×
“There is plenty of evidence that students struggle to develop effective plans [97] and to implement plans that are developed [32, 52].”
BackgroundAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationAsleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 4×
“who categorized experienced programmers’ Copilot use into łexplorationž and łaccelerationž [8].”
BackgroundInfluentialBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401
Self-citationProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388
A systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370 · cited 4×
“attempts to provide more contextualized help to students learning to program, although effective programming support remains a challenge [43].”
BackgroundInfluentialEMOTION IN HUMAN-COMPUTER INTERACTION
(2007) · The Human-Computer Interaction Handbook · doi:10.1201/9781410615862-13 · Cited by 344
100 Questions (and Answers) About Qualitative Research
Given, L. M. (2016) · doi:10.4135/9781483398655 · Cited by 309 · cited 1×
“We found saturation being the point at which ‘additional data do not lead to any new emergent themes’ [36].”
BackgroundPlagiarism in the age of massive generative pre-trained transformers (GPT-3)
Dehouche, N. (2021) · Ethics in Science and Environmental Politics · doi:10.3354/esep00195 · Cited by 296 · cited 1×
“We expect significant implications ahead for issues of academic integrity, and a clear need for an updated definition of plagiarism [30].”
BackgroundThe Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Roselli, D., Matthews, J., & Talagala, N. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317590 · Cited by 237 · cited 1×
“We now return to our research questions to discuss the implications of our findings on novice programmers using LLM-based AI code generators such as Copilot for the first time.”
BackgroundUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · cited 2×
“found that Codex could be used to enhanced programming error messages in some cases, although they note that the performance of the model was not good enough to use with students directly [60].”
BackgroundSelf-citationProgramming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223 · cited 1×
“In the context of programming, it involves how programmers solve problems and the process they go through to do so [53] and the programming environment likely plays a role in this [41].”
BackgroundComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210 · cited 2×
“Large language models have also been found to be capable of producing pedagogical code explanations [58, 67, 94].”
BackgroundSelf-citationBecker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · cited 2×
“Large language models have also been found to be capable of producing pedagogical code explanations [58, 67, 94].”
BackgroundSelf-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Self-citationUsing GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · cited 1×
“Wermelinger [106] evaluated the capabilities of Copilot in the spirit of Finnie-Ainsley et al. [42] who evaluated Codex and found it to still miss the mark more often than not.”
MethodExamining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H. A., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2023) · Proceedings - IEEE Symposium on Security and Privacy/Proceedings of the ... IEEE Symposium on Security and Privacy · doi:10.1109/sp46215.2023.10179420 · Cited by 189
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123433119 · Cited by 187 · cited 1×
“LLMs trained with source code are also good at solving other problems, such as solving university-level math [36], probability and statistics [100], and machine learning [109].”
MethodMy AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 1×
“A later study conducted by the same group found that Codex was similarly performant in Data Structures and Algorithms (CS2) exams [43].”
ResultSelf-citationSubgoals, Context, and Worked Examples in Learning Computing Problem Solving
Morrison, B. B., Margulieux, L. E., & Guzdial, M. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787733 · Cited by 174 · cited 1×
“Another form of scaffolding could be Copilot generating only comments outlining the program’s general structure, similar to subgoal labels [76].”
BackgroundIntelligent tutoring systems for programming education: a systematic review
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160492 · Cited by 166
Self-citationExperiencing programming assignments in CS1: the emotional toll
Kinnunen, P., & Simon, B. (2010) · Proceedings of the Sixth international workshop on Computing education research · doi:10.1145/1839594.1839609 · Cited by 156 · cited 1×
“On the other hand, negative emotions can reduce student ability to persist and complete tasks required during learning [44].”
BackgroundWhat Do We Think We Think We Are Doing? Metacognition and Self-Regulation in Programming
Prather, J. E., Becker, B. A., Craig, M., Denny, P. C., Loksa, D., & Margulieux, L. E. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406263 · Cited by 156 · cited 1×
“” While metacognitive skills in novice programmers are becoming increasingly important [54, 68], there are clear opportunities for tools like Copilot to scaffold and enhance these behaviors from the very start.”
BackgroundSelf-citationAll syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · cited 2×
“Dealing with syntax errors has historically been a no-toriously difficult task for novices learning to program [11, 32].”
BackgroundSelf-citationThe Role of Self-Regulation in Programming Problem Solving Process and Success
Loksa, D., & Ko, A. J. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960334 · Cited by 143 · cited 1×
“Several participants in our experiment took out a notepad and pencil when they became stuck in an attempt to work the problem, which is a clear example of a reflective metacognitive behavior like self-regulation [64].”
BackgroundMetacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · cited 2×
“Programming is a complex cognitive activity [79] that often involves deep metacognitive knowledge and behaviors [66].”
BackgroundSelf-citationMetacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129 · cited 1×
“We discuss these further in Section 5.”
Self-citationPersonifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125 · cited 1×
“Novices tend to view feedback in programming contexts to be the truth and the systems generating it to be infallible [46].”
BackgroundCS majors' self-efficacy perceptions in CS1: results in light of social cognitive theory
Kinnunen, P., & Simon, B. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016917 · Cited by 116 · cited 1×
“When novice programming students experience negative emotions, it can directly negatively impact their feelings of self-efficacy [45].”
BackgroundBeyond usability: evaluating emotional response as an integral part of the user experience
Agarwal, A., & Meyer, A. (2009) · CHI '09 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1520340.1520420 · Cited by 115 · cited 1×
“Emotion is a key part of creating usable designs [2, 18, 56].”
BackgroundSavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · cited 1×
“Savelka et al. evaluated GPT-3 on multiple choice questions, quizzes, and more complex programming projects in introductory and intermediate programming courses and found that it is capable of achieving more than passing scores [95].”
Background50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research
Becker, B. A., & Quille, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287432 · Cited by 107 · cited 5×
“In this work we provide the first exploration of Copilot use by novices in an introductory programming (CS1) class on a typical novice programming task through observations and interviews.”
BackgroundInfluentialSelf-citation(2019) · doi:10.1145/3287324.3287374 · Cited by 104
Self-citationCognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · cited 1×
“Prior work has demonstrated that student learning is impacted by both cognitive [37], and meta-cognitive processes (such as explicitly monitoring their own progress and reflecting on the effectiveness of their problem solving strategies) [65].”
BackgroundCodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92
The effects of pair-programming on performance in an introductory programming course
McDowell, C., Werner, L., Bullock, H., & Fernald, J. (2002) · ACM SIGCSE Bulletin · doi:10.1145/563517.563353 · Cited by 88
Generation CS: the growth of computer science
Camp, T., Adrion, W. R., Bizot, B., Davidson, S., Hall, M., Hambrusch, S., Walker, E., & Zweben, S. (2017) · ACM Inroads · doi:10.1145/3084362 · Cited by 82
Contract cheating and blackmail: a risky business?
Yorke, J., Sefcik, L. T., & Veeran-Colton, T. (2020) · Studies in Higher Education · doi:10.1080/03075079.2020.1730313 · Cited by 77 · cited 1×
“Compared to traditional forms of cheating, such as contract cheating or copying work from other students, AI-generated solutions do not require communication with another person and thus there are fewer risks of being caught [84].”
BackgroundNegotiating the Maze of Academic Integrity in Computing Education
Simon, Sheard, J., Morgan, M., Petersen, A., Settle, A., Sinclair, J., Cross, G., & Riedesel, C. (2016) · Proceedings of the 2016 ITiCSE Working Group Reports · doi:10.1145/3024906.3024910 · Cited by 76 · cited 1×
“Academic misconduct is a widespread problem in many disciplinary areas [59, 76].”
BackgroundOn Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · cited 1×
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [29, 72] and metacognitive behaviors [70].”
BackgroundSelf-citationHuman Cognition and Programming
Ormerod, T. (1990) · Elsevier eBooks · doi:10.1016/b978-0-12-350772-3.50009-4 · Cited by 71 · cited 1×
“Programming is a complex cognitive activity [64] that often involves deep metacognitive knowledge and behaviors [54].”
BackgroundPlagiarism in Programming Assessments: A Systematic Review
Albluwi, I. (2020) · ACM Transactions on Computing Education · doi:10.1145/3371156 · Cited by 63
Do Humans Trust Advice More if it Comes from AI?: An Analysis of Human-AI Interactions
Vodrahalli, K., Daneshjou, R., Gerstenberg, T., & Zou, J. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534150 · Cited by 63
AI-Driven Development Is Here: Should You Worry?
Ernst, N. A., & Bavota, G. (2022) · IEEE Software · doi:10.1109/ms.2021.3133805 · Cited by 62 · cited 2×
“Our participants acknowledged that such over-reliance could hinder their own learning, a concern that has also been expressed by computing educators [32].”
BackgroundFix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61
Self-citationMalmi, L., Sheard, J., Kinnunen, P., Simon, & Sinclair, J. (2020) · Proceedings of the 2020 ACM Conference on International Computing Education Research · doi:10.1145/3372782.3406279 · Cited by 60 · cited 1×
“Emotion is a key part of creating usable designs [2, 18, 56].”
BackgroundOn Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · cited 1×
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [34, 89] and metacognitive behaviors [87].”
BackgroundSelf-citationReeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56 · cited 1×
“evaluated the ability of GPT-3 to solve Parsons Problems and found that the nature of generative AI tools as next-token-predictors means that they often struggle to only use the lines of code provided to them in the prompt to solve the puzzle [90].”
BackgroundSelf-citationBird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2023) · Communications of the ACM · doi:10.1145/3589996 · Cited by 55
Successful students' strategies for getting unstuck
McCartney, R., Eckerdal, A., Mostrom, J. E., Sanders, K., & Zander, C. (2007) · Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1268784.1268831 · Cited by 54 · cited 1×
“To maintain progress and positive learning experiences, an outcome desirable for both teachers and students, there is a need to support students who are stuck [73].”
BackgroundPlagiarism in Take-home Exams: Help-seeking, Collaboration, and Systematic Cheating
Hellas, A., Leinonen, J., & Ihantola, P. (2017) · Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3059009.3059065 · Cited by 50 · cited 1×
“This is a frustrating experience for students [10] that can limit their learning progress, and may result in undesirable behaviors such as copying [38].”
BackgroundSelf-citationSTOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance
Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021) · Proceedings of the 52nd ACM Technical Symposium on Computer Science Education · doi:10.1145/3408877.3432430 · Cited by 48 · cited 2×
“This is similar to earlier results that have found that students sometimes develop an over-reliance on automatically generated feedback from automated assessment systems [7].”
BackgroundResultThe Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior
Karvelas, I., Li, A. D. R., & Becker, B. A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366882 · Cited by 42 · cited 1×
“In the context of programming, it involves how programmers solve problems and the process they go through to do so [65] and the programming environment likely plays a role in this [51].”
BackgroundSelf-citationExploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40 · cited 1×
“observed novice programmers using ive program synthesis tools [50].”
MethodA Survey of Automated Programming Hint Generation: The HINTS Framework
McBroom, J., Koprinska, I., & Yacef, K. (2022) · ACM Computing Surveys · doi:10.1145/3469885 · Cited by 39 · cited 1×
“More recently, automated hint generation systems have been employed to generate hints for any programming exercises [58].”
MethodFooling MOSS Detection with Pretrained Language Models
Biderman, S., & Raff, E. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557079 · Cited by 37 · cited 1×
“Biderman and Raff show that introductory level programming assignments can be completed by AI code generators without triggering suspicion from MOSS, a state of the art code plagiarism detection tool [13].”
BackgroundFooling MOSS Detection with Pretrained Language Models
Biderman, S., & Raff, E. (2022) · arXiv · doi:10.48550/arxiv.2201.07406 · Cited by 37
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Self-citationMetacodenition: Scaffolding the Problem-Solving Process for Novice Programmers
Pechorina, Y., Anderson, K., & Denny, P. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576130 · Cited by 33
Self-citationCan Mobile Gaming Psychology Be Used to Improve Time Management on Programming Assignments?
Irwin, M. S., & Edwards, S. H. (2019) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3300115.3309517 · Cited by 32 · cited 1×
“In the case of these systems, prior work has presented multiple ways of trying to combat over-reliance such as submission penalties [7, 48] or limiting the number of submissions students can have [39, 48].”
BackgroundA Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32 · cited 1×
“In the case of these systems, prior work has presented multiple ways of trying to combat over-reliance such as submission penalties [7, 59] or limiting the number of submissions students can have [35, 49, 59].”
BackgroundSelf-citationPrather, J., Margulieux, L., Whalley, J., Denny, P., Reeves, B. N., Becker, B. A., Singh, P., Powell, G., & Bosch, N. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543970 · Cited by 31 · cited 1×
“Still others utilized the system like a colleague who can help them when stuck, a behavior previously documented between students co-regulating their learning in study groups [86], and which matches the idea of Copilot being łyour AI pair programmer.”
BackgroundSelf-citationExploring the Applicability of Simple Syntax Writing Practice for Learning Programming
Leinonen, A., Nygren, H., Pirttinen, N., Hellas, A., & Leinonen, J. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287378 · Cited by 27 · cited 1×
“This form of typing practice has pedagogical benefits for learners and researchers have explored similar typing exercises, in which learners must type out code examples, as a way to help novice students practice low-level implementation skills [44, 57].”
BackgroundSelf-citationPromoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Denny, P., Whalley, J., & Leinonen, J. (2021) · Proceedings of the 23rd Australasian Computing Education Conference · doi:10.1145/3441636.3442309 · Cited by 24 · cited 1×
“In the case of these systems, prior work has presented multiple ways of trying to combat over-reliance such as submission penalties [7, 59] or limiting the number of submissions students can have [35, 49, 59].”
BackgroundSelf-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 4×
“who categorized experienced programmers’ Copilot use into łexplorationž and łaccelerationž [8].”
BackgroundInfluentialTyping Exercises as Interactive Worked Examples for Deliberate Practice in CS Courses
Gaweda, A. M., Lynch, C. F., Seamon, N., Silva de Oliveira, G., & Deliwa, A. (2020) · Proceedings of the Twenty-Second Australasian Computing Education Conference · doi:10.1145/3373165.3373177 · Cited by 21 · cited 1×
“This form of typing practice has pedagogical benefits for learners and researchers have explored similar typing exercises, in which learners must type out code examples, as a way to help novice students practice low-level implementation skills [35, 47].”
BackgroundPop Quiz! Can a Large Language Model Help With Reverse Engineering?
Pearce, H., Tan, B., Krishnamurthy, P., Khorrami, F., Karri, R., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2202.01142 · Cited by 16
The Role of Working Memory in Program Tracing
Crichton, W., Agrawala, M., & Hanrahan, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445257 · Cited by 13 · cited 1×
“One example of a programming activity that includes both the cognitive and metacognitive aspects is code tracing with concrete inputs, something constrained to working memory [28].”
BackgroundA Dataset and Benchmark for Automatically Answering and Generating Machine Learning Final ExamsPreprint
Zhang, S. J., Shuttleworth, R., Austin, D., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2206.05442 · Cited by 12
Novice Reflections During the Transition to a New Programming Language
Denny, P., Becker, B. A., Bosch, N., Prather, J., Reeves, B., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499314 · Cited by 11 · cited 1×
“The computing education literature documents syntax errors – and the messages they generate – as presenting a significant challenge to both novices and students transitioning to new programming languages [9, 27].”
BackgroundSelf-citationTang, L., Ke, E., Singh, N., Feng, B., Austin, D. W., Verma, N., & Drori, I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_127 · Cited by 9
Computing Education in the Era of Generative AIPreprint
Denny, P. C., Prather, J. E., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02608 · Cited by 9
Self-citationComparative study between automatic hint generation approaches in Intelligent Programming Tutors
MAHDAOUI, M., NOUH, S., ALAOUI, M. S. E., & SADIQ, M. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2021.12.259 · Cited by 8
Concise Graphical Representations of Student Effort on Weekly Many Small Programs
Allen, J. M., & Vahid, F. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432551 · Cited by 5
Student Debugging Practices and Their Relationships with Project Outcomes
Kazerouni, A. M., Mansur, R. S., Edwards, S. H., & Shaffer, C. A. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3293794 · Cited by 4 · cited 1×
“There is plenty of evidence that students struggle to develop effective plans [97] and to implement plans that are developed [32, 52].”
Background
Research works printed but not linked: 3
- Emily M. Bender. 2019. A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help. Future of Artificial Intelligence: Language, Ethics, Technology Workshop.
- 10.1109/SP46215.2023.00001 · doi:10.1109/sp46215.2023.00001
- Ashley D. Rittmayer and Margaret E. Beier. 2008. Overview: Self-efficacy in STEM. SWE-AWE CASEE Overviews 1, 3 (2008), 12.
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Novices may be intimidated by such code generation [19] or may spend too much time reading code that does not further their goals [31].”
BackgroundSelf-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“We posit that these demands are exacerbated by GenAI’s non-determinism and model flexibility (not to be confused with metacognitive flexibility) in terms of (a) the wide range of explicit and implicit parameters that users can adjust, and (b) systems’ ability to work with prompts at a wide range of…”
BackgroundGenerative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda
Bittle, K., & El-Gayar, O. (2025) · Information · doi:10.3390/info16040296 · Cited by 305
“Moreover, the dependency on GenAI to complete academic tasks could reduce students’ critical thinking and independent problem-solving skills [16,36].”
Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citation
Metadata
- Authors
- Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A.
- Affiliations
- Abilene Christian University (US)
- University of Auckland (NZ)
- University College Dublin (IE)
- ORCID
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-11-29
- Volume
- 31
- Issue
- 1
- Pages
- 1-31
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4386099272
- Cited by
- 256
- References
- 91
- Citations by year
- 2026: 89 · 2025: 85 · 2024: 66 · 2023: 16
- FWCI
- 57.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, code generation, novice programmers, introductory programming, CS1, programming education, student perceptions, metacognition, cognitive load, scaffolding, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.