CoChat
Sign Up
Cited by 400Artificial Intelligence in EducationTeaching and Learning ProgrammingArtificial Intelligence in Healthcare and Education

The Robots Are Here: Navigating the Generative AI Revolution in Computing Education

Open access
Prather, James, Denny, Paul, Leinonen, Juho, Becker, Brett A., Albluwi, Ibrahim, Craig, Michelle, Keuning, Hieke, Kiesler, Natalie +7 more·2023·Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education
DOI 10.1145/3623762.3633499
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in artificial intelligence (AI) and specifically generative AI (GenAI) are threatening to fundamentally reshape computing and society. Largely driven by large language models (LLMs), many tools are now able to interpret and generate both natural language instructions and source code. These capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of generative AI in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles, nearly 80% of which have been published in the first 8 months of 2023. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards GenAI/LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of several current GenAI models/tools on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving.

Verified 2026-10-06 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

This ITiCSE working group report explores how generative AI and large language models are reshaping computing education through a literature review, international surveys, educator interviews, ethics/policy analysis, and benchmarking. It finds rapid LLM capability improvement and broad expected impact, with students and instructors aligned on some views but differing on policy clarity and ethical use.

Key findings

  • A scoping literature review synthesised 71 primary articles, nearly 80% published in the first 8 months of 2023; the largest category (35 papers) assessed LLM performance/capabilities, finding LLMs generally perform at or above average students on code generation but weaker on multiple-choice questions, Parsons problems, and computational thinking tasks. The most common risk authors raised was student over-reliance on LLMs.
  • Surveys of 171 students and 57 instructors across 20 countries found similar experiences and expectations, including strong agreement that GenAI cannot replace human instructors; instructors found course/university policies less clear than students did, and 50.8% of instructors believed many or almost all of their students were using GenAI tools unethically.
  • Interviews with 22 computing educators from nine countries and five continents documented shifts toward “process over product,” invigilated/oral exams, reduced weighting of unsupervised homework, and greater emphasis on code reading and critical thinking, while some educators preserved core programming objectives.
  • Benchmarking showed GPT-4 solved all but one problem in a replication of Finnie-Ansley et al. (T2-Q12), outperforming Codex; GPT-3.5 solved most problems; Copilot solved 20/23 exam problems and 4/7 rainfall variants. On APPS, GPT-4 averaged 51.5% of test cases versus GPT-3.5’s 39.2%; on FalconCode, GPT-4 scored 45.4% overall and 74.1% on a cleaned subset.

Methods

  • Scoping literature review using keyword searches in ACM Digital Library, Taylor & Francis Online, and IEEE Xplore, plus seed papers and two forward and one backward snowballing phases; 866 papers were filtered down to 71 relevant papers.
  • Two online surveys conducted July–August 2023 with 57 instructors from 12 countries and 171 students from 17 countries (responses spanning 20 countries), including Likert-scale items and open-response questions analysed with reflexive thematic analysis.
  • Semi-structured interviews with 22 computing educators from nine countries and five continents, analysed thematically; additional ethics/policy review of selected university LLM policies using the ACM Code of Ethics, and benchmarking/replication analyses of GPT-4, GPT-3.5-turbo, and GitHub Copilot on computing education datasets.

Limitations

  • Stated: the literature review covered April–August 2023 and only literature from 2021 to August 2023, used a single snowballing round, and included arXiv/non-peer-reviewed work; survey recruitment via SIGCSE mailing lists and snowball sampling risks selection and response bias; interview sampling was geographically skewed (US 55%, no Africa, one each from Asia, Oceania, and South America; only 3 of 22 identified as women), semi-structured interviews can impose interviewer bias, and interviewee claims were not verified.
  • Stated: benchmarking focused on code generation, and available datasets often lacked starter code, data files, or formatting instructions; test cases could be incomplete or incorrect, and public datasets may have been included in model training.
  • Evident from the text: the field’s rapid pace means findings may quickly become outdated, and the survey samples are modest and not necessarily representative of computing education globally.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3623762.3633499

Cites

153 of 154 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 225 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GPT-4 had only a single piece of research using it in the first snowballing phase.

    “the most recent version of GPT (GPT-4) had only a single piece of research using it.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Backward snowballing phase 1 resulted in 381 papers.

    “The backward snowballing phase, which considered all papers in the reference list for each paper in the reference set, resulted in 381 papers.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    Five papers used LLMs to analyse student work.

    “Five papers used LLMs to analyse student work”
    Quote in the paperNot a substantial claim
  • Findingown data

    Considering all papers up to August 2023, the most commonly used LLM was GPT-3/3.5.

    “Table 4 shows the results considering all the papers we analysed (i.e. up to August 2023), where the most commonly used LLM has become GPT-3/3.5”
    Quote in the paperNot a substantial claim
  • Findingown data

    Twelve papers were surveys or position papers, summarising or discussing research conducted by others.

    “Twelve papers were surveys or position papers, summarising or discussing research conducted by others”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 155
  • May combine two claims. 123
  • The quote only partly states this claim. 3

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Some recent papers found in the second phase snowballing focused on introducing tools around LLMs, such as CodeHelp and Promptly.

    “Indeed, some recent papers found in the second phase snowballing focused on introducing tools around LLMs, such as CodeHelp [127] and Promptly [71].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [127] source

      Not checked in the free run.

    • cites [71] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Findingnone given

    The papers reviewed identified a number of potential opportunities that could positively impact computing education.

    “The papers we reviewed identified a number of potential opportunities that could positively impact computing education.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Some papers looked into other types of exercises such as multiple-choice questions, textbook questions, exam questions, computational thinking tasks, and textual reports.

    “Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [175] source

      Not checked in the free run.

    • cites [176]

      Not checked in the free run.

    • cites [177]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The reviewed papers broadly fall into five categories based on the role that the LLM plays in the study.

    “The papers we reviewed broadly fall into five categories, with respect to the role that the LLM plays in the study”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Slightly more of the work examined presents explicit research questions.

    “and slightly more of the work we examined present explicit research questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The literature review identified 71 relevant papers, comprising 10 reference papers, 28 papers from the first snowballing phase, and 33 papers from the second snowballing phase.

    “These exclusions left us with 71 relevant papers (10 reference papers, 28 papers from the first snowballing phase, and 33 papers from the second).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    A total of nine papers looked into interactions between programmers and LLMs.

    “A total of nine papers looked into interactions between programmers and LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The initial ACM Digital Library search on 26 April 2023 resulted in 19 papers.

    “The search was conducted on 26th April 2023 and resulted in 19 papers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After filtering, 80 papers were identified for careful reading.

    “After the filtering was complete, a total of 80 papers were identified for careful reading.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Five papers in the literature review used interviews to understand user perceptions and attitudes towards LLMs.

    “Five papers in the literature review used interviews to understand user perceptions and attitudes towards LLMs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority of research focuses on Python, with some work involving Java and C.

    “The majority of the research focuses on Python, with some work involving Java and C.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Improving the learning experience for students was another common opportunity that emerged.

    “Improving the learning experience for students was another common opportunity that emerged.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    During in-depth reading, 9 papers were flagged as not relevant.

    “During this indepth reading phase, 9 papers were flagged as not being relevant.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The reference paper set includes 10 papers.

    “The resulting set of reference papers (”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Codex is now no longer available.

    “which is now no longer available”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    More than thirty papers looked into assessing the performance or capabilities of large language models.

    “More than thirty papers looked into assessing the performance or capabilities of large language models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The second snowballing phase covered only a few months but resulted in a number of papers very close to the first phase, which covered around two and a half years.

    “Interestingly, while the second snowballing phase covered only a few months, it resulted in a number of papers that is very close to the number of papers that resulted from the first snowballing phase, which covered a period of around two and a half years.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Several risks were identified by the papers reviewed; authors were concerned that generative AI could be used in ways that limit student learning or make the work of educators more difficult.

    “Several risks were identified by the papers we reviewed.Authors were concerned that generative AI could be used in ways that limit student learning or make the work of educators more difficult [90].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Findingcitation

    LLMs tended to hallucinate issues in student code, which could cause students to focus on non-existent issues instead of actual issues in their code.

    “Hellas et al. [94] found that LLMs tended to hallucinate issues in student code which could cause students to focus on these non-existent issues instead of the actual issues in their code.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [94] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The papers reviewed identify a recurring opportunity for a renewed focus on problem solving in computing education.

    “Another recurring opportunity mentioned in the papers we reviewed was the potential for a renewed focus on problem solving.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [188]

      Not checked in the free run.

    • cites [70] source

      Not checked in the free run.

    • cites [159] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Findingown data

    Other text-based artefacts instructors reported creating were course materials, examples for students, explanations of complex algorithms, and scenarios for highlighting ethical issues in software engineering.

    “Other types of text-based artefacts that instructors reported creating were course materials, examples for students, explanations of complex algorithms, and scenarios for highlighting ethical issues in software engineering.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The qualitative themes identified from open-ended responses are representative of the broader respondent group.

  • May combine two claims.

    Findingown data

    Responses from students and instructors to questions related to experiences and expectations were largely aligned, but some important differences emerged for questions focusing on course policies.

    “Responses from students and instructors to questions related to experiences and expectations were largely aligned.However, some important differences emerged for questions focusing on course policies.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.

  • May combine two claims.

    Findingown data

    Students had mixed opinions about whether university policies were clear, while instructors largely disagreed with the statement that the policies were clear.

    “While students had mixed opinions about whether the university policies were clear, instructors largely disagreed with the statement that the policies were clear.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.

  • May combine two claims.

    Findingnone given

    Several authors raise concerns about the impact of generative AI on teachers and teaching practice, with academic integrity being the most common concern.

    “Several authors raise concerns about the impact of generative AI on teachers and teaching practice.Unsurprisingly, issues of academic integrity were the most common concern raised.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Findingown data

    Both students and instructors expected GenAI tools to play an increasing role in the future of their teaching and learning, as well as in students' future careers.

    “However, both students and instructors also expected GenAI tools to play an increasing role in the future of their teaching and learning, as well as in students' future careers.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.

  • May combine two claims.

    Findingown data

    When used, GenAI tools are primarily used for creating diverse educational materials, but satisfaction regarding quality of outputs varies.

    “The tools, when utilised, are primarily used for creating diverse educational materials, however, satisfaction regarding the quality of outputs vary.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several instructors highlighted the summarisation capacity of GenAI tools to condense long-form text content.

    “Several instructors highlighted the summarisation capacity of GenAI tools since they can effectively condense long-form text content.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The qualitative themes identified from open-ended responses are representative of the broader respondent group.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    What constitutes ethical use is complex; what is considered ethical varied between instructors and students.

    “What constitutes ethical use is complex: What is considered ethical varied between instructors and students.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • May combine two claims.

    Findingown data

    Instructors who used GenAI tools did so to generate a broad array of educational content, including assessment questions, practice exercises and examples, although not all appeared satisfied with the outcomes.

    “Those that were using them were doing so to generate a broad array of educational content, including assessment questions, practice exercises and examples, although not all appeared satisfied with the outcomes.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • May combine two claims.

    Findingown data

    Both students and instructors strongly agreed that GenAI tools cannot replace human instructors and that human teachers provide more effective guidance than GenAI tools.

    “Both groups strongly agreed that GenAI tools cannot replace human instructors and that human teachers provide more effective guidance than GenAI tools.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.

  • Classified as probably not a substantial claim.

    Findingown data

    A number of interviewees highlighted the need to preserve several core learning outcomes.

    “A number of interviewees shared this perspective and highlighted the need to preserve several core learning outcomes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Three students mentioned that standing out to employers will become harder because many applicants might use GenAI to build portfolios that make them appear as solid candidates despite lacking required skills.

    “Three students also mentioned that standing out to employers will become harder, given that many applicants might use GenAI tools to build portfolios that make them appear as solid candidates despite lacking the required skills.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There was a mix of positive and negative responses about effects on employment, consistent with quantitative results.

    “There was a mix of positive and negative responses, which is consistent with the quantitative results of the corresponding Likert question (Question 17 in Appendix B).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingnone given

    Others argued that it is useless to attempt to disallow their use outside closed exam conditions, as students will use them anyway.

    “Others argued that it is useless to attempt to disallow their use outside closedexam conditions, as students will use them anyway.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • May combine two claims.

    Findingown data

    A few responses supported always disallowing them, or disallowing them in all graded assessments.

    “At the other end of the spectrum, a few responses supported always disallowing them, or disallowing them in all graded assessments (regardless of the level, type, topic, etc.).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Another recurring argument was that using GenAI tools in coursework and exams defeats the purpose of assessments.

    “Another recurring argument was that using GenAI tools in coursework and exams defeats the purpose of assessments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Some students believed relying on GenAI tools may reduce their understanding of the material and thus affect their future chances of employment.

    “According to those students, relying on GenAI tools may reduce their understanding of the material and thus affect their future chances of employment.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some instructors argued that it is a function of complexity rather than course-level.

    “Some instructors argued that it is a function of complexity rather than course-level.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Many educators acknowledge the dynamic nature of technology and anticipate potential adjustments to their learning objectives in the near future.

    “Many educators acknowledge the dynamic nature of technology and anticipate potential adjustments to their learning objectives in the near future.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Student answers were more diverse and more polarised than those of the instructors.

    “Student answers were more diverse and more polarised than those of the instructors.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some instructors said their use is fine as long as the student acknowledges that appropriately.

    “Some said that their use is fine as long as the student acknowledges that appropriately.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some students argued for completely disallowing them even outside assessments.

    “Some also argued for completely disallowing them even outside assessments (i.e. while learning).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Several instructors discussed changes in their upcoming course learning objectives.

    “Several instructors discussed changes in their upcoming course learning objectives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Educators seem to be aware of the need to prepare students for industry expectations, which is likely to include LLM use.

    “At the same time, educators seem to be aware of the need to prepare students for industry expectations, which is likely to include LLM use.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Students who use GenAI tools most commonly apply them for paraphrasing or summarising text, for debugging errors in their own code, and slightly less often for code generation.

    “Those who do use GenAI tools most commonly apply them for paraphrasing or summarising text, for debugging errors in their own code, and slightly less often, for code generation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Classified as probably not a substantial claim.

    Findingreasoning

    Computing educators express the need to balance between leveraging LLMs for problem-solving while ensuring that students continue to develop competencies in algorithmic thinking, program comprehension, and debugging.

    “To conclude, computing educators express the need to balance between leveraging LLMs for problem-solving while ensuring that students continue to develop competencies in algorithmic thinking, program comprehension, and debugging.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    The rise of LLMs is causing many instructors to rethink their course learning objectives.

    “The rise of LLMs is causing many instructors to rethink their course learning objectives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Educators who do not alter their learning objectives stress their focus on teaching fundamental programming concepts.

    “Educators who do not alter their learning objectives stress their focus on teaching fundamental programming concepts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Some educators, including Mark Liffiton, view LLMs as valuable tools that can aid students in coding tasks.

    “Some educators, including Mark Liffiton (Illinois Wesleyan University, USA), view LLMs as valuable tools that can aid students in coding tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some students argued that software engineering jobs are particularly at risk.

    “Some students argued that software engineering jobs are particularly at risk.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
    • Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Some instructors have indicated that they will not be dedicating classroom time to explicitly teaching students LLMs.

    “However, in the same way that they expect students to learn to use an IDE on their own time, some instructors have indicated that they will not be dedicating classroom time to explicitly teaching students LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    This September (2023) marks the first academic year where LLMs are a nearly ubiquitous topic.

    “Given that public awareness about LLMs occurred mid-academic year for many institutions (almost universally for North America and Europe in addition to many other parts of the world) this September (2023) marks the first academic year where LLMs are a nearly ubiquitous topic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Caspersen noted that students read more than they write when learning to read natural language, but to-date this has not been the traditional approach when it comes to programming.

    “Caspersen noted that students read more than they write when learning to read natural language, but to-date this has not been the traditional approach when it comes to programming.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Some educators have already started using LLMs as part of the learning process.

    “Some educators have already started using LLMs as part of the learning process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Peter Mawhorter's local policy is that LLMs are not allowed in class.

    “Peter Mawhorter's local policy is that LLMs are not allowed in class.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Viraj Kumar used GitHub Copilot during class for live coding and told students they were free to use it or any other Generative AI for coding; most students polled said they were not using ChatGPT, with the consensus explanation being that they wanted to learn programming themselves and expected to code without assistance in technical job interviews.

    “Viraj Kumar used GitHub Copilot during class for live coding and told students they were free to use it, or any other Generative AI for coding.Students were polled after the course and asked if they were using ChatGPT.Most said that they were not.The consensus explanation seemed to be that they wanted to learn programming themselves and their thinking seemed to be that when they go for technical”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Rather than assessing final solutions as products, some educators are increasingly focusing on assessing learning processes such as submit-in-stages; solution reflection, commentary, critique; interviews; portfolios or learning journals; and presentations.

    “Rather than assessing final solutions as products, some educators are increasingly focusing on assessing learning processes (which is common in other disciplines, e.g., teacher education) such as submit-in-stages; solution reflection, commentary, critique; interviews; portfolios or learning journals; and presentations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several educators mentioned oral exams.

    “Several educators mentioned oral exams.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Liffiton has developed a tool called CodeHelp that uses LLMs, which he is going to use with his students.

    “Liffiton has also developed a tool called CodeHelp [127] that uses LLMs, which he is going to use with his students.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [127] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingreasoning

    Finding and organizing enough computer-based testing facilities is (to date) a challenge as computer labs and institutional machines have become less frequent over the years.

    “However, finding and organizing enough computer-based testing facilities is (to date) a challenge as computer labs and institutional machines have become less frequent over the years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Sven Strickroth noted another challenge caused by institutional policy: significant changes to learning objectives are not easily possible due to the local accreditation cycle which dictates that this occurs only every five years in many institutions.

    “Sven Strickroth (LMU Munich, Germany) noted another challenge that is caused by institutional policy-significant changes to learning objectives are not easily possible due to the local accreditation cycle which dictates that this occurs only every five years in many institutions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Some of these findings support those of Lau and Guo, notably the increased emphasis on invigilated exams, oral exams, and process-based assessments.

    “It is worth noting that some of these findings support those of Lau and Guo [121], notably the increased emphasis on invigilated exams, oral exams, and process-based assessments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [121] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Daniel Zingaro and Leo Porter's new textbook begins by introducing students to the GitHub Copilot plugin before students have learned to write a single line of Python code, and students create their first programs by typing English comments and letting Copilot generate the code.

    “The book begins by introducing students to the GitHub Copilot plugin within the Visual Studio Code IDE before students have learned to write a single line of Python code.Students create their first programs by typing English comments and letting Copilot generate the code.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [155]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Institutional initiatives are beginning to emerge as educators start thinking about the coming academic year.

    “Nonetheless, they are beginning to emerge as educators start thinking about the coming academic year.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Leo Porter is embracing LLMs and is aiming to use them, for instance, in quizzes.

    “On the other side of the coin, Leo Porter is embracing LLMs and is aiming to use them, for instance, in quizzes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Several educators worried that some classes do not lend themselves to the approaches mentioned above.

    “Several educators worried that some classes do not lend themselves to the approaches mentioned above.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Leo Porter is the only interviewee who mentioned pair programming directly.

    “Interestingly, Leo Porter is the only interviewee who mentioned pair programming directly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Many of the student-initiated possibilities discussed come with academic integrity concerns.

    “Many of the student-initiated possibilities discussed above come with academic integrity concerns we discuss in Section 6.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Some instructors are moving towards invigilated exams and assessments, and these may be worth more marks.

    “Some instructors are moving towards invigilated exams and assessments, and these may be worth more marks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Tomaschitz was concerned that the ChatGPT group was over-reliant on the tool and did not understand the output as well as the e-book group.

    “Tomaschitz was concerned that the ChatGPT group was over-reliant on the tool and did not understand the output as well as the e-book group.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingnone given

    In a high school introductory programming course, Christian Tomaschitz had half of his students use ChatGPT and the other half use an e-book; the ChatGPT students were faster solving problems but seemed not to critically reflect on the output and understood less.

    “In a high school introductory programming course, Christian Tomaschitz (Theresianum Eisenstadt, Austria) had half of his students use ChatGPT and the other half use an e-book.ChatGPT was not introduced to students and students registered OpenAI accounts with their school e-mail addresses.All students had the same exam which was half open-book (including internet access), half closed-book.Tomaschit”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Jean Mehta plans to use 20-30 minute one-on-one oral exams at the end of every section.

    “Jean Mehta plans to use 20-30 minute one-on-one oral exams at the end of every section.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Most of the discussion around unsupervised activities is centred around ensuring that students are doing their own work and that students understand the code that these tools produce.

    “Most of the discussion around unsupervised activities is centred around ensuring that students are doing their own work (even if using Generative AI as a tool for help) and that students understand the code that these tools produce.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    We have yet to arrive at a steady state in terms of institutional initiatives.

    “Given that institutions are typically slow to react mid-year-if they react at all, we have yet to arrive at a steady state in terms of institutional initiatives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Porter's institutional challenge is finding and organising enough computer-based testing facilities.

    “However Porter's institutional challenge is finding and organising enough computer-based testing facilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Several interviewees including Leo Porter and Michael Caspersen believe that LLMs will cause educators to focus more on code reading and less on writing from scratch.

    “Several interviewees including Leo Porter and Michael Caspersen believe that LLMs will cause educators to focus more on code reading and less on writing from scratch.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several interviewees stressed the need for educators to emphasise the importance of students doing their work.

    “Several interviewees were aware of this, stressing the need for educators to emphasise the importance of students doing their work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Students can now execute code that they do not understand, designed through natural language prompts.

    “Potentially more dangerous is the fact that students can now execute code that they do not understand, yet designed through natural language prompts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The problems where any model struggled were either complex or had vague problem descriptions, leading to some edge cases being ignored.

    “The problems where any model struggled were either complex (e.g., printing bar graphs as text) or had vague problem descriptions, leading to some edge cases being ignored (e.g., rainfall variants that did not specify what should happen when the list is empty).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the APPS dataset, relatively few problems explicitly reference object-oriented topics (4.1%) or common data structures (6.6%).

    “At the same time, relatively few explicitly reference OO topics (4.1%) or common data structures (6.6%).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [95]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A review of Table 7 suggests a number of limitations in the data available for pursuing LLM research in an educational context.

    “A review of Table 7 suggests a number of limitations in the data available for pursuing LLM research in an educational context.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Table 7

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Copilot failed T1-Q11 because all ten completions had logical flaws and did not pass all the tests.

    “While Copilot took the constraints into account, all ten completions had logical flaws and did not pass all the tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    GPT-4 outperformed GPT-3.5 and Copilot, corroborating previous work where GPT-4 has outperformed other LLMs for various tasks.

    “Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [21]

      Not checked in the free run.

    • cites [151]

      Not checked in the free run.

    • cites [175] source

      Not checked in the free run.

    • cites [184]

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Almost all problems in the analysed datasets were small exercises, with relatively few requiring multiple functions to solve.

    “In addition, while reviewing the problems, we found that almost all were examples of small exercises, with relatively few requiring multiple functions to solve.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We found evaluation to be limited, with some datasets requiring manual intervention to complete evaluation.

    “We also found evaluation to be limited, with some datasets requiring manual intervention to complete evaluation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under a strict pass/fail criterion, 36.1% of GPT-4 solutions pass all tests, as do 21.0% of GPT-3.5 solutions.

    “Keeping a strict pass/fail criterion (all tests should pass), only 36.1% of the GPT-4 solutions pass all tests, as do 21.0% of GPT-3.5 solutions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 solutions to introductory APPS problems scored as high as a 72.2% test case average, while competition level problems scored only 28.7%.

    “We also observe a large difference between problem types, with GPT-4 solutions to introductory problems scoring as high as a 72.2% test case average, but the most difficult, competition level problems scoring only 28.7%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Copilot performed the worst out of the three evaluated models, successfully solving 20 out of the 23 exam problems and four out of the seven rainfall variants.

    “Copilot performed the worst out of the three evaluated models, successfully solving 20 out of the 23 exam problems and four out of the seven rainfall variants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the Finnie-Ansley et al. [86] dataset, the majority of content was primarily suitable for an introductory course, with relatively few questions asking about object-oriented code or data structures.

    “Again, we found that the majority of the content was primarily suitable for an introductory course, with relatively few questions asking about object-oriented code or data structures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [86] source

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Relatively few datasets contain chat logs where students interact with an LLM or student-submitted code with syntax errors.

    “In contrast, relatively few datasets contain chat logs where students interact with an LLM or student-submitted code with syntax errors (for code repair tasks), but additional publicly available data for questions beyond code-generation would be beneficial, as that would allow researchers operating in smaller educational settings, where collecting sufficient data may be difficult, to engage in LLM”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Table 7

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingcitation

    LLMs struggle with trivial formatting issues, such as missing punctuation or producing extra output not specified by the problem description.

    “As noted in the original study, LLMs struggle with trivial formatting issues, such as missing punctuation or producing extra output that is not specified by the problem description.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A common thread of work to date has been to evaluate the performance of various models.

    “a common thread of work to date has been to evaluate the performance of various models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    It is unclear how students are being educated about the ethical use of generative AI in the classroom.

    “However, it is unclear how students are being educated about the ethical use of generative AI in the classroom.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The pdf file for the drone simulation project was not released with the dataset, so the LLM could not be provided adequate instructions to generate a solution; this was the case for all 9 projects.

    “Since the pdf file has not been released with the dataset we could not provide the LLM with adequate instructions to generate a solution.This is the case for all 9 projects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There was a trend towards placing a greater emphasis on invigilated assessments such as exams, with a reduction to the grade weighting placed on unsupervised homework assignments.

    “In addition, there was also a trend towards placing a greater emphasis on invigilated assessments such as exams, with a reduction to the grade weighting placed on unsupervised homework assignments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A substantial number of Skill assignments required an external data file, and even more commonly starter code was needed to complete the assignment.

    “There are a substantial number of situations where the Skill assignment required an external data file, and even more commonly starter code was needed to complete the assignment”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There were instances where the LLM used an unexpected library (not part of the Python standard library), causing the program to crash and unit tests to fail.

    “We also detected instances where the LLM used an unexpected library (not part of the Python standard library) and hence the program would crash, i.e., the unit tests would fail.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some instructors were concerned that their students were using generative AI tools inappropriately, and a small fraction of students refused to use generative AI tools for ethical reasons and due to concerns about harming their learning.

    “Some instructors were concerned that their students were using such tools inappropriately, and a small fraction of students refused to use generative AI tools for ethical reasons and due to concerns about harming their learning.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Many LLMs are easily available and free for students, and early reports reveal that students are already using them for assistance on their assignments.

    “Many of these models are easily available and free for students, and early reports reveal that students are already using them for assistance on their assignments [44].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Leinonen et al. found that Codex could enhance programming error messages approximately 54% of the time on average, noting that this performance is not good enough for using the model directly with students.

    “Leinonen et al. found that Codex could enhance programming error messages-which are notoriously hard for students to understand-approximately 54% of the time on average, noting that this performance is not good enough for using the model directly with students [125].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [125] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Using code generator tools did not seem to introduce new security risks.

    “Sandoval et al. found that using code generator tools did not seem to introduce new security risks [173].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [173]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Most students found prompt writing to be beneficial, while a few voiced concerns.

    “Most students found the prompt writing to be beneficial, while a few voiced concerns.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [71] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Bull and Kharrufa argue that the type of scaffolding AI tools can provide can support the student in their learning and offload some burden from the educator.

    “Bull and Kharrufa argue that the type of scaffolding that AI tools can provide can”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Student written prompts were found to be an effective benchmark for evaluating LLMs.

    “Another study (which we classified as assessing LLMs, i.e. category (i)) used student written prompts in order to evaluate LLMs and found it to be an effective benchmark [29].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.

    “They found that students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [108] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students generally found code explanations generated by LLMs useful for learning, but perceived utility and engagement varied by explanation type.

    “MacNeil et al. found that students generally found code explanations generated by LLMs useful for learning, but the perceived utility of the explanations and students' engagement with them varied by explanation type [137].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [137] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The exercises generated by LLMs were novel and sensible, although the authors cautioned that the quality might not be good enough to provide the LLM-generated exercises directly to students.

    “In addition, the exercises generated by LLMs were novel and sensible, although the authors cautioned that the quality might not be good enough to provide the LLM-generated exercises directly to students.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [75]

      Not checked in the free run.

    • cites [174] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Generative AI models can provide wrong or biased answers, and students may become over-reliant on LLM tools or generate code plagiarised from online sources by the model.

    “Furthermore, generative AI models can provide wrong or biased answers, and students may also become over-reliant on LLM tools or generate code plagiarised from online sources by the model [33].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    One study reports experiences of five students using generative AI for assignments, highlighting both effective support and many limitations.

    “One study reports on the experiences of five students using generative AI for assignments [89], highlighting both aspects of where it offered effective support, but also many limitations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [89] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The models might generate code students do not understand or may distract them with large blocks of text they did not write.

    “The models might generate code students do not understand [108] or may distract them with large blocks of text they did not write [159].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [108] source

      Not checked in the free run.

    • cites [159] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    All papers suggest that using LLMs could help instructors save time in generating teaching materials.

    “All papers suggest that using LLMs could help instructors save time in generating teaching materials.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The tools do not require training to use, but developers will need to understand the generated code for quality assurance, and to avoid over-reliance as the quality of code produced by these tools can vary.

    “They note that while the tools do not require training to use, developers will need to understand the generated code for quality assurance, and to avoid over-reliance as the quality of code produced by these tools can vary.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLM-generated content quality is comparable to student-generated content.

    “Another study that compared LLM-generated content with student-generated content concluded that the quality is comparable, but still recommends further research [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Tools that assess whether a given text was generated by an LLM show a large number of false positives and should not be trusted blindly.

    “However, tools that assess whether a given text was generated by an LLM show a large number of false positives and should not be trusted blindly [144].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [144]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Three papers report interviews with students and instructors finding no consensus about the use of LLMs in higher education, its benefits or risks, but general awareness of the problem of academic integrity.

    “Three other papers report on interviews with students and instructors about their experiences with, and attitudes towards LLMs [121,164,198], finding that there is no consensus about the use of LLMs in higher education, its benefits or risks, but there is general awareness of the problem of academic integrity in the light of LLMs being used by students.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [121] source

      Not checked in the free run.

    • cites [164] source

      Not checked in the free run.

    • cites [198]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    One paper presents a tool to help students with issues without revealing the full solution and reports positive feedback from both students and instructors.

    “One paper presents a tool to help students with issues without revealing the full solution [127] and reports positive feedback from both students and instructors.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [127] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Phung et al. found that in the best case, their proposed method can achieve a precision of 72.4% with a coverage of 64.2% for one dataset and a precision of 76% and a coverage of 31.2% for the other dataset.

    “They found that in the best case, their proposed method can achieve a precision of 72.4% with a coverage of 64.2% for one of the datasets they used and a precision of 76% and a coverage of 31.2% for the other dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    LLMs will have a profound impact on computing education in the next decade as the technology matures and as teachers and researchers identify opportunities.

    “Large language models will have a profound impact on computing education in the next decade as the technology matures and as teachers and researchers identify opportunities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    LLMs present clear opportunities for instructors to rethink assessment practices and reconsider what assessment means in computing courses.

    “Finally, LLMs present clear opportunities for instructors to rethink assessment practices and reconsider what assessment means in computing courses [39,70,153,176].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [39]

      Not checked in the free run.

    • cites [70] source

      Not checked in the free run.

    • cites [153]

      Not checked in the free run.

    • cites [176]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    LLMs will change how, what, and whom we teach not only in computing but in all of education.

    “LLMs will change how, what, and whom we teach not only in computing but in all of education [68].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Data extraction was guided by a set of questions implemented as an online form to help standardise the process.

    “Data extraction was guided by a set of questions implemented as an online form to help standardise the process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Example programs used by students for learning could have poor implementation or poor style, which may result in students acquiring undesirable programming habits.

    “Example programs used by students for learning could have poor implementation or poor style, which may result in students acquiring undesirable programming habits [85].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The literature review used a scoping review methodology.

    “We chose to perform a scoping review to rapidly identify gaps and major themes in the literature”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Leinonen et al. suggest that LLMs could help students understand terse error messages which have traditionally been a source of difficulty for novice learners.

    “Leinonen et al. suggest that LLMs could help students understand terse error messages [125] which have traditionally been a source of difficulty for novice learners [36].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [125] source

      Not checked in the free run.

    • cites [36] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Two separate phases of forward snowballing were conducted, one in May 2023 and one in August 2023.

    “We conducted two separate phases of forward snowballing, one in May 2023 and one in August 2023”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Category (i) being the largest group indicates a strong desire to assess the current capabilities and limitations of LLMs in computing education contexts.

    “Category (i) is by far the largest group, indicating a strong desire to assess the current capabilities and limitations of LLMs in computing education contexts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Several papers described the idea of using an LLM as an assistant or pair programmer, which represents a significant change from current pedagogical practice.

    “Several papers described the idea of using an LLM as an assistant or pair programmer, which represents a significant change from current pedagogical practice [123,152,169].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [123] source

      Not checked in the free run.

    • cites [152]

      Not checked in the free run.

    • cites [169]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Dakhel et al. and Prather et al. caution that although AI tools can be a great asset for professional developers, they may be less helpful for novices if the tools generate non-optimal or erroneous outputs which could cause confusion.

    “Despite the promise of using AI tools for learning support, Dakhel et al. and Prather et al. caution that although they can be a great asset for professional developers, they may be less helpful for novices if the tools generate non-optimal or erroneous outputs which could cause confusion [61,159].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [61]

      Not checked in the free run.

    • cites [159] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Resources created by LLMs might have less variation than those created by humans, limiting the variety of examples to which students are exposed.

    “The resources created by LLMs might also have less variation than those created by humans [69] and thus limit the variety of examples to which students are exposed.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    As the quality and performance of the models improve, the issue of non-optimal or erroneous outputs may be less of an issue as time goes by.

    “As the quality and performance of the models improve, this may be less of an issue as time goes by.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    AI-generated content is not curated for specific courses, so generated learning material could include syntax, programming constructs, or other content inappropriate for students in a given course.

    “AI generated content is not curated for specific courses, so learning material generated could potentially include syntax, programming constructs, or other content that is inappropriate for students in a given course [33,94].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [33] source

      Not checked in the free run.

    • cites [94] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    There is little doubt that LLMs and other forms of GenAI will have a profound impact on computing education over the coming years.

    “There is little doubt that LLMs and other forms of GenAI will have a profound impact on computing education over the coming years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All evaluators were instructed to interpret the criteria generously to avoid exclusion of potentially relevant work.

    “All evaluators were instructed to interpret the criteria generously, to avoid exclusion of potentially relevant work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    AI-generated content is presented authoritatively and is frequently correct, so students are unlikely to question it and may learn incorrect information.

    “Since AI generated content is presented authoritatively (and is frequently correct), students are unlikely to question the content and may learn incorrect information [52,103].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [52]

      Not checked in the free run.

    • cites [103]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationreasoning

    The papers reviewed suggest that LLMs present a wide array of opportunities for computing education, including improving instructor productivity by reducing workload, enhancing student learning experiences, enabling a greater emphasis on problem-solving, and suggesting new assessment practices.

    “In summary, the papers we reviewed suggest that LLMs present a wide array of opportunities for computing education such as improving instructor productivity by reducing workload, enhancing student learning experiences, enabling a greater emphasis on problem-solving, and suggesting new assessment practices.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    The fact that Python emerges as the most popular language is not too surprising because popular LLMs such as Codex have been reported to be most proficient in Python.

    “The fact that Python emerges as the most popular language in this work is not too surprising, however, as popular LLMs such as Codex have been reported to be most proficient in Python [56].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [56]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The quality metrics from Hellas et al. were selected and updated to focus on whether threats to validity were discussed in an explicit subsection.

    “We selected the most generally applicable questions, and we updated their question about threats to validity to focus specifically on whether they were discussed in an explicit subsection.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [92]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The review used one step of forward and backward snowballing from a set of reference papers.

    “We chose to perform one step of forward and backward snowballing [194] from a set of reference papers”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [194]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs can be used to generate a variety of flawed solutions, providing opportunities for incorporating code review tasks.

    “LLMs can also be used to generate a variety of flawed solutions, providing plentiful opportunities for incorporating code review tasks [85].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    Line-by-line explanations are akin to a multi-structural explanation, which may focus student attention at lower levels of the SOLO taxonomy rather than thinking about the overall purpose of code.

    “This is akin to a multi-structural explanation [180], which may focus student attention at lower levels of the SOLO taxonomy, rather than thinking about the overall purpose of code.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [180]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    AI code generators may widen the gap between over- and under-achievers, exacerbating teaching challenges arising from classes with heterogeneous ability levels.

    “These findings suggest that AI code generators may widen the gap between over-and under-achievers, exacerbating teaching challenges arising from classes with heterogeneous ability levels.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    There is a growing need to teach students how to use generative AI appropriately, but it is unclear how to do so.

    “Although there is a growing need to teach students how to use generative AI appropriately, it is unclear how we should do so.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    As more code is likely to be generated automatically, there is potential for biases to be unintentionally introduced due to inherent bias in models and because of bias in the training data.

    “As more code is likely to be generated automatically, there is potential for biases to be unintentionally introduced due to inherent bias in models and because of bias in the training data [33].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    Generative AI is reported to perform very well in assessments commonly used in introductory courses, raising concerns that students will submit solutions they have not created themselves.

    “Generative AI is reported to perform very well in assessments that are commonly used in introductory courses, raising concerns that students will submit solutions that they have not created themselves [33,70,85,86,121,152,169,176,177].”
    Quote in the paperSubstantial claim0 of 9 sources checked
    • cites [33] source

      Not checked in the free run.

    • cites [70] source

      Not checked in the free run.

    • cites [85] source

      Not checked in the free run.

    • cites [86] source

      Not checked in the free run.

    • cites [121] source

      Not checked in the free run.

    • cites [152]

      Not checked in the free run.

    • cites [169]

      Not checked in the free run.

    • cites [176]

      Not checked in the free run.

    • cites [177]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    During deep review, a few concerns about quality of a given paper arose; these papers were retained as they met inclusion criteria, and on the whole the papers appear ready for review and demonstrate many criteria for quality proposed by Hellas et al.

    “During our deep review of the included papers, there were a few concerns about the quality of a given paper.We retained these papers as they met the inclusion criteria, and note that, on the whole, the papers appear to be ready for review and demonstrate many of the criteria for quality proposed by Hellas et al. [92].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [92]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These acceptable uses are situations where the tool could save time without negatively affecting learning outcomes.

    “These are situations where the tool could save time without negatively affecting learning outcomes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Teachers concerned about the impact of generative AI may intentionally modify course delivery in ways that reduce the effectiveness of their teaching practice, or adjust the curriculum to shift focus away from code writing, leaving students poorly prepared for subsequent programming courses.

    “Teachers who are concerned about the impact of generative AI may intentionally modify course delivery in ways that reduce the effectiveness of their teaching practice (e.g., by increasing academic integrity at the cost of scaffolded programming exercises), or adjust the curriculum to shift focus away from code writing, leaving students poorly prepared for subsequent programming courses [165].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [165]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    Solutions generated by AI might not be easily identified as AI-generated, which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained.

    “The solutions generated by AI might not be easily identified as AI-generated [160], which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained [191].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [160]

      Not checked in the free run.

    • cites [191] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims. The quote only partly states this claim.

    Study design and conductnone given

    A single round of snowballing was conducted, which may have omitted some work that failed to reference the seed papers, but the authors do not believe this will include significant numbers of papers or change the general trends and themes identified.

    “For this reason we conducted a single round of snowballing (i.e., we did not do subsequent snowballing on the papers found in the first round of snowballing).This may have omitted some work that failed to reference the most visible early work (our”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Study design and conductreasoning

    The literature review was conducted from April to August 2023, so work published subsequently is not included.

    “Our literature review was conducted from April to August 2023.Therefore work published subsequently is not included.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Students who rely on generative AI may be misled into believing they are making progress, and this illusion of capability may reduce their self-understanding about their level of mastery of the subject matter.

    “Students who rely on generative AI may be misled into believing they are making progress, and this illusion of capability may reduce their self-understanding about their level of mastery of the subject matter [158,159].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [158]

      Not checked in the free run.

    • cites [159] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationcitation

    Generated code may contain security vulnerabilities and bugs.

    “Generated code may contain security vulnerabilities and bugs [33,52,173].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [33] source

      Not checked in the free run.

    • cites [52]

      Not checked in the free run.

    • cites [173]

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationown data

    This points to an emergent interest in GenAI from instructors in our survey and recognition of the use of GenAI for teaching.

    “This points to an emergent interest in GenAI from instructors in our survey and recognition of the use of GenAI for teaching.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    As introductory students realise that generative AI can outperform them on most tasks, they may lose motivation to learn the material and become demoralised about the future of computing.

    “As introductory students realise that generative AI can outperform them on most tasks, they may lose motivation to learn the material, and become demoralised about the future of computing [123].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [123] source

      Not checked in the free run.

    • Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

  • May combine two claims.

    Interpretationown data

    Instructor responses reveal potential for GenAI to aid student learning but also highlight a concerning trend of academic misconduct and over-reliance.

    “While instructor responses to this question do reveal the potential for GenAI tools to be used to aid student learning, they also highlight a concerning trend of academic misconduct and over-reliance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills in order to become proficient computing experts and to use LLMs effectively.

    “Although LLM-based tools like Copilot may serve as a useful springboard for solving CS1 problems, students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills [191] in order to become not only proficient computing experts but also to use LLMs effectively.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [191] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Perceptions and experiences of both students and educators vary, often focusing on difficulty; LLMs may impact this long-standing issue.

    “Additionally, perceptions and experiences of both students and educators also vary, often focusing on difficulty [35,132] and LLMs may stand to impact this long-standing issue.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [35]

      Not checked in the free run.

    • cites [132]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two authors reviewed both verbatim interview transcripts and identified themes for all interviews.

    “Two of the authors reviewed both the verbatim interview transcripts and the identified themes for all interviews.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs are available on nearly everyone's phone and laptop; they make knowledge, problem explanations, and solutions instantly retrievable, but code may not be correct.

    “With LLMs available on nearly everyone's phone and laptop 4 , it is not only knowledge that is instantly retrievable but also problem explanations and solutions -in the form of programming code that is not necessarily correct.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    The interviews were fully transcribed verbatim and served as a basis for thematic analysis.

    “The interviews, which reflect practices from the teaching community in this new context of Generative AI were fully transcribed verbatim and served as a basis for thematic analysis [47,170].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [47] source

      Not checked in the free run.

    • cites [170] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Educational systems are notoriously slow to change.

    “Given the myriad impacts of LLMs, it is important to acknowledge that educational systems are notoriously slow to change.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [182]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Interviews were conducted via Zoom and automatically transcribed via speech recognition software, followed by a human correction loop where interviewers checked transcripts of other reviewers, not their own.

    “They were conducted via Zoom and automatically transcribed via speech recognition software, followed by a correction loop by a human (interviewers checked transcripts of other reviewers, not their own).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Audio and video recordings were deleted after transcripts were finalised, in accordance with a protocol approved by the University of Toronto Research Ethics Board prior to the study.

    “After the transcripts were finalised, we deleted all audio and video recordings in accordance with the protocol submitted to the University of Toronto Research Ethics Board, who approved it prior to the study.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The most important inclusion criterion was that educators would have concrete plans or views toward changing their current course structure, assessment, or classroom practices in light of LLMs.

    “The most important criterion for inclusion was that educators would have concrete plans or views toward changing their current course structure, assessment, or classroom practices in light of LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Participants were allowed to review a draft manuscript before final publication to ensure they are not misrepresented.

    “We also allowed participants to review a draft manuscript before final publication, to ensure that they are not misrepresented.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    It is inevitable that educators consider LLM impact on teaching, learning, assessment, and delivery, leading to possible redesign of courses at all levels.

    “Now, however, with the seeming ubiquity of LLMs, it is inevitable that educators consider their impact on teaching, learning, assessment, and delivery, leading to possible redesign of their courses at all of these levels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    For students in non-computing majors who currently only take one or a few programming courses to learn enough to write simple programs, using an LLM tool may be all that is needed, making a programming-specific course unnecessary.

    “For students in non-computing majors who currently only take one (or just a few) programming course(s) to learn enough to write simple programs, it may be that using an LLM tool is all that is needed, making a programmingspecific course unnecessary [152].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [152]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Themes were discussed and specified among four authors.

    “Next, themes were discussed and specified among four of the authors.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Students still need to learn to code and practice.

    “Thus, students still need to learn to code and practice.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Based on Constructive Alignment [42], learning objectives need to be aligned with exercises, assignments, and assessment methods.

    “Based on the concept of Constructive Alignment [42], learning objectives need to be aligned with exercises, assignments, and assessment methods.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The last category assumes that, in general, the use of GenAI tools is unethical unless it is out of necessity.

    “The last category above is interesting as it assumes that, in general, the use of GenAI tools is unethical unless it is out of necessity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The paradigm shift from LLMs is more comparable to the advent of the internet than other developments, but internet adoption was slower.

    “Given the pervasiveness of LLMs, this paradigm shift regarding the availability of knowledge, solutions, examples, and content (particularly in the form of code) is more comparable with the advent of the internet than other developments in the annals of how we teach and learn computing-yet the speed of internet adoption was much slower as a whole.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The sample comprises 22 computing educators from nine countries and five continents.

    “The sample comprises 22 computing educators from nine countries and five continents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Interview summaries were compiled based on themes and double-checked against themes and transcripts by a second author.

    “Interview summaries were then compiled based on these themes and double-checked against the themes and transcripts by a second author.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    As students increasingly use LLMs to write programs, instructors might need to put less emphasis on writing code and more emphasis on reading code.

    “For instance, as students increasingly use LLMs to write programs, instructors might need to put less emphasis on writing code and more emphasis on reading code.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Students will increasingly use LLMs to write programs.

  • May combine two claims.

    Premisecitation

    Although some institutions have universally banned the use of LLM tools in student work, others are starting to embrace them.

    “Although some institutions (at least initially) have universally banned the use of LLM tools in student work [142], others are starting to embrace them.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [142]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    LLMs can have knock-on effects that could affect not only the course in question but others via resource allocation and timetabling.

    “This is one example of how LLMs can have knock-on effects that could not only affect the course in question but others via resource allocation and timetabling.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Getting students to recognise when it is a shortcut versus when it actually is not a shortcut is something that novices typically are not in a position to assess.

    “Getting students to recognise when it is a shortcut versus when it actually is not a shortcut is something that novices typically are not in a position to assess.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students rated the explanations on a 5-point Likert scale, confirming that explanations matched the code and were useful for learning in general, whereas line-by-line explanations were rated as least helpful.

    “Students rated these explanations on a 5-point Likert scale, confirming that explanations matched the code and were useful for learning in general, whereas line-by-line explanations were rated as least helpful.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [137] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The knock-on effects of LLM use also need to be considered.

    “The knock-on effects of LLM use also need to be considered.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    LLMs can ease the burden often felt by teaching assistants and could be a first line of help for students.

    “This usecase exemplifies the potential for LLMs to ease the burden often felt by teaching assistants, and could be a first line of help for students [174].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [174] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Research has demonstrated that LLMs can answer typical CS1 assessments that involve writing simple functions, perform in the upper quartile of real students on CS1 exams, perform just as well on CS2 exams as CS1 exams, and solve most exercises in introductory programming courses.

    “Research on LLMs has demonstrated their ability to answer typical CS1 assessments that involve writing simple functions [70,117] as well as more advanced material.Similarly, it has been shown that they perform in the upper quartile of real students on CS1 exams [85].It has also been shown that they perform just as well on CS2 exams as CS1 exams [86] suggesting that LLMs may soon be capable of”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [70] source

      Not checked in the free run.

    • cites [117]

      Not checked in the free run.

    • cites [85] source

      Not checked in the free run.

    • cites [86] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    There are other hidden learning objectives spanning knowledge, skills, dispositions, inter- and intra-personal competencies, and other aspects relating to the whole person.

    “There is little doubt that there are other such hidden learning objectives spanning not only knowledge, and skills, such as reading and tracing code but also dispositions, inter-and intra-personal competencies, and other aspects relating to the whole person [163] which the emergence of LLMs might bring to the fore.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [163]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This approach shifts focus from memorising knowledge to the application of skill and critical thinking.

    “This approach also shifts focus from memorising knowledge (or in a rote manner, the process that leads to the product without questioning the process) to the application of skill and critical thinking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    An exploration of the potential of LLMs to generate formative programming feedback suggests that ChatGPT performs reasonably well in generating feedback to students' solutions to introductory programming tasks, indicating that students can potentially benefit in unsupervised learning scenarios.

    “An exploration of the potential of LLMs to generate formative programming feedback [115] suggests that ChatGPT performs reasonably well in generating feedback to students' solutions to introductory programming tasks, indicating that students can potentially benefit in unsupervised learning scenarios.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [115]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Denny et al. found that Copilot's performance is substantially improved when it is prompted with individual problem-solving steps in natural language, and they explicitly encourage teaching prompt engineering to students.

    “Denny et al. [70] found that Copilot's performance is substantially improved when it is prompted with individual problem-solving steps in natural language and they explicitly encourage teaching prompt engineering to students.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [70] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    While LLMs can often correctly answer more than half of coding-based multiple-choice problems, they answer double-digit percentages of multiple-choice problems incorrectly.

    “Similarly, while LLMs can often correctly answer more than half of coding-based multiple-choice problems, they answer double-digit percentages of multiple-choice problems incorrectly, leading to the hypothesis that either a combination of natural language and a code snippet, and/or chain-of-reasoning steps pose a challenge for LLMs [177].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [177]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    There may be less summative unsupervised work because instructors no longer trust that unsupervised work is the students' own.

    “There may be less summative unsupervised work because instructors no longer trust that unsupervised work is the students' own.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    It may fall on educators to guide how to use generated feedback, as it can contain misleading information for novices.

    “However, it may fall on educators to guide how to use the generated feedback, as it can contain misleading information for novices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Harm to institutions can come from students generating incorrect, offensive, or inappropriate work, dragging the university into potential altercation.

    “Harm to institutions can come in the form of students using LLM tools to generate work that is incorrect, offensive, or otherwise inappropriate and therefore dragging the university into potential altercation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The author is fully responsible for the accuracy of any information provided by the generative AI tool and for correctly referencing any supporting work on which that information depends.

    “The author is fully responsible for the accuracy of any information provided by the [generative AI] tool and for correctly referencing any supporting work on which that information depends.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Authors are accountable for the accuracy, integrity and originality of their research papers, including for any use of AI.

    “Authors are accountable for the accuracy, integrity and originality of their research papers, including for any use of AI.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Simon et al. [183] highlights the importance of educating students about academic integrity and reveals a wide variety of ways that academic integrity is communicated to students.

    “Simon et al. [183] highlights the importance of educating students about academic integrity and reveals a wide variety of ways that academic integrity is communicated to students.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [183]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Harm to others could come from using the work of others without citation.

    “Harm to others could come in the form of using the work of others without citation [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Using a tool to produce required output circumvents learning and deprives students of the opportunity to learn.

    “Using a tool to produce the required output circumvents that learning, and deprives students from the opportunity to learn [100].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [100]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Authors are fully responsible for the content of their manuscript, including parts produced by an AI tool, and are liable for any breach of publication ethics.

    “Authors are fully responsible for the content of their manuscript, even those parts produced by an AI tool, and are thus liable for any breach of publication ethics.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [143]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The United States forms 55% of responses.

    “As expected the United States forms the bulk (55%) of responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Algorithmic systems tend to be large, complex and highly modularised, making it difficult to assign responsibilities.

    “Moreover, Mittelstadt et al. [140] point out that algorithmic systems tend to be large, complex and highly modularised, which makes it difficult in general to assign responsibilities.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [140] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Semi-structured interviews can impose interviewer bias.

    “Additionally, the interviews were semi-structured and although this is a common approach it can impose interviewer bias, although it also designed to result in a coherent set of interviews which focus on common topics while still allowing interviewers to express their own views and experiences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Using a large language model has been shown to affect the user's own opinions.

    “Furthermore, using a large language model has been shown to affect the user's own opinions [102].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [102] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors did not verify interviewee claims about their classes, departments, or institutions.

    “Finally, we did not attempt to verify claims that interviewees made about their classes, departments, or institutions, taking interviewee statements at face value.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation

    Generative AI and AI-assisted technologies should only be used to improve readability and language of the work, with human oversight and control, and authors are ultimately responsible for the contents.

    “Where authors use generative AI and AI-assisted technologies in the writing process, these technologies should only be used to improve readability and language of the work.Applying the technology should be done with human oversight and control and authors should carefully review and edit the result, because AI can generate authoritative-sounding output that can be incorrect, incomplete or biased.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Use of LLMs can lead to poor outcomes due to overreliance, ease of breaching academic integrity, and use of incorrect information.

    “Use of LLMs can lead to poor outcomes due to overreliance, ease of breaching academic integrity, and use of incorrect information leading to poor products.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The interview pool had no interviewees from Africa and only one each from Asia, Oceania, and South America.

    “Although our interview pool spanned five continents, we had no interviewees from Africa and only one interviewee each from Asia (India), Oceania (New Zealand), and South America (Brazil).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Students who utilise generative AI inappropriately in lower-level courses may cause harm to themselves by preventing their own preparation for upper-level coursework.

    “Not only could this violate academic integrity, but as discussed above, students who utilise these resources inappropriately in lower-level courses may cause harm to themselves by preventing their own preparation for upper-level coursework.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In 100 highly cited papers, performance was the most frequent value.

    “In a total of 100 highly cited papers, the study found that performance was clearly the most frequent value and pointed out that performance is not a neutral term but comes with ethical implications.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Birhane et al. identified ethical values in ML research literature.

    “In a study on the values encoded in the ML research literature, Birhane et al. identified a number of ethical values [45], from which the papers would draw motivation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The authors' sampling of university policies is not a systematic attempt.

    “We do not consider this a systematic attempt, nor would that be presently possible, given the conditions described above.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    In upper-level courses, generative AI may be an appropriate productivity tool that students would be permitted to use, as corroborated by survey data in Figure 3.

    “However, in upper-level courses, it may be an appropriate productivity tool that students would be permitted to use, which the survey data in Figure 3 seems to corroborate given responses to questions about generating pieces of an assignment, help with style, or fixing bugs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Figure 3

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Little is known about the true role of notional machines in the classroom.

    “It is also interesting, given how little is known about the true role of notional machines in the classroom [76], that generative AI is likely to alter how students conceptualise the computer and program execution.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Inappropriate LLM use can call into question the legitimacy of learning outcomes and verification, jeopardising accreditation and institutional reputation.

    “It can also call into question the legitimacy of the learning outcomes and verification of them, which can jeopardise accreditation and institutional reputation or the community's trust in that institution.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Taken together, many available datasets and published results reflect an early introductory programming context (CS1) with relatively few examples of common CS2 material and even fewer covering more advanced topics.

    “Taken together, this analysis suggests that many of the available datasets-and as a result, the published results-reflect an early introductory programming context (CS1) with relatively few examples of common CS2 material (Cipriano and Alves [57] is a recent exception demonstrating research on OO topics) and even fewer covering any more advanced topics.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The FalconCode dataset contains 661 introductory Python programming problems used in 4 courses.

    “The dataset contains 661 introductory Python programming problems used in 4 courses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We performed a thematic analysis in which causes of each failed assignment were extracted as codes and then collated into higher-level themes.

    “Specifically, we performed a thematic analysis in which causes of each failed assignment were extracted as codes and then collated into higherlevel themes [47].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [47] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The Finnie-Ansley et al. [86] dataset was tagged by two authors, and Cohen's kappa showed near perfect agreement (0.94).

    “This time, the dataset was tagged by two authors (one who had tagged the previously discussed datasets and a second experienced instructor); we calculated Cohen's kappa and found near perfect agreement (0.94).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [86] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    The less variation in Copilot completions might be due to the use of a relatively high temperature value for GPT-4 and GPT-3.5, while Copilot likely uses a lower temperature value.

    “This might be due to the use of a relatively high temperature value for these models-previous work found similar results and speculated that Copilot likely uses a lower temperature value [191].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [191] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming, Socratic questioning of novices, solving multiple-choice questions and programming exercises, and that performance can be close to human tutors for some tasks while not for others.

    “Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [98]

      Not checked in the free run.

    • cites [184]

      Not checked in the free run.

    • cites [21]

      Not checked in the free run.

    • cites [175] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Plagiarism is not usually a concern with LLM-generated text; instead, users would be responsible for any falsification produced from uncritical use of LLM-generated artefacts.

    “This has implications for academic integrity, in that plagiarism is not usually a concern but instead users would be responsible for any falsification produced from uncritical use of LLMgenerated artefacts.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The user of the LLM should be considered the author of the generated text.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    LLMs present significant challenges but also new opportunities for computing educators.

    “it is clear that LLMs present significant challenges but also new opportunities for computing educators.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Study design and conductown data

    The authors used an analysis of GPT-4 failure reasons to filter the FalconCode dataset down to a clean version appropriate for evaluation.

    “We used this analysis to filter the dataset down to a clean version that is appropriate to use for the evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Internet access is required to access LLMs.

    “Acknowledging that internet access is required to access LLMs”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    To assess model performance on multiple types of tasks, for different programming languages, or at different levels, will require new datasets.

    “To assess model performance on multiple types of tasks, for different programming languages, or at different levels, will require new datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Test cases can be incorrect or too strict, exceeding what is required in the instructions, lowering the potential performance of models.

    “Alternately, test cases can even be incorrect, or too strict, exceeding what is required in the instructions, lowering the potential performance of models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Subscription-based services that could be superior to free ones present issues of access based on means and potentially open new divides.

    “subscriptionbased services which could be superior to free ones present issues of access based on means, and potentially opening new divides”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The literature review inclusion criterion rules out posters and abstracts.

    “This criterion rules out posters and abstracts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing datasets may need to be augmented to evaluate LLM-generated code accurately.

    “As suggested by Liu et al. [129], existing datasets may need to be augmented to evaluate LLMgenerated code accurately.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [129]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Assessing solutions for other types of exercises common in CS (e.g., regular expressions, UML-diagrams, automata), LLM-generated exercises, feedback, and explanations require additional datasets and may require a qualitative framework that could be difficult to provide or to transfer to another research team.

    “Outside of code generation, assessing solutions for other types of exercises common in CS (e.g.regular expressions, UML-diagrams, automata), LLM-generated exercises, feedback, and explanations require additional datasets and may require a qualitative framework that could be difficult to provide or to transfer to another research team.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The authors have not attempted to automatically assess code quality and suitability of solutions in their study.

    “These latter aspects could also be assessed automatically, but we have not attempted to do so in our study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Testing generated code can be easily automated by running test cases, although this might not capture all aspects relevant to computing educators, such as code quality and suitability of the solution with regards to which concepts the student has learned so far.

    “Testing generated code can be easily automated by running test cases, although this might not capture all aspects relevant to computing educators, such as code quality and suitability of the solution with regards to which concepts the student has learned so far.”
    Quote in the paperNot a substantial claim
229 of 229

  • The paper relies on this without support.

    The set of reference papers and the snowballing procedure captured the relevant literature on LLMs in computing education.

    “We chose to perform one step of forward and backward snowballing [194] from a set of reference papers that were identified as being currently significant work in the area of large language models in computing education.”

  • The paper relies on this without support.

    The 71 papers included in the review are representative of the current state of research on LLMs in computing education.

    “These exclusions left us with 71 relevant papers (10 reference papers, 28 papers from the first snowballing phase, and 33 papers from the second).”

  • The paper relies on this without support.

    The survey respondents from 20 countries are representative of computing students and instructors.

    “we report the findings of a survey of computing students and instructors from across 20 countries”

  • The paper relies on this without support.

    The interview participants from five continents are representative of computing educators.

    “we offer insights collected from in-depth interviews with 22 computing educators from five continents.”

  • The paper relies on this without support.

    The benchmark datasets used are appropriate for evaluating GenAI models and tools in computing education.

    “we benchmark the performance of several current GenAI models/tools on various computing education datasets”

  • The paper relies on this without support.

    The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.

    “but we do not believe that this will include significant numbers of papers or change the general trends and themes identified in our literature analysis.”

  • The paper relies on this without support.

    The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.

    “We recruited 57 instructors and 171 students to complete the corresponding online surveys.”

  • The paper relies on this without support.

    The survey respondents are representative of computing educators and students in the relevant populations.

    “The survey revealed that most instructors are not currently using GenAI tools for text or code generation, but some have expressed plans to integrate them in the future.”

  • The paper relies on this without support.

    Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.

    “The findings reveal a consensus among instructors that auto-generating an entire assignment solution is considered unethical when students lack comprehension of the generated code.”

  • The paper relies on this without support.

    The qualitative themes identified from open-ended responses are representative of the broader respondent group.

    “The primary theme to emerge from responses around the current usage of text generation tools was for the creation of a wide variety of learning resources.”

  • The paper relies on this without support.

    Students will increasingly use LLMs to write programs.

    “as students increasingly use LLMs to write programs”

  • The paper relies on this without support.

    The influence of media coverage of computing topics is a large factor in pre-university students' decisions about what courses to pursue at university.

    “Additionally the influence of media coverage of computing topics is known to be a large factor in the decisions pre-university students make in terms of what courses to pursue at university.”

  • The paper relies on this without support.

    ChatGPT offers a range of models with different performance characteristics, not all of which are free.

    “On the other hand, ChatGPT offers a range of models with different performance characteristics, not all of which are free.”

  • The authors state this as an assumption.

    Both parties in collusion have agency.

    “This requires both parties to willingly agree to work together and therefore assumes that both parties have agency.”

  • The paper relies on this without support.

    Many students will be able to fix small mistakes made by LLMs themselves.

    “For instance, we expect that many students will be able to fix small mistakes made by LLMs themselves.”

  • The paper relies on this without support.

    Tasks in existing benchmarks might not match those typically found in computing education courses.

    “Similarly, tasks in existing benchmarks might not match those typically found in computing education courses.”

  • The paper relies on this without support.

    A slight variation in a prompt might generate quite different results.

    “Furthermore, a slight variation in a prompt might generate quite different results.”

  • The paper relies on this without support.

    GPT-4 is the state-of-the-art LLM at the time of writing.

    “At the time of writing, GPT-4 is the state-of-the-art LLM, so it is interesting to see how it performs on solving the problems.”

  • The paper relies on this without support.

    GPT-3.5-turbo is likely what many students will be using.

    “GPT-3.5-turbo is the model that powers the free version of ChatGPT, which is likely what many students will be using, and thus it is also interesting to include.”

  • The paper relies on this without support.

    GitHub Copilot is free for students and educators, and students may be using it.

    “Lastly, GitHub Copilot is free for students and educators, and is directly embedded into the IDE, so it is possible that students will be using it too, which is why we decided to include it in the replication.”

  • The paper relies on this without support.

    The HTML may encode important information through formatting that a state-of-the-art LLM such as GPT-3.5/4 may be capable of leveraging.

    “We decided against extracting plain text of the problem statements because the HTML may encode important information through formatting that a state-of-the-art LLM such as GPT-3.5/4may be capable of leveraging.”

  • The paper relies on this without support.

    It is possible that solutions for the APPS dataset have been included in training recent models.

    “It is possible that solutions for it have been included in training recent models.”

  • The paper relies on this without support.

    Skill assignments are supposed to require only small solutions, not exceeding several lines of code.

    “As Skill assignments are supposed to require only small solutions, not exceeding several lines of code”

  • The paper relies on this without support.

    The APPS dataset problems are significantly more challenging than those typically provided in CS1 and CS2 courses.

    “those problems are significantly more challenging than those typically provided in CS1 and CS2 courses.”

  • The paper relies on this without support.

    The publicly available datasets used are of high quality.

    “even though the datasets themselves are of high quality.”

  • The authors state this as an assumption.

    The user of the LLM should be considered the author of the generated text.

    “We agree with the position articulated by many publishers that the user of the LLM should be considered the author of the generated text.”
26 of 26

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multiple studies showed evidence of bimodal distributions of GenAI usage, with some students using GenAI tools every day and many others never having used these tools.

    “multiple studies showed evidence of bimodal distributions of GenAI usage [146, 279], with some students using GenAI tools every day and many others never having used these tools.”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [279]

    Section 3.4.3 reports that around 40% of surveyed students had not used GenAI for text or code generation in their courses, supporting the claim that many were non-users in that setting. Our copy does not establish that some students used the tools every day or that multiple studies found a bimodal distribution; searches for “every day,” “bimodal,” and usage frequency did not locate those findings. Page numbers were not available in the indexed passages. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • In an introductory programming course, students expressed concerns about privacy when using Github Copilot.

    “In an introductory programming course, students expressed concerns about privacy when using Github Copilot [278].”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]

    Section 4.3 discusses GitHub Copilot in introductory programming, and “Respect privacy” says students may be worried about AI tools taking their intellectual property. Neither passage reports that students in an introductory course expressed privacy concerns specifically when using Copilot; the course poll in Section 4.3.2 instead describes students wanting to learn to code without assistance. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Students risk becoming overly reliant on GenAI tools, which might hinder their independent problem-solving skills.

    “Other authors note the risk of students becoming overly reliant on these tools, which might hinder their independent problem-solving skills [23].”

    From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [23]

  • There is a risk of students becoming overly reliant on AI assistance, which may impede development of fundamental programming skills and critical thinking.

    “There is also the risk of students becoming overly reliant on AI assistance, which may impede the development of fundamental programming skills and critical thinking (Prather et al., 2023).”

    From The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: a comparative study with traditional pair programming and individual approaches (2025) · cites this paper as Prather et al., 2023

  • Prather et al. [278] found differing opinions about the ethics of this practice when students possessed a full understanding of the generated code or wrote the code in a different language and then translated it into the language of instruction.

    “however, there were differing opinions about the ethics of this practice when students possessed a full understanding of the generated code or wrote the code in a different language and then translated it into the language of instruction.”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure 3 does not show students' ethical-use responses

    Minor

    The text draws a conclusion about both instructors' and students' views on ethical use from Figure 3, but the figure caption says its ethical-use responses come from instructors; the student responses shown are help-seeking rankings.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [n. d.]. Authorship and Contributorship. https://www.cambridge.org/core/services/authors/publishing-ethics/research-publishing-ethics-guidelines-for-journals/authorship-and-contributorship#ai-contributions-to-research-content Accessed: 08 July 2023.
  2. [n. d.]. Best Practice Guidelines on Research Integrity and Publishing Ethics. https://authorservices.wiley.com/ethics-guidelines/index.html#5 Accessed: 08 July 2023.
  3. [n. d.]. Publishing Ethics | Elsevier Policy. https://beta.elsevier.com/about/policies-and-standards/publishing-ethics?trial=true#4-duties-of-authors Accessed: 08 July 2023.
  4. [n. d.]. Using ChatGPT or Other Generative AI Tool on a Marked Assessment. https://www.academicintegrity.utoronto.ca/perils-and-pitfalls/using-chatgpt-or-other-ai-tool-on-a-marked-assessment/. Accessed: 06 September 2023.
  5. 2019. AAAI Code of Professional Ethics and Conduct. https://aaai.org/about-aaai/ethics-and-diversity/. Accessed: 08 July 2023.
  6. 2020. IEEE Code of Ethics. https://www.ieee.org/about/corporate/governance/ p7--8.html Accessed: 08 July 2023.
  7. 2023. ACM Code of Ethics. https://www.acm.org/code-of-ethics Accessed: 08 July 2023.
  8. 2023. ACM Policy on Authorship. https://www.acm.org/publications/policies/new-acm-policy-on-authorship Accessed: 08 July 2023.
  9. Mikko Apiola, Sonsoles López-Pernas, and Mohammed Saqr. 2023. Past, Present and Future of Computing Education Research: A Global Perspective. Springer Nature.
  10. Brett A. Becker. 2017. Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going. Ireland's Yearbook of Education 2018 (2017), 42--46.
  11. John Biggs. 1996. Enhancing Teaching Through Constructive Alignment. Higher education 32, 3 (1996), 347--364.
  12. Robert W Brennan and Jonathan Lesage. 2022. Exploring the Implications of Openai Codex on Education for Industry 4.0. In International Workshop on Service Orientation in Holonic and Multi-Agent Manufacturing. Springer, 254--266.
  13. Peter Brusilovsky Barbara J. Ericson Cay S. Horstmann Christian Servin Frank Vahid and Craig Zilles. 2023. The Future of Computing Education Materials. https://csed.acm.org/wp-content/uploads/2023/03/Educational-Materials-First-Draft-1.pdf First Draft to be published in the CS2023: ACM/IEEE-CS/AAAI Computer Science Curricula.
  14. Christopher Bull and Ahmed Kharrufa. 2023. Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into Educational Practice, Not Instinctively Defending Against it. IEEE Software (2023).
  15. Victoria Clarke and Virginia Braun. 2021. Thematic Analysis: A Practical Guide. Thematic Analysis (2021), 1--100.
  16. Alison Clear, Allen Parrish, John Impagliazzo, Pearl Wang, Paolo Ciancarini, Ernesto Cuadros-Vargas, Stephen Frezza, Judith Gal-Ezer, Arnold Pears, Shingo Takada, Heikki Topi, Gerrit van der Veer, Abhijat Vichare, LesWaguespack, and Ming Zhang. 2020. Computing Curricula 2020 Paradigms for Global Computing Education. ACM, New York.
  17. Catherine D'ignazio and Lauren F Klein. 2020. Data Feminism. MIT press.
  18. Thomas Dohmke Marco Iansiti and Greg Richards. 2023. Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle. arXiv:2306.15033 [econ.GN] · arXiv
  19. Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests. arXiv preprint arXiv:2306.05715 (2023). · arXiv
  20. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021).
  21. Wayne Holmes. 2023. Special Issue on Artificial Intelligence in Education: Coming of Age? International Journal of Artificial Intelligence in Education (2023), 1--11. https://link.springer.com/collections/igedgdciea
  22. Maurice Isserman. 2003. Plagiarism: A Lie of the Mind. The Chronicle Review 49 (2003). Issue 34. http://chronicle.com
  23. Brandon Jaipersaud, Paul Zhang, Jimmy Ba, Andrew Petersen, Lisa Zhang, and Michael R. Zhang. 2023. Decomposed Prompting to Answer Questions on a Course Discussion Board. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Ning Wang, Genaro Rebolledo- Mendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 218--223.
  24. Karl O. Jones, Reid Juliet, and Bartlett Rebecca. 2008. Cyber Cheating in an Information Technology Age. Digithum 10 (Dec. 2008). https://raco.cat/index. php/Digithum/article/view/394993
  25. Natalie Kiesler. 2022. Kompetenzförderung in der Programmierausbildung durch Modellierung von Kompetenzen und Informativem Feedback. Dissertation. Johann Wolfgang Goethe-Universität Frankfurt am Main. Fachbereich Informatik und Mathematik.
  26. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. SPOC: Search-based Pseudocode to Code. Advances in Neural Information Processing Systems 32 (2019).
  27. Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. arXiv preprint arXiv:2210.14494 (2022). · arXiv
  28. Juho Leinonen Paul Denny Stephen MacNeil Sami Sarsa Seth Bernstein Joanne Kim Andrew Tran and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938 [cs.CY] · DOI · Paper page
  29. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level Code Generation with AlphaCode. Science 378, 6624 (2022), 1092--1097.
  30. Committee on Publication Ethics. 2023. Authorship and AI tools. https://publicationethics.org/cope-position-statements/ai-author Accessed: 08 July 2023.
  31. Maciej Pankiewicz and Ryan S. Baker. 2023. Large Language Models (GPT) for automating Feedback on Programming Assignments. arXiv:2307.00150 [cs.HC] · arXiv
  32. Fynn Petersen-Frey, Marcus Soll, Louis Kobras, Melf Johannsen, Peter Kling, and Chris Biemann. 2022. Dataset of Student Solutions to Algorithm and Data Structure Programming Assignments. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 956--962.
  33. Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many Bioinformatics Programming Tasks Can be Automated With ChatGPT. arXiv preprint arXiv:2303.13528 (2023). · arXiv
  34. Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming With GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA. https://www.manning.com/books/learn-ai-assisted-python-programming
  35. Ben Puryear and Gina Sprint. 2022. Github Copilot in the Classroom: Learning to Code with AI Assistance. J. Comput. Sci. Coll. 38, 1 (nov 2022), 37--47.
  36. Steven I. Ross, Michael Muller, Fernando Martinez, Stephanie Houde, and Justin D Weisz. 2023. A Case Study in Engineering a Conversational Programming Assistant's Persona. In Joint Proceedings of the ACM IUI Workshops 2023, March 2023, Sydney, Australia.
  37. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI '22). ACM, NY NY, USA, 1--7.
  38. Tianjia Wang, Daniel Vargas-Diaz, Chris Brown, and Yan Chen. 2023. Towards Adapting Computer Science Courses to AI Assistants' Capabilities. arXiv preprint arXiv:2306.03289 (2023). · arXiv
  39. Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large Language Models Meet NL2Code: A Survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7443--7464.

Cited by

Sorted by citations

Related

Authors
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., MacNeil, S., Petersen, A., Pettit, R., Reeves, B. N., & Savelka, J.
Affiliations
  • Abilene Christian University (US)
  • University of Auckland (NZ)
  • University College Dublin (IE)
  • Princess Sumaya University for Technology, Amman, Jordan
  • University of Toronto (CA)
  • Utrecht University (NL)
  • DIPF | Leibniz Institute for Research and Information in Education (DE)
  • Karlsruhe Institute of Technology (DE)
  • Temple University (US)
  • University of Virginia (US)
  • Carnegie Mellon University, Pittsburgh, PA, USA
ORCID
Venue
Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education
Publisher
ACM
Published
2023-12-22
Pages
108-159
DOI
10.1145/3623762.3633499
OpenAlex
W4390315357
Cited by
396
References
112
Citations by year
2026: 135 · 2025: 155 · 2024: 101 · 2023: 5
FWCI
101.70
Type
conference paper
Language
en
Open access
Licence
Locations
ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
Funders
Ulla Tuominen Foundation
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
generative AI, large language models, computing education, artificial intelligence in education, pedagogy, ACM Code of Ethics, AI ethics, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review