The DOI matches its Crossref record; 154 of the 155 references it cites were found in a registry or an index, and 1 are not checked yet; 2 of the 225 sources its claims cite were checked for support.
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Open accessAbstract
Recent advancements in artificial intelligence (AI) and specifically generative AI (GenAI) are threatening to fundamentally reshape computing and society. Largely driven by large language models (LLMs), many tools are now able to interpret and generate both natural language instructions and source code. These capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of generative AI in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles, nearly 80% of which have been published in the first 8 months of 2023. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards GenAI/LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of several current GenAI models/tools on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving.
Verified 2026-10-06 · 1 reference not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This ITiCSE working group report explores how generative AI and large language models are reshaping computing education through a literature review, international surveys, educator interviews, ethics/policy analysis, and benchmarking. It finds rapid LLM capability improvement and broad expected impact, with students and instructors aligned on some views but differing on policy clarity and ethical use.
Key findings
- A scoping literature review synthesised 71 primary articles, nearly 80% published in the first 8 months of 2023; the largest category (35 papers) assessed LLM performance/capabilities, finding LLMs generally perform at or above average students on code generation but weaker on multiple-choice questions, Parsons problems, and computational thinking tasks. The most common risk authors raised was student over-reliance on LLMs.
- Surveys of 171 students and 57 instructors across 20 countries found similar experiences and expectations, including strong agreement that GenAI cannot replace human instructors; instructors found course/university policies less clear than students did, and 50.8% of instructors believed many or almost all of their students were using GenAI tools unethically.
- Interviews with 22 computing educators from nine countries and five continents documented shifts toward “process over product,” invigilated/oral exams, reduced weighting of unsupervised homework, and greater emphasis on code reading and critical thinking, while some educators preserved core programming objectives.
- Benchmarking showed GPT-4 solved all but one problem in a replication of Finnie-Ansley et al. (T2-Q12), outperforming Codex; GPT-3.5 solved most problems; Copilot solved 20/23 exam problems and 4/7 rainfall variants. On APPS, GPT-4 averaged 51.5% of test cases versus GPT-3.5’s 39.2%; on FalconCode, GPT-4 scored 45.4% overall and 74.1% on a cleaned subset.
Methods
- Scoping literature review using keyword searches in ACM Digital Library, Taylor & Francis Online, and IEEE Xplore, plus seed papers and two forward and one backward snowballing phases; 866 papers were filtered down to 71 relevant papers.
- Two online surveys conducted July–August 2023 with 57 instructors from 12 countries and 171 students from 17 countries (responses spanning 20 countries), including Likert-scale items and open-response questions analysed with reflexive thematic analysis.
- Semi-structured interviews with 22 computing educators from nine countries and five continents, analysed thematically; additional ethics/policy review of selected university LLM policies using the ACM Code of Ethics, and benchmarking/replication analyses of GPT-4, GPT-3.5-turbo, and GitHub Copilot on computing education datasets.
Limitations
- Stated: the literature review covered April–August 2023 and only literature from 2021 to August 2023, used a single snowballing round, and included arXiv/non-peer-reviewed work; survey recruitment via SIGCSE mailing lists and snowball sampling risks selection and response bias; interview sampling was geographically skewed (US 55%, no Africa, one each from Asia, Oceania, and South America; only 3 of 22 identified as women), semi-structured interviews can impose interviewer bias, and interviewee claims were not verified.
- Stated: benchmarking focused on code generation, and available datasets often lacked starter code, data files, or formatting instructions; test cases could be incomplete or incorrect, and public datasets may have been included in model training.
- Evident from the text: the field’s rapid pace means findings may quickly become outdated, and the survey samples are modest and not necessarily representative of computing education globally.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
155 Not checked yet · 150 Own result · 68 No basis given · 50 Own interpretation · 27 Own method · 1 Supported · 1 Supported so far
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Although the ability to both read and trace code has always been important [19,128], an examination of course syllabi showed that this is often an afterthought [37,112,114].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2 Learning objectives
What the sources say
- Supportedcites [37] source
Table 2 (section 4.1.2; page unavailable in our copy) places both program comprehension and tracing program execution among the less frequently listed concepts in the surveyed CS1 syllabi: each appears in fewer than 19% of syllabi with explicit learning outcomes and fewer than 16% overall. That supports the paraphrase that reading and tracing receive relatively little attention in the syllabi. The finding concerns documented CS1 expectations, not how much time instructors actually spend teaching those skills. (LLM, conf=0.86)
- Not checked yetcites [112] source
- Not checked yetcites [114] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“They found that Codex 7 had better performance than the median student on the same exam exercises.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
What the sources say
- Supportedcites [85] source
Section 4.1.1 compares Codex’s scores on Tests 1 and 2 with those of 71 students who completed the same tests. Codex ranked 17th, in the top quartile, so it performed better than the median student. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the most recent version of GPT (GPT-4) had only a single piece of research using it.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The backward snowballing phase, which considered all papers in the reference list for each paper in the reference set, resulted in 381 papers.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.2 Snowballing (phase 1).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Five papers used LLMs to analyse student work”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 shows the results considering all the papers we analysed (i.e. up to August 2023), where the most commonly used LLM has become GPT-3/3.5”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Twelve papers were surveys or position papers, summarising or discussing research conducted by others”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For forward snowballing, we used the”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.2 Snowballing (phase 1).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the papers in our dataset, there was broad agreement that LLMs would have a major impact on teaching and learning in computing courses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.4 Impact on teaching and learning
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the most recent version of GPT has 11 papers using it.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Compared to their results, we find that the work in this area is reported more clearly in all four aspects measured.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
What the sources say
- Not checked yetcites [92] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Category (i) is by far the largest group”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3 Classification of literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the most commonly used LLM considering papers from the first snowballing phase only (i.e. up to May 2023) was Codex”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most of these looked into the performance of LLMs in generating code”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“synthesise findings from 71 primary articles, nearly 80% of which have been published in the first 8 months of 2023.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Five papers looked into using LLMs to generate teaching materials”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“One of the prominent opportunities that emerged was related to reducing instructor workload”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all three snowballing phases (two forward, one backward), a total of 866 papers were identified for filtering”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.3 Snowballing (phase 2).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the second forward snowballing, a total of 353 new papers (including duplicates) had cited the seed papers.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.3 Snowballing (phase 2).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“threats to validity are explicitly discussed in the majority, rather than minority, of cases”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems [33,85,123,126,169] and assist in debugging code [125,150,199].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“Wermelinger [191] observed that explanations generated by Copilot focused on a line-by-line description of what the code did rather than how it achieved the goal desired.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [191] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“However, non-code models such as GPT-3 have been found to be more apt at creating higher level code explanations [137].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [137] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“This is supported by the findings of Sarsa et al. [174] who noted that Codex seems most proficient at crafting line-by-line code explanations, as opposed to higher level summaries of the code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [174] source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students and instructors were aligned in their belief that some restrictions should be placed on the use of GenAI tools in their coursework.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The findings reveal a consensus among instructors that auto-generating an entire assignment solution is considered unethical when students lack comprehension of the generated code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.4 Ethical use of Generative AI tools.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, fewer individuals in both groups used GenAI tools for tasks involving images.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Upperlevel students exhibit a greater tendency to use generative AI tools over other resources, including peers, instructors, and teaching assistants.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.3 Help-seeking behaviours.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students had slightly more experience using GenAI for writing code than instructors.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, the extent of students' reliance on GenAI tools appears to be influenced by their academic program stage.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.3 Help-seeking behaviours.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, first-year students still exhibit a preference for seeking assistance from their peers when facing challenges.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.3 Help-seeking behaviours.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this was relatively less common (reported by fewer than one-third of participants), suggesting that instructors have observed their students using GenAI tools more than the instructors use the tools themselves.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, there may be some misalignment around what those restrictions should be.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common response to this question, mirroring the earlier results regarding their own use of the tools, was that they had not observed students using GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A small number of these instructors (two and four for text and code generation, respectively) indicated they planned to use GenAI tools in the near future.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Along this line of reasoning, instructors generally concurred that it is acceptable for students to employ these tools to generate solutions for specific portions of assignments, facilitate code debugging, elucidate concepts, or enhance the style and readability of their code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.4 Ethical use of Generative AI tools.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overwhelmingly, for both types of content, the most common response from instructors was that they were not currently using GenAI tools.”
- Also stated in: § 3.4.1 Instructor use of GenAI. · plain
“Low uptake of GenAI tools: The survey revealed that most instructors are not currently using GenAI tools for text or code generation, but some have expressed plans to integrate them in the future.”
- Also stated in: § 3.4.1 Instructor use of GenAI. · plain
“Overall, most instructors who participated in our survey were not currently using GenAI tools, although several were explicit in their plans to do so in the near future.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
- Also stated in: § 3.4.1 Instructor use of GenAI. · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The primary theme to emerge from responses around the current usage of text generation tools was for the creation of a wide variety of learning resources.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The qualitative themes identified from open-ended responses are representative of the broader respondent group.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on the responses from students and instructors, there was a close alignment in expectations and beliefs regarding GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Expectations and beliefs.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
In the paper
“Also very concerning, Kazemitabaar et al. [108] found that students with more knowledge benefited more from code generation than students with less knowledge.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.3
What the sources say
- Not checked yetcites [108] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“Similarly, Nam et al. [141] found that professionals benefited more from AI code generation than students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.3
What the sources say
- Not checked yetcites [141]
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results indicate that students predominantly continue to favour web searches as their primary resource for help.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.3 Help-seeking behaviours.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, it should be noted that students and instructors agreed slightly more strongly that course policies were clearly defined.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Half of instructors reported they had not used GenAI tools for text generation and 40% said they had not used such tools for code generation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, GenAI tools are progressively establishing themselves as a dependable resource, surpassing online discussion forums as a preferred source of help.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.3 Help-seeking behaviours.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Half of instructors reported they had not used GenAI tools for text generation and 40% said they had not used such tools for code generation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students and instructors shared similar experiences with GenAI tools, using them primarily for writing code and working with text.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While many students raised concerns, a good number of students (28) indicated that they are not concerned about AI taking over their jobs or about the job market being significantly impacted.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Student Adoption of GenAI Tools: Most students have explored GenAI tools for text or code generation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.3 Student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, fewer students argued for always allowing the use of GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditional Acceptance of GenAI Tools: Both instructors and students suggested that whether the use of GenAI tools is permissible should depend on factors such as the course level, assessment type, purpose of the task, and how the tools are used.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, many instructors indicated that GenAI tools should be disallowed in lower-level courses but allowed in upper-level courses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A considerable number of students (33) seemed concerned that GenAI tools will reduce job opportunities.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, a good number of students (n=29) argued for disallowing the use of GenAI tools in all coursework and exams.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plainn = 29- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another recurring and related theme was that allowing the use of GenAI tools depends on the course and assessment learning outcomes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The next most common theme that emerged, appearing in 20% of responses, was around academic misconduct.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Eleven students mentioned that they expect more job opportunities to emerge because of advances in GenAI tools, and 32 students indicated that GenAI will have a positive effect on their productivity in the workplace.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instructor responses for this theme indicated that students frequently use generative AI tools to cheat on their assignments, in-class exercises, projects and on exams.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A related, but less common theme, was around the use of GenAI tools as a conceptual learning aid.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The prevailing sentiment in the responses was that GenAI tools should not be used when students are learning the basics.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority of students argued for a situational or a conditional use of GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two instructors mentioned that the students they had observed using GenAI for writing code actually found the experience frustrating, noting that it would have been easier to write the code themselves.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, expecting an increase in work efficiency was the most commonly occurring comment amongst student responses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preservation of core learning objectives, particularly those related to basic programming, remains a consistent concern among educators.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Eleven students mentioned that they expect more job opportunities to emerge because of advances in GenAI tools, and 32 students indicated that GenAI will have a positive effect on their productivity in the workplace.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, given the rapid emergence of LLMs in computing education, in most cases, these changes are not yet reflected in official curricula or course syllabi.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A minority of the instructor responses supported unconditionally allowing the use of GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This proportion was similar to that of the instructors who had reported not using GenAI.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.3 Student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, some emphatically refuse to use these tools due to concerns around risks to learning and ethical issues about originality.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.3 Student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A recurring theme among the educators we interviewed was the elevation of code comprehension and critical thinking skills.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Instead, some educators are changing more fine-grained learning objectives in their courses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the students in our survey had not used GenAI in their courses, with around 40% of participants responding in this way for both text and code generation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.3 Student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interviewees mentioned unsupervised activities less than inclass activities in general.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Another issue is the presence of hidden or implicit learning objectives in existing computing programs-something Leo Porter described regarding problem decomposition in Section 4.2.1.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“LLM-based tools have motivated some changes to activities that specifically happen during classes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.1 In-class Activities.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is particularly the case since the papers in our study did not propose performance improvements, but rather built on available performance.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The universities we sampled all drew attention to the potential to create harm to others, institutions, and even the students themselves.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Only MIT suggested that the arrival of LLMs into education is an opportunity to think about student academic well-being [17].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [17]
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Figure 3 shows responses to various ethical questions where it seems that many instructors and students agree that using generative AI tools to create an entire answer is wrong.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.5 Use of unauthorised resources
What the sources say
- Not checked yetcites Figure 3
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Many of the policies we sampled mentioned that students can prevent the growth of critical thinking skills and competencies in crucial areas by taking shortcuts through inappropriate use of these models [1,5,13,18].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [1]
- Not checked yetcites [5]
- Not checked yetcites [13]
- Not checked yetcites [18]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Universities seemed most interested in helping students to understand the potential harm that students could incur upon themselves through inappropriate use of AI resources.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As of this writing, many universities do not currently have an official policy publicly available online.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No universities sampled contained anything about general human well-being or the idea that we are all stakeholders in computing, or by extension, education or society in general.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Instead, many universities are still presently working through the policy implications of large language models through task forces and other initiatives, such as at the University of Virginia [4], which is a top-ranked school for computer science in the USA.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [4]
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast to what Birhane et al. report, the focus on performance in our dataset seems to be secondary as a means to highlight the timeliness of the research.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Most universities did not explicitly mention coding in their policies, with the exception of Yale University [13] and University of Adelaide [14].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [13]
- Not checked yetcites [14]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, while performance in the ML research community typically relates to specific (and often well-known) benchmarks, we found that performance in the papers we reviewed usually referred to either the LLMs' ability to solve exercises, assignments or passing exams, or the LLMs' production of teaching materials, say, such as exercises.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In line with Birhane et al. we observed that performance, generalisation, efficiency, novelty, and scalability were often mentioned.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The free availability of large language models (i.e. that students can access LLMs at no cost) was a recurring theme in our surveyed literature.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 performed markedly better than GPT-3.5.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is clear that GPT-4 outperforms Codex, which is the model used in the original study.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
What the sources say
- Not checked yetcites [85] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It turns out that the problem statement included in the dataset only points to an external pdf file that contains the actual instructions, e.g.:”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We did not find existing evaluations of recent LLM performance on the programming tasks found in either dataset.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.4 Novel analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to GPT-4, GPT-3.5 could not solve one of the Rainfall variants, specifically the one from Simon.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the original study, the results of Codex were similar to students in the top quartile; in our replication, GPT-4 would have been one of the top students in the class.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3.5 performed only slightly worse compared to GPT-4.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both the Soloway variant and the variant from Guzdial, none of the completions generated by Copilot handled the edge case where the list is empty, leading to division by zero.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Most of the datasets contain exercises (described in more or less structured natural language).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites Table 7
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“We found that in these two datasets, the vast majority of problems only cover material suitable in an introductory programming setting.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.2 Analysis of problem types
What the sources say
- Not checked yetcites [56]
- Not checked yetcites [64] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We noticed less variation in the completions generated by Copilot compared to the ones from GPT-4 and GPT-3.5.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Overall, all models performed quite well, having performance that would have allowed them to pass the exams.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was not taken into account in the code that GPT-3.5 generated as the code for all ten completions would simply ignore any days with negative rainfall values.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The code generated by Copilot would not remove the full-stop from the end of the sentence, resulting in failure in the edge case when the last word of the sentence needed to be printed where the tests assumed the full-stop is not included in the word.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After removing entries that have identical HTML problem descriptions, we were left with 385 unique problem descriptions.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.4.2 FalconCode.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the rainfall variants, Copilot was unable to correctly solve the one from Soloway, the one from Simon, and the one from Guzdial.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Omitting this in a first test run showed many tests failing because the model returned a method definition.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 performs quite well overall, with an average score of 51.5% on test cases.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Many of the other datasets in Table 7 are similar to the two datasets we analysed, in that they appear to focus on introductory material.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.2 Analysis of problem types
What the sources say
- Not checked yetcites Table 7
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing to GPT-3.5, which scored 39.2%, performance has clearly improved.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The overall performance of 45.4% suggests a rather weak performance of the LLM in handling these introductory programming problems.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The only problem that GPT-4 was not able to solve was the last problem of the second test (T2-Q12), which involved drawing bar graphs using text where the shape of the bar graph was determined by values in a dict passed to the function.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first analysed the performance on Projects since it amounted to 0.0%.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All problems were solved in under ten attempts, except for the last question in the second test (T2-Q12).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Also, as described in the next section, many of the available datasets focus on small exercises used in introductory programming, with relatively few sources of data available to examine larger programming problems or content for more advanced courses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites Table 7
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Almost all of the datasets focus on Python (e.g., providing docstrings as input, Python starter or solution code, or student chats featuring Python code), with a smaller number of datasets featuring other languages, such as C/C++.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites Table 7
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Project | 9 | 0.0% | 0.0% | 0 | - | - |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Lab | 214 | 29.8% 51.4% | 155 | 47.5% 71.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Overall | 385 | 29.3% 45.4% | 236 | 51.4% 74.1% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Hence, one can conclude that we only observed a minimal amount of cases where GPT-4 would produce a truly incorrect solution to the problem (most certainly in fewer than 5% of cases).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Skill | 162 | 30.3% 40.0% | 81 | 57.6% 80.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Overall | 385 | 29.3% 45.4% | 236 | 51.4% 74.1% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that, in general, students and instructors had similar perceptions about LLMs with respect to questions around experiences, expectations and beliefs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on the above analysis, we report another set of results (Clean) on the subset of 236 FalconCode problems that provide sufficient information for the LLM and are associated with valid test cases.The success rate increases from 45.4% to 74.1%.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that some instructors were beginning to place a greater emphasis on 'process over product'.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Most datasets we found could support code generation, using Python, of CS1 problems.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the remaining cases GPT-4 generated a genuinely incorrect solution.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common type of paper to date involves evaluating the performance of LLMs when applied to tasks such as solving programming exercises.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We reviewed academic integrity policies that mentioned generative AI from major universities around the world and found that these explicitly addressed many of the principles stated in the ACM code of ethics.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our analysis revealed that the main causes of the poor performance are related to the same cause detected with the Project tasks.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Skill | 162 | 30.3% 40.0% | 81 | 57.6% 80.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We encountered a number of challenges applying LLMs to publicly available datasets, even though the datasets themselves are of high quality.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The publicly available datasets used are of high quality.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A key finding, which justifies some of the widely voiced concerns around academic misuse of LLMs, is that current models tend to perform at least as well as most students on typical introductory-level programming tasks.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many of the respondents to our survey had very little experience using generative AI tools at the current time”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Due to rapid changes in the field and the slow pace of publishing in traditional venues, much of this work was available only as pre-prints on platforms such as arXiv.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, they differed in their perceptions of how clear course policies were about the allowed use of LLMs, with instructorssomewhat surprisingly-finding these policies to be less clear than students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Skill | 162 | 30.3% 40.0% | 81 | 57.6% 80.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Project | 9 | 0.0% | 0.0% | 0 | - | - |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some have already made concrete changes to their curricula and assessments.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Overall | 385 | 29.3% 45.4% | 236 | 51.4% 74.1% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Skill | 162 | 30.3% 40.0% | 81 | 57.6% 80.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Lab | 214 | 29.8% 51.4% | 155 | 47.5% 71.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of 40.0% is rather unexpected”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Skill assignments are supposed to require only small solutions, not exceeding several lines of code.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Lab | 214 | 29.8% 51.4% | 155 | 47.5% 71.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our own replication of this prior work, using a state-of-the-art model, shows an extraordinary performance improvement over the span of two years since the original work was carried out.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is worth emphasising that if we would also disregard the cases where GPT-4 produced the correct solution using an unexpected structure (e.g., a function returning a value instead of a program asking user for an input and printing to a terminal) or utilised an unexpected library, the success rate would increase to 86.8%.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most common cause of failed unit tests for Lab assignments was also a missing data file and/or starter code (not released with the dataset).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also identified several instances where GPT-4 generated a genuinely incorrect solution to the problem statement that provided sufficient information.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Lab | 214 | 29.8% 51.4% | 155 | 47.5% 71.0% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the risks that were identified, the most common concern expressed by authors was that students would become overly-reliant on using LLMs to generate and debug code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a few cases, the unit tests would be missing or incorrect.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In many cases perceptions were well-aligned-both students and instructors felt strongly that there should be some restrictions on the allowed usage of generative AI tools for coursework.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Overall | 385 | 29.3% 45.4% | 236 | 51.4% 74.1% |”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, a large portion of the genuinely incorrect solutions are rather superficial problems, such as not rounding to a single decimal as demonstrated in the example provided in the instructions.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another common cause was an incorrect structure of the (possibly correct) solution.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“researchers have demonstrated that these models have an increasing capacity to perform source code generation and interpretation through a natural language interface [74].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [74]
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“there is now at least one textbook, published in September 2023, which features the use of Generative AI-specifically GitHub Copilot and ChatGPT-from day 1 of introductory programming courses [155].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [155]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Similarly, LLMs were found to be able to generate reasonable learning objectives [186].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
What the sources say
- Not checked yetcites [186]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“When used to generate questions or hints to support students in solving problems, the research is inconclusive [21,30,82,151,192]”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“A study found that code generated by ChatGPT had enough differences from student code that it could be detected very accurately (between 96-98% accuracy) [99].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
- Not checked yetcites [99]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In general, LLMs seem to perform at a level that is equivalent to or better than that of average students, at least for code generation tasks.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Ahmed et al. achieved a repair rate of 89.4% [20].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
What the sources say
- Not checked yetcites [20] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“For other tasks, such as answering MCQs [176], Parsons problems [167] and computational thinking tasks [39], the performance of LLMs is currently not as great.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The evidence gathered over the past few decades about how students learn best supports the commonly adopted approach of having students write many small programs checked by automated assessment tools over the course of their introductory terms [23].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [23] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All papers suggest that LLMs will have substantial impact on computing education and programming more generally.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“and are able to explain their answers to textbook questions in about half of the cases [103].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
- Not checked yetcites [103] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“One of the main findings in both papers was that LLMs can be coaxed into generating exercises with prescribed themes (such as basketball or cooking) and programming concepts (such as loops or conditionals).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
plainconditionaln = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“evidence is mounting that these tools are currently ineffective [144].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [144]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The studies that examined the performance in bug repair both reported that their results surpassed previous state-of-the-art automated program repair results.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Instructors can also benefit as these models can rapidly generate new and personalised teaching materials and programming assignments [75,174].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Zhang et al. reported an overall repair rate of up to 96.5% using Codex with few-shot examples and iterative prompting [199]”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
What the sources say
- Not checked yetcites [199]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“although LLMs are able to provide better explanations of code than students [123]”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
- Not checked yetcites [123] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Automatically generated tests may be partially complete, leading students to inadequately test their code [191].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [191] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The rapid pace of the field is a potential threat, however, to the results being published.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The impressive documented performance of large language models in solving typical CS1 and CS2 problems suggests that some rethinking of assessments is essential [86].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [86] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The learning resources produced by generative AI pose significant risks to student success.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The use of generative AI may also result in students spending time in unproductive ways.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Including a second snowballing phase was motivated by the very fast pace at which the literature is growing in this area, which this finding supports.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.5 In-Depth Reading.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to the fast pace of work on Generative AI in all disciplines including computing education, only literature published in less than a three year period (2021 to August 2023) was considered.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.7 Limitations and threats to validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Teachers who use generative AI to assist in the creation of learning resources may unintentionally produce exercises that are underspecified or that contain incorrect reference solutions or inadequate/incorrect test cases [174].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
What the sources say
- Not checked yetcites [174] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“Further, novice learners of programming may become overwhelmed and confused by generated code, which could add to the high levels of frustration that are common in introductory programming courses [71,188].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [71] source
- Not checked yetcites [188]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that while GenAI tools do not currently replace the value provided by instructors, it is important for instructors to reflect on and clearly define their value to students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Expectations and beliefs.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the inclusion of these sources admits work that has not yet undergone peer review.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.7 Limitations and threats to validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that instructors may be supportive of students using these tools as long as students demonstrate a clear understanding of the task and achieve the intended learning outcomes of the course.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.4 Ethical use of Generative AI tools.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Teachers who shift away from using many small problems to create larger and more authentic problems in an attempt to reduce reliance of generative AI will lose access to the quick and easy assessment methods such as automated grading associated with many introductory programming courses [86].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
What the sources say
- Not checked yetcites [86] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these developments were important for many reasons and groups of (present and future) students, they are not comparable, at least not in pace and ubiquity, to the rapid changes Large Language Models (LLMs) are currently triggering in higher education, the computing disciplines, and CS1 in particular.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“This is one of the main ways the present work differs from that of Lau and Guo [121] discussed in Section 3.1.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
What the sources say
- Not checked yetcites [121] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“LLMs can be useful in generating several kinds of learning activities, including novel variations of programming assignments [174].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
- Not checked yetcites [174] source
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this was relatively less common (reported by fewer than one-third of participants), suggesting that instructors have observed their students using GenAI tools more than the instructors use the tools themselves.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“LLMs can also generate good explanations of code [94,123,137,174].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“But curricula often do not explicitly include these program-level outcomes in the learning outcomes for a particular course [38,113].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [38] source
- Not checked yetcites [113]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Students can continually alter prompts to refine the model output helping them to 'build-up' a solution [196].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
- Not checked yetcites [196] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“although LLMs have been shown to perform well on more advanced (coding competition) problems [126].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4 Assessment
What the sources say
- Not checked yetcites [126]
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“However, evidence also shows that LLMs do not perform as effectively on computational thinking problems that do not involve code writing [39].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4 Assessment
What the sources say
- Not checked yetcites [39] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This provides a mechanism for novices struggling to understand the run-time behaviour of novel problems to get auto-generated and hopefully helpful explanations.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“The same is true for course activities and assessments.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [38] source
- Not checked yetcites [113]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The potential biases reinforced by models trained on large datasets [77] are concerning.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [77]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“While a number of researchers have looked at how current developers may use LLM-based tools [32,52,81] and how LLM-tools may be incorporated into professional software development tools [84], the participants in these studies have been programmers who learned to code initially without using LLMs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“LLMs can help instructors with the tedium of such minutiae [137,181].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Hence, with an increasingly opaque training procedure of LLMs and a lack of clear responsibility in terms of authorship [172], the use of large language models naturally raises a number of concerns pertaining to academic integrity.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5 ETHICS
What the sources say
- Not checked yetcites [172] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The use of AI tools could become a liability if inexperienced developers fail to remove or correct the tool's incorrect suggestions [61].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [61]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“It is also interesting, given how little is known about the true role of notional machines in the classroom [76], that generative AI is likely to alter how students conceptualise the computer and program execution.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [76] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps as a result of this our geographic representation is skewed.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
hedgedn = 22- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, only 14% (3) of the 22 identified as women.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
plainn = 22- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It seems the ideas in these policies are limited to the specific implications of using LLMs, rather than the general.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One of the most significant concerns is the ease with which LLMs can produce the output that we ask students to produce.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In some courses, such as introductory programming, the use of generative AI may be undesirable as it can solve problems with minimal intellectual input from students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.5 Use of unauthorised resources
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Although professional developers may benefit from AI's human-quality suggestions, novice developers lack the expertise to recognise and understand buggy or non-optimal solutions [61].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [61]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“For example, it was shown that LLMs can successfully solve 97% of the programming problems in a bioinformatics course [152].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [152]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some students might therefore have access to more powerful tools than others.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: ChatGPT offers a range of models with different performance characteristics, not all of which are free.
In the paper
“Finally, as previously identified by Liu et al. [129], many of the published datasets do not provide robust evaluation of the exercises they include.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites [129]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“FalconCode is a novel collection of over 1.5 million Python programs from over 2,000 undergraduate students at the United States Air Force Academy [64].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.4.2 FalconCode.
What the sources say
- Not checked yetcites [64] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Finnie-Ansley et al. [85] published the first paper that examined the performance of large language models in solving introductory programming exercises.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.3 Replication: The robots are coming
What the sources say
- Not checked yetcites [85] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the currently published literature which has focused on largely on GPT-3 may underestimate what the newest models, such as GPT-4, can do.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7 BENCHMARKING LARGE LANGUAGE MODELS FOR COMPUTING EDUCATION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, in the context of computing education, several problems in this dataset might not be suitable for novice programmers.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the starter code has not been included in the dataset, the LLM does not have the complete information to produce the desired output.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Producing a dataset that contains everything necessary for high-quality LLM research (in particular, accurate evaluation of the artefacts generated by LLMs) is challenging and needs to be encouraged by the community.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Our replication of the Finnie-Ansley et al. [85] paper suggests that new LLM models are significantly more capable than is currently reported in the literature.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
What the sources say
- Not checked yetcites [85] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiment with the FalconCode dataset further supports this conclusion”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, replicating prior work using newer models is difficult, given that a wide variety of parameters, prompts, and evaluation approaches have been used, and not all methods are reported with sufficient detail.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number and quality of the test cases provided might not completely cover all exercise requirements and edge cases, therefore giving a false positive result.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 155
- It may state two separate claims as one. 123
- The quote only partly states this claim. 3
None of the sources cited for this claim have been checked yet.
In the paper
“Indeed, some recent papers found in the second phase snowballing focused on introducing tools around LLMs, such as CodeHelp [127] and Promptly [71].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The papers we reviewed identified a number of potential opportunities that could positively impact computing education.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Some looked into other types of exercises such as multiple-choice questions [175][176][177]189], textbook questions [103], exam questions [79], computational thinking tasks [39], and textual reports [48].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
- Not checked yetcites [175] source
- Not checked yetcites [176]
- Not checked yetcites [177]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The papers we reviewed broadly fall into five categories, with respect to the role that the LLM plays in the study”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3 Classification of literature
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and slightly more of the work we examined present explicit research questions.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These exclusions left us with 71 relevant papers (10 reference papers, 28 papers from the first snowballing phase, and 33 papers from the second).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.5 In-Depth Reading.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“A total of nine papers looked into interactions between programmers and LLMs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The search was conducted on 26th April 2023 and resulted in 19 papers.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.1 Reference papers.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After the filtering was complete, a total of 80 papers were identified for careful reading.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Title Venue Year Citation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Five papers in the literature review used interviews to understand user perceptions and attitudes towards LLMs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority of the research focuses on Python, with some work involving Java and C.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Improving the learning experience for students was another common opportunity that emerged.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“During this indepth reading phase, 9 papers were flagged as not being relevant.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.5 In-Depth Reading.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The resulting set of reference papers (”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.1 Reference papers.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“which is now no longer available”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“More than thirty papers looked into assessing the performance or capabilities of large language models.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, while the second snowballing phase covered only a few months, it resulted in a number of papers that is very close to the number of papers that resulted from the first snowballing phase, which covered a period of around two and a half years.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.5 In-Depth Reading.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Several risks were identified by the papers we reviewed.Authors were concerned that generative AI could be used in ways that limit student learning or make the work of educators more difficult [90].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6 Risks
What the sources say
- Not checked yetcites [90]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“Hellas et al. [94] found that LLMs tended to hallucinate issues in student code which could cause students to focus on these non-existent issues instead of the actual issues in their code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [94] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“Another recurring opportunity mentioned in the papers we reviewed was the potential for a renewed focus on problem solving.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other types of text-based artefacts that instructors reported creating were course materials, examples for students, explanations of complex algorithms, and scenarios for highlighting ethical issues in software engineering.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The qualitative themes identified from open-ended responses are representative of the broader respondent group.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Responses from students and instructors to questions related to experiences and expectations were largely aligned.However, some important differences emerged for questions focusing on course policies.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While students had mixed opinions about whether the university policies were clear, instructors largely disagreed with the statement that the policies were clear.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.2 Comparison of student and instructor perceptions.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Several authors raise concerns about the impact of generative AI on teachers and teaching practice.Unsurprisingly, issues of academic integrity were the most common concern raised.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, both students and instructors also expected GenAI tools to play an increasing role in the future of their teaching and learning, as well as in students' future careers.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Expectations and beliefs.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The tools, when utilised, are primarily used for creating diverse educational materials, however, satisfaction regarding the quality of outputs vary.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several instructors highlighted the summarisation capacity of GenAI tools since they can effectively condense long-form text content.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The qualitative themes identified from open-ended responses are representative of the broader respondent group.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“What constitutes ethical use is complex: What is considered ethical varied between instructors and students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.4 Ethical use of Generative AI tools.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Those that were using them were doing so to generate a broad array of educational content, including assessment questions, practice exercises and examples, although not all appeared satisfied with the outcomes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both groups strongly agreed that GenAI tools cannot replace human instructors and that human teachers provide more effective guidance than GenAI tools.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Expectations and beliefs.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A number of interviewees shared this perspective and highlighted the need to preserve several core learning outcomes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three students also mentioned that standing out to employers will become harder, given that many applicants might use GenAI tools to build portfolios that make them appear as solid candidates despite lacking the required skills.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was a mix of positive and negative responses, which is consistent with the quantitative results of the corresponding Likert question (Question 17 in Appendix B).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Others argued that it is useless to attempt to disallow their use outside closedexam conditions, as students will use them anyway.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the other end of the spectrum, a few responses supported always disallowing them, or disallowing them in all graded assessments (regardless of the level, type, topic, etc.).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another recurring argument was that using GenAI tools in coursework and exams defeats the purpose of assessments.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“According to those students, relying on GenAI tools may reduce their understanding of the material and thus affect their future chances of employment.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some instructors argued that it is a function of complexity rather than course-level.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many educators acknowledge the dynamic nature of technology and anticipate potential adjustments to their learning objectives in the near future.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.1
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Student answers were more diverse and more polarised than those of the instructors.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some said that their use is fine as long as the student acknowledges that appropriately.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some also argued for completely disallowing them even outside assessments (i.e. while learning).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Several instructors discussed changes in their upcoming course learning objectives.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“At the same time, educators seem to be aware of the need to prepare students for industry expectations, which is likely to include LLM use.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Those who do use GenAI tools most commonly apply them for paraphrasing or summarising text, for debugging errors in their own code, and slightly less often, for code generation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.3 Student use of GenAI.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To conclude, computing educators express the need to balance between leveraging LLMs for problem-solving while ensuring that students continue to develop competencies in algorithmic thinking, program comprehension, and debugging.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The rise of LLMs is causing many instructors to rethink their course learning objectives.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2 Learning objectives
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Educators who do not alter their learning objectives stress their focus on teaching fundamental programming concepts.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Some educators, including Mark Liffiton (Illinois Wesleyan University, USA), view LLMs as valuable tools that can aid students in coding tasks.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some students argued that software engineering jobs are particularly at risk.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.4 Student perceptions of the effects of GenAI tools on employment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The survey respondents are representative of computing educators and students in the relevant populations.
- Rests on an assumption: Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in the same way that they expect students to learn to use an IDE on their own time, some instructors have indicated that they will not be dedicating classroom time to explicitly teaching students LLMs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.1 In-class Activities.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Given that public awareness about LLMs occurred mid-academic year for many institutions (almost universally for North America and Europe in addition to many other parts of the world) this September (2023) marks the first academic year where LLMs are a nearly ubiquitous topic.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Caspersen noted that students read more than they write when learning to read natural language, but to-date this has not been the traditional approach when it comes to programming.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Some educators have already started using LLMs as part of the learning process.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.3 Process over product.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Peter Mawhorter's local policy is that LLMs are not allowed in class.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Viraj Kumar used GitHub Copilot during class for live coding and told students they were free to use it, or any other Generative AI for coding.Students were polled after the course and asked if they were using ChatGPT.Most said that they were not.The consensus explanation seemed to be that they wanted to learn programming themselves and their thinking seemed to be that when they go for technical”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rather than assessing final solutions as products, some educators are increasingly focusing on assessing learning processes (which is common in other disciplines, e.g., teacher education) such as submit-in-stages; solution reflection, commentary, critique; interviews; portfolios or learning journals; and presentations.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.3 Process over product.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several educators mentioned oral exams.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.1 Exams.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Liffiton has also developed a tool called CodeHelp [127] that uses LLMs, which he is going to use with his students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
- Not checked yetcites [127] source
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, finding and organizing enough computer-based testing facilities is (to date) a challenge as computer labs and institutional machines have become less frequent over the years.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sven Strickroth (LMU Munich, Germany) noted another challenge that is caused by institutional policy-significant changes to learning objectives are not easily possible due to the local accreditation cycle which dictates that this occurs only every five years in many institutions.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“It is worth noting that some of these findings support those of Lau and Guo [121], notably the increased emphasis on invigilated exams, oral exams, and process-based assessments.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.3 Process over product.
What the sources say
- Not checked yetcites [121] source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The book begins by introducing students to the GitHub Copilot plugin within the Visual Studio Code IDE before students have learned to write a single line of Python code.Students create their first programs by typing English comments and letting Copilot generate the code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
- Not checked yetcites [155]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nonetheless, they are beginning to emerge as educators start thinking about the coming academic year.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other side of the coin, Leo Porter is embracing LLMs and is aiming to use them, for instance, in quizzes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Several educators worried that some classes do not lend themselves to the approaches mentioned above.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.3 Process over product.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Interestingly, Leo Porter is the only interviewee who mentioned pair programming directly.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.1 In-class Activities.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the student-initiated possibilities discussed above come with academic integrity concerns we discuss in Section 6.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Some instructors are moving towards invigilated exams and assessments, and these may be worth more marks.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.1 Exams.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tomaschitz was concerned that the ChatGPT group was over-reliant on the tool and did not understand the output as well as the e-book group.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In a high school introductory programming course, Christian Tomaschitz (Theresianum Eisenstadt, Austria) had half of his students use ChatGPT and the other half use an e-book.ChatGPT was not introduced to students and students registered OpenAI accounts with their school e-mail addresses.All students had the same exam which was half open-book (including internet access), half closed-book.Tomaschit”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Jean Mehta plans to use 20-30 minute one-on-one oral exams at the end of every section.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.1 Exams.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Most of the discussion around unsupervised activities is centred around ensuring that students are doing their own work (even if using Generative AI as a tool for help) and that students understand the code that these tools produce.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Given that institutions are typically slow to react mid-year-if they react at all, we have yet to arrive at a steady state in terms of institutional initiatives.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However Porter's institutional challenge is finding and organising enough computer-based testing facilities.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Several interviewees including Leo Porter and Michael Caspersen believe that LLMs will cause educators to focus more on code reading and less on writing from scratch.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several interviewees were aware of this, stressing the need for educators to emphasise the importance of students doing their work.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Potentially more dangerous is the fact that students can now execute code that they do not understand, yet designed through natural language prompts.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The problems where any model struggled were either complex (e.g., printing bar graphs as text) or had vague problem descriptions, leading to some edge cases being ignored (e.g., rainfall variants that did not specify what should happen when the list is empty).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“At the same time, relatively few explicitly reference OO topics (4.1%) or common data structures (6.6%).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Dataset
What the sources say
- Not checked yetcites [95]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“A review of Table 7 suggests a number of limitations in the data available for pursuing LLM research in an educational context.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites Table 7
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While Copilot took the constraints into account, all ten completions had logical flaws and did not pass all the tests.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Unsurprisingly, GPT-4 outperformed GPT-3.5 and Copilot, which corroborates previous work where GPT-4 has outperformed other LLMs for various tasks [21,151,175,184].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
What the sources say
- Not checked yetcites [21]
- Not checked yetcites [151]
- Not checked yetcites [175] source
- Not checked yetcites [184]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, while reviewing the problems, we found that almost all were examples of small exercises, with relatively few requiring multiple functions to solve.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.2 Analysis of problem types
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found evaluation to be limited, with some datasets requiring manual intervention to complete evaluation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Keeping a strict pass/fail criterion (all tests should pass), only 36.1% of the GPT-4 solutions pass all tests, as do 21.0% of GPT-3.5 solutions.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe a large difference between problem types, with GPT-4 solutions to introductory problems scoring as high as a 72.2% test case average, but the most difficult, competition level problems scoring only 28.7%.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Problem
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Copilot performed the worst out of the three evaluated models, successfully solving 20 out of the 23 exam problems and four out of the seven rainfall variants.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Again, we found that the majority of the content was primarily suitable for an introductory course, with relatively few questions asking about object-oriented code or data structures.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.2 Analysis of problem types
What the sources say
- Not checked yetcites [86] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In contrast, relatively few datasets contain chat logs where students interact with an LLM or student-submitted code with syntax errors (for code repair tasks), but additional publicly available data for questions beyond code-generation would be beneficial, as that would allow researchers operating in smaller educational settings, where collecting sufficient data may be difficult, to engage in LLM”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.1 Review of empirical datasets
What the sources say
- Not checked yetcites Table 7
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“As noted in the original study, LLMs struggle with trivial formatting issues, such as missing punctuation or producing extra output that is not specified by the problem description.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“a common thread of work to date has been to evaluate the performance of various models.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, it is unclear how students are being educated about the ethical use of generative AI in the classroom.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since the pdf file has not been released with the dataset we could not provide the LLM with adequate instructions to generate a solution.This is the case for all 9 projects.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, there was also a trend towards placing a greater emphasis on invigilated assessments such as exams, with a reduction to the grade weighting placed on unsupervised homework assignments.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are a substantial number of situations where the Skill assignment required an external data file, and even more commonly starter code was needed to complete the assignment”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also detected instances where the LLM used an unexpected library (not part of the Python standard library) and hence the program would crash, i.e., the unit tests would fail.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some instructors were concerned that their students were using such tools inappropriately, and a small fraction of students refused to use generative AI tools for ethical reasons and due to concerns about harming their learning.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Many of these models are easily available and free for students, and early reports reveal that students are already using them for assistance on their assignments [44].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [44] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Leinonen et al. found that Codex could enhance programming error messages-which are notoriously hard for students to understand-approximately 54% of the time on average, noting that this performance is not good enough for using the model directly with students [125].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
What the sources say
- Not checked yetcites [125] source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Sandoval et al. found that using code generator tools did not seem to introduce new security risks [173].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
What the sources say
- Not checked yetcites [173]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Most students found the prompt writing to be beneficial, while a few voiced concerns.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
What the sources say
- Not checked yetcites [71] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Bull and Kharrufa argue that the type of scaffolding that AI tools can provide can”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [52]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Another study (which we classified as assessing LLMs, i.e. category (i)) used student written prompts in order to evaluate LLMs and found it to be an effective benchmark [29].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
What the sources say
- Not checked yetcites [29]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They found that students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
What the sources say
- Not checked yetcites [108] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“MacNeil et al. found that students generally found code explanations generated by LLMs useful for learning, but the perceived utility of the explanations and students' engagement with them varied by explanation type [137].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.3 Studying the interactions between programmers and LLMs (9):
What the sources say
- Not checked yetcites [137] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“In addition, the exercises generated by LLMs were novel and sensible, although the authors cautioned that the quality might not be good enough to provide the LLM-generated exercises directly to students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
What the sources say
- Not checked yetcites [75]
- Not checked yetcites [174] source
hedgedn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Furthermore, generative AI models can provide wrong or biased answers, and students may also become over-reliant on LLM tools or generate code plagiarised from online sources by the model [33].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [33] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“One study reports on the experiences of five students using generative AI for assignments [89], highlighting both aspects of where it offered effective support, but also many limitations.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
What the sources say
- Not checked yetcites [89] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The models might generate code students do not understand [108] or may distract them with large blocks of text they did not write [159].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“All papers suggest that using LLMs could help instructors save time in generating teaching materials.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They note that while the tools do not require training to use, developers will need to understand the generated code for quality assurance, and to avoid over-reliance as the quality of code produced by these tools can vary.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
What the sources say
- Not checked yetcites [52]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Another study that compared LLM-generated content with student-generated content concluded that the quality is comparable, but still recommends further research [69].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.5 Using LLMs to generate teaching materials (5):
What the sources say
- Not checked yetcites [69]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, tools that assess whether a given text was generated by an LLM show a large number of false positives and should not be trusted blindly [144].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.1 Assessing the performance, capabilities and limitations of
What the sources say
- Not checked yetcites [144]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Three other papers report on interviews with students and instructors about their experiences with, and attitudes towards LLMs [121,164,198], finding that there is no consensus about the use of LLMs in higher education, its benefits or risks, but there is general awareness of the problem of academic integrity in the light of LLMs being used by students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.2 Position papers and surveys/interviews (17):
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“One paper presents a tool to help students with issues without revealing the full solution [127] and reports positive feedback from both students and instructors.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
What the sources say
- Not checked yetcites [127] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“They found that in the best case, their proposed method can achieve a precision of 72.4% with a coverage of 64.2% for one of the datasets they used and a precision of 76% and a coverage of 31.2% for the other dataset.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3.4 Using LLMs to analyse student work (5): Five papers used
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Large language models will have a profound impact on computing education in the next decade as the technology matures and as teachers and researchers identify opportunities.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Finally, LLMs present clear opportunities for instructors to rethink assessment practices and reconsider what assessment means in computing courses [39,70,153,176].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [39]
- Not checked yetcites [70] source
- Not checked yetcites [153]
- Not checked yetcites [176]
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“LLMs will change how, what, and whom we teach not only in computing but in all of education [68].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 1 INTRODUCTION
What the sources say
- Not checked yetcites [68]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Data extraction was guided by a set of questions implemented as an online form to help standardise the process.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1.5 In-Depth Reading.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Example programs used by students for learning could have poor implementation or poor style, which may result in students acquiring undesirable programming habits [85].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [85] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We chose to perform a scoping review to rapidly identify gaps and major themes in the literature”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1 Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Leinonen et al. suggest that LLMs could help students understand terse error messages [125] which have traditionally been a source of difficulty for novice learners [36].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We conducted two separate phases of forward snowballing, one in May 2023 and one in August 2023”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1 Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Category (i) is by far the largest group, indicating a strong desire to assess the current capabilities and limitations of LLMs in computing education contexts.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.3 Classification of literature
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Several papers described the idea of using an LLM as an assistant or pair programmer, which represents a significant change from current pedagogical practice [123,152,169].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [123] source
- Not checked yetcites [152]
- Not checked yetcites [169]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Despite the promise of using AI tools for learning support, Dakhel et al. and Prather et al. caution that although they can be a great asset for professional developers, they may be less helpful for novices if the tools generate non-optimal or erroneous outputs which could cause confusion [61,159].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [61]
- Not checked yetcites [159] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“The resources created by LLMs might also have less variation than those created by humans [69] and thus limit the variety of examples to which students are exposed.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [69]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the quality and performance of the models improve, this may be less of an issue as time goes by.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“AI generated content is not curated for specific courses, so learning material generated could potentially include syntax, programming constructs, or other content that is inappropriate for students in a given course [33,94].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is little doubt that LLMs and other forms of GenAI will have a profound impact on computing education over the coming years.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All evaluators were instructed to interpret the criteria generously, to avoid exclusion of potentially relevant work.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Title Venue Year Citation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Since AI generated content is presented authoritatively (and is frequently correct), students are unlikely to question the content and may learn incorrect information [52,103].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [52]
- Not checked yetcites [103]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In summary, the papers we reviewed suggest that LLMs present a wide array of opportunities for computing education such as improving instructor productivity by reducing workload, enhancing student learning experiences, enabling a greater emphasis on problem-solving, and suggesting new assessment practices.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“The fact that Python emerges as the most popular language in this work is not too surprising, however, as popular LLMs such as Codex have been reported to be most proficient in Python [56].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
What the sources say
- Not checked yetcites [56]
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“We selected the most generally applicable questions, and we updated their question about threats to validity to focus specifically on whether they were discussed in an explicit subsection.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.2 Descriptive statistics
What the sources say
- Not checked yetcites [92]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“We chose to perform one step of forward and backward snowballing [194] from a set of reference papers”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.1 Method
What the sources say
- Not checked yetcites [194]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“LLMs can also be used to generate a variety of flawed solutions, providing plentiful opportunities for incorporating code review tasks [85].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.5 Opportunities
What the sources say
- Not checked yetcites [85] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“This is akin to a multi-structural explanation [180], which may focus student attention at lower levels of the SOLO taxonomy, rather than thinking about the overall purpose of code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [180]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings suggest that AI code generators may widen the gap between over-and under-achievers, exacerbating teaching challenges arising from classes with heterogeneous ability levels.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.3
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although there is a growing need to teach students how to use generative AI appropriately, it is unclear how we should do so.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“As more code is likely to be generated automatically, there is potential for biases to be unintentionally introduced due to inherent bias in models and because of bias in the training data [33].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.3
What the sources say
- Not checked yetcites [33] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“Generative AI is reported to perform very well in assessments that are commonly used in introductory courses, raising concerns that students will submit solutions that they have not created themselves [33,70,85,86,121,152,169,176,177].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
What the sources say
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
In the paper
“During our deep review of the included papers, there were a few concerns about the quality of a given paper.We retained these papers as they met the inclusion criteria, and note that, on the whole, the papers appear to be ready for review and demonstrate many of the criteria for quality proposed by Hellas et al. [92].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.7 Limitations and threats to validity
What the sources say
- Not checked yetcites [92]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These are situations where the tool could save time without negatively affecting learning outcomes.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.3.4 Ethical use of Generative AI tools.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Teachers who are concerned about the impact of generative AI may intentionally modify course delivery in ways that reduce the effectiveness of their teaching practice (e.g., by increasing academic integrity at the cost of scaffolded programming exercises), or adjust the curriculum to shift focus away from code writing, leaving students poorly prepared for subsequent programming courses [165].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
What the sources say
- Not checked yetcites [165]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“The solutions generated by AI might not be easily identified as AI-generated [160], which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained [191].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.2 Risks for teachers.
What the sources say
- Not checked yetcites [160]
- Not checked yetcites [191] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For this reason we conducted a single round of snowballing (i.e., we did not do subsequent snowballing on the papers found in the first round of snowballing).This may have omitted some work that failed to reference the most visible early work (our”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.7 Limitations and threats to validity
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our literature review was conducted from April to August 2023.Therefore work published subsequently is not included.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.7 Limitations and threats to validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Students who rely on generative AI may be misled into believing they are making progress, and this illusion of capability may reduce their self-understanding about their level of mastery of the subject matter [158,159].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [158]
- Not checked yetcites [159] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
None of the sources cited for this claim have been checked yet.
In the paper
“Generated code may contain security vulnerabilities and bugs [33,52,173].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.3
What the sources say
- Not checked yetcites [33] source
- Not checked yetcites [52]
- Not checked yetcites [173]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This points to an emergent interest in GenAI from instructors in our survey and recognition of the use of GenAI for teaching.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.1 Instructor use of GenAI.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As introductory students realise that generative AI can outperform them on most tasks, they may lose motivation to learn the material, and become demoralised about the future of computing [123].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 2.6.1 Risks for students.
What the sources say
- Not checked yetcites [123] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While instructor responses to this question do reveal the potential for GenAI tools to be used to aid student learning, they also highlight a concerning trend of academic misconduct and over-reliance.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.2 Instructor observations of student use of GenAI.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Although LLM-based tools like Copilot may serve as a useful springboard for solving CS1 problems, students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills [191] in order to become not only proficient computing experts but also to use LLMs effectively.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
What the sources say
- Not checked yetcites [191] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Additionally, perceptions and experiences of both students and educators also vary, often focusing on difficulty [35,132] and LLMs may stand to impact this long-standing issue.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
What the sources say
- Not checked yetcites [35]
- Not checked yetcites [132]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two of the authors reviewed both the verbatim interview transcripts and the identified themes for all interviews.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“With LLMs available on nearly everyone's phone and laptop 4 , it is not only knowledge that is instantly retrievable but also problem explanations and solutions -in the form of programming code that is not necessarily correct.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
What the sources say
- Not checked yetcites [4]
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The interviews, which reflect practices from the teaching community in this new context of Generative AI were fully transcribed verbatim and served as a basis for thematic analysis [47,170].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Given the myriad impacts of LLMs, it is important to acknowledge that educational systems are notoriously slow to change.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
What the sources say
- Not checked yetcites [182]
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“They were conducted via Zoom and automatically transcribed via speech recognition software, followed by a correction loop by a human (interviewers checked transcripts of other reviewers, not their own).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“After the transcripts were finalised, we deleted all audio and video recordings in accordance with the protocol submitted to the University of Toronto Research Ethics Board, who approved it prior to the study.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important criterion for inclusion was that educators would have concrete plans or views toward changing their current course structure, assessment, or classroom practices in light of LLMs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also allowed participants to review a draft manuscript before final publication, to ensure that they are not misrepresented.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Now, however, with the seeming ubiquity of LLMs, it is inevitable that educators consider their impact on teaching, learning, assessment, and delivery, leading to possible redesign of their courses at all of these levels.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
In the paper
“For students in non-computing majors who currently only take one (or just a few) programming course(s) to learn enough to write simple programs, it may be that using an LLM tool is all that is needed, making a programmingspecific course unnecessary [152].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.3 Towards conversational computing.
What the sources say
- Not checked yetcites [152]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Next, themes were discussed and specified among four of the authors.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, students still need to learn to code and practice.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2.2 Preserving core learning outcomes.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Based on the concept of Constructive Alignment [42], learning objectives need to be aligned with exercises, assignments, and assessment methods.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
What the sources say
- Not checked yetcites [42]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The last category above is interesting as it assumes that, in general, the use of GenAI tools is unethical unless it is out of necessity.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 3.4.5 Student and instructor perspectives on when GenAI tools should be allowed.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the pervasiveness of LLMs, this paradigm shift regarding the availability of knowledge, solutions, examples, and content (particularly in the form of code) is more comparable with the advent of the internet than other developments in the annals of how we teach and learn computing-yet the speed of internet adoption was much slower as a whole.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4 CURRICULUM AND ASSESSMENT
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The sample comprises 22 computing educators from nine countries and five continents.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Interview summaries were then compiled based on these themes and double-checked against the themes and transcripts by a second author.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.1 Methodology for expert interviews
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, as students increasingly use LLMs to write programs, instructors might need to put less emphasis on writing code and more emphasis on reading code.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.2 Learning objectives
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Students will increasingly use LLMs to write programs.
In the paper
“Although some institutions (at least initially) have universally banned the use of LLM tools in student work [142], others are starting to embrace them.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
What the sources say
- Not checked yetcites [142]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is one example of how LLMs can have knock-on effects that could not only affect the course in question but others via resource allocation and timetabling.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Getting students to recognise when it is a shortcut versus when it actually is not a shortcut is something that novices typically are not in a position to assess.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Students rated these explanations on a 5-point Likert scale, confirming that explanations matched the code and were useful for learning in general, whereas line-by-line explanations were rated as least helpful.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
What the sources say
- Not checked yetcites [137] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The knock-on effects of LLM use also need to be considered.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.5 Institutional initiatives, policy, and context
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“This usecase exemplifies the potential for LLMs to ease the burden often felt by teaching assistants, and could be a first line of help for students [174].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3 Course activities
What the sources say
- Not checked yetcites [174] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Research on LLMs has demonstrated their ability to answer typical CS1 assessments that involve writing simple functions [70,117] as well as more advanced material.Similarly, it has been shown that they perform in the upper quartile of real students on CS1 exams [85].It has also been shown that they perform just as well on CS2 exams as CS1 exams [86] suggesting that LLMs may soon be capable of”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4 Assessment
What the sources say
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“There is little doubt that there are other such hidden learning objectives spanning not only knowledge, and skills, such as reading and tracing code but also dispositions, inter-and intra-personal competencies, and other aspects relating to the whole person [163] which the emergence of LLMs might bring to the fore.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [163]
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach also shifts focus from memorising knowledge (or in a rote manner, the process that leads to the product without questioning the process) to the application of skill and critical thinking.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.3 Process over product.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“An exploration of the potential of LLMs to generate formative programming feedback [115] suggests that ChatGPT performs reasonably well in generating feedback to students' solutions to introductory programming tasks, indicating that students can potentially benefit in unsupervised learning scenarios.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
What the sources say
- Not checked yetcites [115]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Denny et al. [70] found that Copilot's performance is substantially improved when it is prompted with individual problem-solving steps in natural language and they explicitly encourage teaching prompt engineering to students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.1 In-class Activities.
What the sources say
- Not checked yetcites [70] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Similarly, while LLMs can often correctly answer more than half of coding-based multiple-choice problems, they answer double-digit percentages of multiple-choice problems incorrectly, leading to the hypothesis that either a combination of natural language and a code snippet, and/or chain-of-reasoning steps pose a challenge for LLMs [177].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4 Assessment
What the sources say
- Not checked yetcites [177]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There may be less summative unsupervised work because instructors no longer trust that unsupervised work is the students' own.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.4.2 Homework.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, it may fall on educators to guide how to use the generated feedback, as it can contain misleading information for novices.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.3.2 Unsupervised Activities.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Harm to institutions can come in the form of students using LLM tools to generate work that is incorrect, offensive, or otherwise inappropriate and therefore dragging the university into potential altercation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The author is fully responsible for the accuracy of any information provided by the [generative AI] tool and for correctly referencing any supporting work on which that information depends.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.4 Falsification
What the sources say
- Not checked yetcites [3]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Authors are accountable for the accuracy, integrity and originality of their research papers, including for any use of AI.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.4 Falsification
What the sources say
- Not checked yetcites [2]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Simon et al. [183] highlights the importance of educating students about academic integrity and reveals a wide variety of ways that academic integrity is communicated to students.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.6 Advice for students
What the sources say
- Not checked yetcites [183]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Harm to others could come in the form of using the work of others without citation [12].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [12]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Using a tool to produce the required output circumvents that learning, and deprives students from the opportunity to learn [100].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
What the sources say
- Not checked yetcites [100]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Authors are fully responsible for the content of their manuscript, even those parts produced by an AI tool, and are thus liable for any breach of publication ethics.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.4 Falsification
What the sources say
- Not checked yetcites [143]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected the United States forms the bulk (55%) of responses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
plainn = 22- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Moreover, Mittelstadt et al. [140] point out that algorithmic systems tend to be large, complex and highly modularised, which makes it difficult in general to assign responsibilities.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5 ETHICS
What the sources say
- Not checked yetcites [140] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the interviews were semi-structured and although this is a common approach it can impose interviewer bias, although it also designed to result in a coherent set of interviews which focus on common topics while still allowing interviewers to express their own views and experiences.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Furthermore, using a large language model has been shown to affect the user's own opinions [102].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5 ETHICS
What the sources say
- Not checked yetcites [102] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we did not attempt to verify claims that interviewees made about their classes, departments, or institutions, taking interviewee statements at face value.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Where authors use generative AI and AI-assisted technologies in the writing process, these technologies should only be used to improve readability and language of the work.Applying the technology should be done with human oversight and control and authors should carefully review and edit the result, because AI can generate authoritative-sounding output that can be incorrect, incomplete or biased.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.4 Falsification
What the sources say
- Not checked yetcites [6]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Use of LLMs can lead to poor outcomes due to overreliance, ease of breaching academic integrity, and use of incorrect information leading to poor products.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although our interview pool spanned five continents, we had no interviewees from Africa and only one interviewee each from Asia (India), Oceania (New Zealand), and South America (Brazil).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.8 Limitations and threats to validity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Not only could this violate academic integrity, but as discussed above, students who utilise these resources inappropriately in lower-level courses may cause harm to themselves by preventing their own preparation for upper-level coursework.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.5 Use of unauthorised resources
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“In a total of 100 highly cited papers, the study found that performance was clearly the most frequent value and pointed out that performance is not a neutral term but comes with ethical implications.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
What the sources say
- Not checked yetcites [45] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In a study on the values encoded in the ML research literature, Birhane et al. identified a number of ethical values [45], from which the papers would draw motivation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 5.1 Ethics in LLM literature
What the sources say
- Not checked yetcites [45] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not consider this a systematic attempt, nor would that be presently possible, given the conditions described above.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, in upper-level courses, it may be an appropriate productivity tool that students would be permitted to use, which the survey data in Figure 3 seems to corroborate given responses to questions about generating pieces of an assignment, help with style, or fixing bugs.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.5 Use of unauthorised resources
What the sources say
- Not checked yetcites Figure 3
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“It is also interesting, given how little is known about the true role of notional machines in the classroom [76], that generative AI is likely to alter how students conceptualise the computer and program execution.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 4.7 Discussion
What the sources say
- Not checked yetcites [76]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can also call into question the legitimacy of the learning outcomes and verification of them, which can jeopardise accreditation and institutional reputation or the community's trust in that institution.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > ACM General Ethical Principles.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Taken together, this analysis suggests that many of the available datasets-and as a result, the published results-reflect an early introductory programming context (CS1) with relatively few examples of common CS2 material (Cipriano and Alves [57] is a recent exception demonstrating research on OO topics) and even fewer covering any more advanced topics.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Dataset
What the sources say
- Not checked yetcites [57]
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The dataset contains 661 introductory Python programming problems used in 4 courses.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.4.2 FalconCode.
What the sources say
- Not checked yetcites [64]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Specifically, we performed a thematic analysis in which causes of each failed assignment were extracted as codes and then collated into higherlevel themes [47].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results analysis.
What the sources say
- Not checked yetcites [47] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“This time, the dataset was tagged by two authors (one who had tagged the previously discussed datasets and a second experienced instructor); we calculated Cohen's kappa and found near perfect agreement (0.94).”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.2 Analysis of problem types
What the sources say
- Not checked yetcites [86] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“This might be due to the use of a relatively high temperature value for these models-previous work found similar results and speculated that Copilot likely uses a lower temperature value [191].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Results (Copilot):
What the sources say
- Not checked yetcites [191] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Some recent work has found that GPT-4 outperforms earlier models in tasks such as visual programming [98,184], Socratic questioning of novices [21], solving multiple-choice questions and programming exercises [175], and that performance can be close to human tutors for some tasks [151] while not for others [21].”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7 BENCHMARKING LARGE LANGUAGE MODELS FOR COMPUTING EDUCATION
What the sources say
- Not checked yetcites [98]
- Not checked yetcites [184]
- Not checked yetcites [21]
- Not checked yetcites [175] source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This has implications for academic integrity, in that plagiarism is not usually a concern but instead users would be responsible for any falsification produced from uncritical use of LLMgenerated artefacts.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- The authors state this as an assumption: The user of the LLM should be considered the author of the generated text.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is clear that LLMs present significant challenges but also new opportunities for computing educators.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 8 CONCLUSIONS
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We used this analysis to filter the dataset down to a clean version that is appropriate to use for the evaluation.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Acknowledging that internet access is required to access LLMs”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess model performance on multiple types of tasks, for different programming languages, or at different levels, will require new datasets.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Alternately, test cases can even be incorrect, or too strict, exceeding what is required in the instructions, lowering the potential performance of models.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“subscriptionbased services which could be superior to free ones present issues of access based on means, and potentially opening new divides”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This criterion rules out posters and abstracts.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“As suggested by Liu et al. [129], existing datasets may need to be augmented to evaluate LLMgenerated code accurately.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5 Discussion
What the sources say
- Not checked yetcites [129]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Outside of code generation, assessing solutions for other types of exercises common in CS (e.g.regular expressions, UML-diagrams, automata), LLM-generated exercises, feedback, and explanations require additional datasets and may require a qualitative framework that could be difficult to provide or to transfer to another research team.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5.1 Limitations.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These latter aspects could also be assessed automatically, but we have not attempted to do so in our study.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5.1 Limitations.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Testing generated code can be easily automated by running test cases, although this might not capture all aspects relevant to computing educators, such as code quality and suitability of the solution with regards to which concepts the student has learned so far.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 7.5.1 Limitations.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
The set of reference papers and the snowballing procedure captured the relevant literature on LLMs in computing education.
“We chose to perform one step of forward and backward snowballing [194] from a set of reference papers that were identified as being currently significant work in the area of large language models in computing education.”
The paper relies on this without support.
The 71 papers included in the review are representative of the current state of research on LLMs in computing education.
“These exclusions left us with 71 relevant papers (10 reference papers, 28 papers from the first snowballing phase, and 33 papers from the second).”
The paper relies on this without support.
The survey respondents from 20 countries are representative of computing students and instructors.
“we report the findings of a survey of computing students and instructors from across 20 countries”
The paper relies on this without support.
The interview participants from five continents are representative of computing educators.
“we offer insights collected from in-depth interviews with 22 computing educators from five continents.”
The paper relies on this without support.
The benchmark datasets used are appropriate for evaluating GenAI models and tools in computing education.
“we benchmark the performance of several current GenAI models/tools on various computing education datasets”
The paper relies on this without support.
The reviewed literature, despite single-round snowballing and inclusion of non-peer-reviewed arXiv work, is representative enough that omitted work would not significantly change the general trends and themes identified.
“but we do not believe that this will include significant numbers of papers or change the general trends and themes identified in our literature analysis.”
The paper relies on this without support.
The survey samples of 57 instructors and 171 students are sufficiently representative of computing instructors and students to support the reported comparisons and trends.
“We recruited 57 instructors and 171 students to complete the corresponding online surveys.”
The paper relies on this without support.
The survey respondents are representative of computing educators and students in the relevant populations.
“The survey revealed that most instructors are not currently using GenAI tools for text or code generation, but some have expressed plans to integrate them in the future.”
The paper relies on this without support.
Instructors' and students' self-reported uses and ethical judgments accurately reflect their actual practices and views.
“The findings reveal a consensus among instructors that auto-generating an entire assignment solution is considered unethical when students lack comprehension of the generated code.”
The paper relies on this without support.
The qualitative themes identified from open-ended responses are representative of the broader respondent group.
“The primary theme to emerge from responses around the current usage of text generation tools was for the creation of a wide variety of learning resources.”
The paper relies on this without support.
Students will increasingly use LLMs to write programs.
“as students increasingly use LLMs to write programs”
The paper relies on this without support.
The influence of media coverage of computing topics is a large factor in pre-university students' decisions about what courses to pursue at university.
“Additionally the influence of media coverage of computing topics is known to be a large factor in the decisions pre-university students make in terms of what courses to pursue at university.”
The paper relies on this without support.
ChatGPT offers a range of models with different performance characteristics, not all of which are free.
“On the other hand, ChatGPT offers a range of models with different performance characteristics, not all of which are free.”
The authors state this as an assumption.
Both parties in collusion have agency.
“This requires both parties to willingly agree to work together and therefore assumes that both parties have agency.”
The paper relies on this without support.
Many students will be able to fix small mistakes made by LLMs themselves.
“For instance, we expect that many students will be able to fix small mistakes made by LLMs themselves.”
The paper relies on this without support.
Tasks in existing benchmarks might not match those typically found in computing education courses.
“Similarly, tasks in existing benchmarks might not match those typically found in computing education courses.”
The paper relies on this without support.
A slight variation in a prompt might generate quite different results.
“Furthermore, a slight variation in a prompt might generate quite different results.”
The paper relies on this without support.
GPT-4 is the state-of-the-art LLM at the time of writing.
“At the time of writing, GPT-4 is the state-of-the-art LLM, so it is interesting to see how it performs on solving the problems.”
The paper relies on this without support.
GPT-3.5-turbo is likely what many students will be using.
“GPT-3.5-turbo is the model that powers the free version of ChatGPT, which is likely what many students will be using, and thus it is also interesting to include.”
The paper relies on this without support.
GitHub Copilot is free for students and educators, and students may be using it.
“Lastly, GitHub Copilot is free for students and educators, and is directly embedded into the IDE, so it is possible that students will be using it too, which is why we decided to include it in the replication.”
The paper relies on this without support.
The HTML may encode important information through formatting that a state-of-the-art LLM such as GPT-3.5/4 may be capable of leveraging.
“We decided against extracting plain text of the problem statements because the HTML may encode important information through formatting that a state-of-the-art LLM such as GPT-3.5/4may be capable of leveraging.”
The paper relies on this without support.
It is possible that solutions for the APPS dataset have been included in training recent models.
“It is possible that solutions for it have been included in training recent models.”
The paper relies on this without support.
Skill assignments are supposed to require only small solutions, not exceeding several lines of code.
“As Skill assignments are supposed to require only small solutions, not exceeding several lines of code”
The paper relies on this without support.
The APPS dataset problems are significantly more challenging than those typically provided in CS1 and CS2 courses.
“those problems are significantly more challenging than those typically provided in CS1 and CS2 courses.”
The paper relies on this without support.
The publicly available datasets used are of high quality.
“even though the datasets themselves are of high quality.”
The authors state this as an assumption.
The user of the LLM should be considered the author of the generated text.
“We agree with the position articulated by many publishers that the user of the LLM should be considered the author of the generated text.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multiple studies showed evidence of bimodal distributions of GenAI usage [146, 279], with some students using GenAI tools every day and many others never having used these tools.”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [279]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In an introductory programming course, students expressed concerns about privacy when using Github Copilot [278].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other authors note the risk of students becoming overly reliant on these tools, which might hinder their independent problem-solving skills [23].”
From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“There is also the risk of students becoming overly reliant on AI assistance, which may impede the development of fundamental programming skills and critical thinking (Prather et al., 2023).”
From The impact of AI-assisted pair programming on student motivation, programming anxiety, collaborative learning, and programming performance: a comparative study with traditional pair programming and individual approaches (2025) · cites this paper as Prather et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“however, there were differing opinions about the ethics of this practice when students possessed a full understanding of the generated code or wrote the code in a different language and then translated it into the language of instruction.”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]
The cited source says what the claim attributes to it.
In the citing paper
“While acknowledging benefits to students' learning from productive uses of GenAI, unauthorized use of the tools for learning tasks and assessment was of widespread concern [278].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]
The cited source says what the claim attributes to it.
In the citing paper
“Whereas a study (n=57) by Prather et al. [278] found consensus among academics that submitting an auto-generated solution without understanding to be unethical”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]
The cited source says what the claim attributes to it.
In the citing paper
“This not only impacts their learning but has implications for their future professional work, particularly in areas such as programming where an ability for independent reasoning is critical [278].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [278]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > 6.5 Use of unauthorised resources
“Figure 3 shows responses to various ethical questions where it seems that many instructors and students agree that using generative AI tools to create an entire answer is wrong.”
§ The Robots are Here: Navigating the Generative AI Revolution in Computing Education > Academic misconduct
“Figure3: Summaries of the survey responses from 171 students and 57 instructors: 1) students ranked their help seeking preferences from 1 to 6, and 2) instructors shared their beliefs about the ethical use of Generative AI Tools.”
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Qualitative research & evaluation methods
Patton, M. Q. (2002) · Sage eBooks · Cited by 15,530
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“The broader literature on LLMs and their potential effects is often organized around the dichotomy of opportunities and risks [44, 100, 151, 174].”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Ryan, G. W., & Bernard, H. R. (2003) · Field Methods · doi:10.1177/1525822x02239569 · Cited by 5,475
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“The possibility that generative AI invents “facts” is wellknown, and typically described by the term hallucination [97].”
BackgroundThe ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 4,567 · cited 1×
“[130] point out that algorithmic systems tend to be large, complex and highly modularised, which makes it difficult in general to assign responsibilities.”
BackgroundResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Research works printed but not linked: 39
- [n. d.]. Authorship and Contributorship. https://www.cambridge.org/core/services/authors/publishing-ethics/research-publishing-ethics-guidelines-for-journals/authorship-and-contributorship#ai-contributions-to-research-content Accessed: 08 July 2023.
- [n. d.]. Best Practice Guidelines on Research Integrity and Publishing Ethics. https://authorservices.wiley.com/ethics-guidelines/index.html#5 Accessed: 08 July 2023.
- [n. d.]. Publishing Ethics | Elsevier Policy. https://beta.elsevier.com/about/policies-and-standards/publishing-ethics?trial=true#4-duties-of-authors Accessed: 08 July 2023.
- [n. d.]. Using ChatGPT or Other Generative AI Tool on a Marked Assessment. https://www.academicintegrity.utoronto.ca/perils-and-pitfalls/using-chatgpt-or-other-ai-tool-on-a-marked-assessment/. Accessed: 06 September 2023.
- 2019. AAAI Code of Professional Ethics and Conduct. https://aaai.org/about-aaai/ethics-and-diversity/. Accessed: 08 July 2023.
- 2020. IEEE Code of Ethics. https://www.ieee.org/about/corporate/governance/ p7--8.html Accessed: 08 July 2023.
- 2023. ACM Code of Ethics. https://www.acm.org/code-of-ethics Accessed: 08 July 2023.
- 2023. ACM Policy on Authorship. https://www.acm.org/publications/policies/new-acm-policy-on-authorship Accessed: 08 July 2023.
- Mikko Apiola, Sonsoles López-Pernas, and Mohammed Saqr. 2023. Past, Present and Future of Computing Education Research: A Global Perspective. Springer Nature.
- Brett A. Becker. 2017. Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going. Ireland's Yearbook of Education 2018 (2017), 42--46.
- John Biggs. 1996. Enhancing Teaching Through Constructive Alignment. Higher education 32, 3 (1996), 347--364.
- Robert W Brennan and Jonathan Lesage. 2022. Exploring the Implications of Openai Codex on Education for Industry 4.0. In International Workshop on Service Orientation in Holonic and Multi-Agent Manufacturing. Springer, 254--266.
- Peter Brusilovsky Barbara J. Ericson Cay S. Horstmann Christian Servin Frank Vahid and Craig Zilles. 2023. The Future of Computing Education Materials. https://csed.acm.org/wp-content/uploads/2023/03/Educational-Materials-First-Draft-1.pdf First Draft to be published in the CS2023: ACM/IEEE-CS/AAAI Computer Science Curricula.
- Christopher Bull and Ahmed Kharrufa. 2023. Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into Educational Practice, Not Instinctively Defending Against it. IEEE Software (2023).
- Victoria Clarke and Virginia Braun. 2021. Thematic Analysis: A Practical Guide. Thematic Analysis (2021), 1--100.
- Alison Clear, Allen Parrish, John Impagliazzo, Pearl Wang, Paolo Ciancarini, Ernesto Cuadros-Vargas, Stephen Frezza, Judith Gal-Ezer, Arnold Pears, Shingo Takada, Heikki Topi, Gerrit van der Veer, Abhijat Vichare, LesWaguespack, and Ming Zhang. 2020. Computing Curricula 2020 Paradigms for Global Computing Education. ACM, New York.
- Catherine D'ignazio and Lauren F Klein. 2020. Data Feminism. MIT press.
- Thomas Dohmke Marco Iansiti and Greg Richards. 2023. Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle. arXiv:2306.15033 [econ.GN] · arXiv
- Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests. arXiv preprint arXiv:2306.05715 (2023). · arXiv
- Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021).
- Wayne Holmes. 2023. Special Issue on Artificial Intelligence in Education: Coming of Age? International Journal of Artificial Intelligence in Education (2023), 1--11. https://link.springer.com/collections/igedgdciea
- Maurice Isserman. 2003. Plagiarism: A Lie of the Mind. The Chronicle Review 49 (2003). Issue 34. http://chronicle.com
- Brandon Jaipersaud, Paul Zhang, Jimmy Ba, Andrew Petersen, Lisa Zhang, and Michael R. Zhang. 2023. Decomposed Prompting to Answer Questions on a Course Discussion Board. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Ning Wang, Genaro Rebolledo- Mendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 218--223.
- Karl O. Jones, Reid Juliet, and Bartlett Rebecca. 2008. Cyber Cheating in an Information Technology Age. Digithum 10 (Dec. 2008). https://raco.cat/index. php/Digithum/article/view/394993
- Natalie Kiesler. 2022. Kompetenzförderung in der Programmierausbildung durch Modellierung von Kompetenzen und Informativem Feedback. Dissertation. Johann Wolfgang Goethe-Universität Frankfurt am Main. Fachbereich Informatik und Mathematik.
- Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. SPOC: Search-based Pseudocode to Code. Advances in Neural Information Processing Systems 32 (2019).
- Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. arXiv preprint arXiv:2210.14494 (2022). · arXiv
- Juho Leinonen Paul Denny Stephen MacNeil Sami Sarsa Seth Bernstein Joanne Kim Andrew Tran and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938 [cs.CY] · DOI · Paper page
- Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level Code Generation with AlphaCode. Science 378, 6624 (2022), 1092--1097.
- Committee on Publication Ethics. 2023. Authorship and AI tools. https://publicationethics.org/cope-position-statements/ai-author Accessed: 08 July 2023.
- Maciej Pankiewicz and Ryan S. Baker. 2023. Large Language Models (GPT) for automating Feedback on Programming Assignments. arXiv:2307.00150 [cs.HC] · arXiv
- Fynn Petersen-Frey, Marcus Soll, Louis Kobras, Melf Johannsen, Peter Kling, and Chris Biemann. 2022. Dataset of Student Solutions to Algorithm and Data Structure Programming Assignments. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 956--962.
- Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many Bioinformatics Programming Tasks Can be Automated With ChatGPT. arXiv preprint arXiv:2303.13528 (2023). · arXiv
- Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming With GitHub Copilot and ChatGPT. Manning, Shelter Island, NY, USA. https://www.manning.com/books/learn-ai-assisted-python-programming
- Ben Puryear and Gina Sprint. 2022. Github Copilot in the Classroom: Learning to Code with AI Assistance. J. Comput. Sci. Coll. 38, 1 (nov 2022), 37--47.
- Steven I. Ross, Michael Muller, Fernando Martinez, Stephanie Houde, and Justin D Weisz. 2023. A Case Study in Engineering a Conversational Programming Assistant's Persona. In Joint Proceedings of the ACM IUI Workshops 2023, March 2023, Sydney, Australia.
- Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI '22). ACM, NY NY, USA, 1--7.
- Tianjia Wang, Daniel Vargas-Diaz, Chris Brown, and Yan Chen. 2023. Towards Adapting Computer Science Courses to AI Assistants' Capabilities. arXiv preprint arXiv:2306.03289 (2023). · arXiv
- Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large Language Models Meet NL2Code: A Survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7443--7464.
Cited by
Sorted by citations
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Ethical challenges and solutions of generative AI: an interdisciplinary perspective
Al-kfairy, M., Mustafa, D., Kshetri, N., Insiew, M., & Alfandi, O. (2024) · Informatics · doi:10.3390/informatics11030058 · Cited by 328
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“For a more technical overview of the historical developments and future trends of language models, we direct the reader to the CACM article by Li [20].”
MethodSelf-citationDeveloping evaluative judgement for a time of generative artificial intelligence
Bearman, M., Tai, J., Dawson, P., Boud, D., & Ajjawi, R. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2024.2335321 · Cited by 312
“…across various means of communication (e.g. text, images, sound), but also as inherent to a domain, subject or discipline – e.g. writing assistance (Escalante, Pack, and Barrett 2023), qualitative analysis methods (Siiman et al. 2023) or computer code generation/programming (Prather et al. 2023).”
Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda
Bittle, K., & El-Gayar, O. (2025) · Information · doi:10.3390/info16040296 · Cited by 305
“Others recommend teaching ethical GenAI use, explicitly stating restrictions in syllabi and requiring students to disclose GenAI use [23].”
Metadata
- Authors
- Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., MacNeil, S., Petersen, A., Pettit, R., Reeves, B. N., & Savelka, J.
- Affiliations
- Abilene Christian University (US)
- University of Auckland (NZ)
- University College Dublin (IE)
- Princess Sumaya University for Technology, Amman, Jordan
- University of Toronto (CA)
- Utrecht University (NL)
- DIPF | Leibniz Institute for Research and Information in Education (DE)
- Karlsruhe Institute of Technology (DE)
- Temple University (US)
- University of Virginia (US)
- Carnegie Mellon University, Pittsburgh, PA, USA
- ORCID
- Venue
- Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2023-12-22
- Pages
- 108-159
- OpenAlex
- W4390315357
- Cited by
- 396
- References
- 112
- Citations by year
- 2026: 135 · 2025: 155 · 2024: 101 · 2023: 5
- FWCI
- 101.70
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
- Funders
- Ulla Tuominen Foundation
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- generative AI, large language models, computing education, artificial intelligence in education, pedagogy, ACM Code of Ethics, AI ethics, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
