CoChat
Sign Up
Cited by 54Teaching and Learning ProgrammingArtificial Intelligence in EducationSoftware Engineering Research

Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study

Open access
Sheard, Judy, Denny, Paul, Hellas, Arto, Leinonen, Juho, Malmi, Lauri, Simon·2024·Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
DOI 10.1145/3626252.3630880
Open this paper in CoChat — free accountOpen PDF

Abstract

Much of the recent work investigating large language models and AI Code Generation tools in computing education has focused on assessing their capabilities for solving typical programming problems and for generating resources such as code explanations and exercises. If progress is to be made toward the inevitable lasting pedagogical change, there is a need for research that explores the instructor voice, seeking to understand how instructors with a range of experiences plan to adapt. In this paper, we report the results of an interview study involving 12 instructors from Australia, Finland and New Zealand, in which we investigate educators' current practices, concerns, and planned adaptations relating to these tools. Through this empirical study, our goal is to prompt dialogue between researchers and educators to inform new pedagogical strategies in response to the rapidly evolving landscape of AI code generation tools.

AI snapshot

Written from the paper’s full text.

TL;DR

This qualitative interview study reports perspectives of 12 programming instructors from Australia, Finland, and New Zealand on AI code generation tools in computing education. Instructors saw opportunities for learning and teaching but raised major concerns about over-reliance, assessment integrity, and the need to rethink programming education.

Key findings

  • Instructors identified opportunities for AI tools to support learning, such as generating examples, providing tailored explanations, and giving students a starting point when stuck; some likened this to fading worked examples. They also saw teaching efficiencies, including creating exercise banks, personalized tasks, model solutions, marking guides, and feedback.
  • All 12 interviewees expressed concerns about students missing out on learning by using AI tools, including over-reliance, inability to judge correctness, and giving agency to the tool to “learn for them.” Assessment was the area with the most perceived threats, especially unauthorized use of tools for take-home assignments, which several saw as a form of contract cheating that is difficult to detect.
  • Proposed responses included detection tools, process monitoring such as code snapshots or keystroke data, oral assessment, clear guidelines requiring acknowledgment of AI assistance, invigilated exams, and unique individualized projects or portfolios. Many recognized a need to rethink assessment design.
  • For programming education more broadly, interviewees agreed AI tools will dramatically change the field. Most felt tools should be introduced after students are proficient with programming, not in introductory courses, and that teaching should shift toward code integration, evaluation, and testing while still teaching effective interaction with AI tools.

Methods

  • Qualitative phenomenological interview study with 12 programming instructors purposively selected from universities in Australia, Finland, and New Zealand. Participants came from six institutions; four were female and eight male; teaching experience ranged from 10 to 50 years; 9/12 had taught both introductory and advanced courses, one only introductory, and two only advanced.
  • Data collection used semi-structured interviews lasting 40–60 minutes, conducted in person or via Zoom between December 2022 and May 2023. Participants were shown a brief video of Copilot before the interview; the interview guide is available as an online appendix.
  • Analysis followed Braun and Clarke’s six-step thematic analysis. Transcripts were audio-recorded, transcribed verbatim, and pseudonymized; codes, themes, and subthemes were reviewed by another author, and eight themes were organized under the three research questions.

Limitations

  • The paper states that only 12 academics from six universities were interviewed, so their thoughts may not accurately represent the larger population of computing educators; however, the authors note similar thoughts emerged and some saturation may have been reached.
  • The paper states that large language models are advancing very fast, so results can become dated; the instructors’ views reflect AI capabilities available at the time of interviews in early 2023 and might differ now.
  • Evident from the text: the sample is limited to three countries and six institutions, and the study relies on self-reported interview data rather than observed teaching or assessment practice or measured student outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3626252.3630880

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 42 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    There was general agreement among interviewees that it is important to teach students how to interact effectively with AI tools.

    “There was general agreement that it is important to teach students how to interact effectively with AI tools.”
    Quote in the paperSubstantial claim
  • Findingown data

    Several interviewees mentioned the need to suddenly rethink and redesign their courses, as all the current exercises they had for the students could be completed by the tools.

    “Several interviewees mentioned the need to suddenly rethink and redesign their courses, as all the current exercises they had for the students could be completed by the tools.”
    Quote in the paperSubstantial claim
  • Findingown data

    All interviewees expressed concerns about students missing out on learning by using AI tools.

    “All interviewees expressed concerns about students missing out on learning by using AI tools, a similar finding to the study by Zastudil et al. [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Findingown data

    Assessment was the area where interviewees perceived the most threats.

    “Assessment was the area where interviewees perceived the most threats, which has also been found in other studies [19,39].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [19]

      Not checked in the free run.

    • cites [39] source

      Not checked in the free run.

  • Findingown data

    A common concern among interviewees was that students would not go through the crucial thinking processes and steps necessary to learn programming.

    “A common concern was that students would not go through the crucial thinking processes and steps necessary to learn programming.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 86
  • May combine two claims. 55

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    AI tools could be used to create a bank of exercises providing a greater number and variety of programming tasks than is typically feasible at present.

    “For example, the tools could be used to create a bank of exercises which would provide a greater number and variety of programming tasks than is typically feasible at present.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Overall, it was recognized that there is a need to rethink assessment design.

    “Overall, it was recognized that there is a need to rethink assessment design.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    One interviewee saw this as a form of surrogacy, which will not work.

    “One interviewee saw this as a form of surrogacy, which of course will not work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The interviewees mentioned a number of ways AI tools could assist students in learning programming but also expressed many concerns.

    “The interviewees mentioned a number of ways in which AI tools could assist students in their learning of programming but they also expressed many concerns.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    A major challenge facing programming educators is deciding what to teach in programming courses, because what will be required for a programming job is very likely to change in the future.

    “A major challenge facing programming educators is deciding 'what to teach' in programming courses, as what will be required for a programming job is very likely to change in the future.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Portfolio assessments were mentioned by interviewees but viewed by some as problematic because it is now more difficult to tell whether the portfolio is the student's own work.

    “Portfolio assessments were also mentioned, but they were viewed by some as problematic as it is now more difficult to tell whether the portfolio is the work of the student.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Most interviewees felt it was important that programming professionals have knowledge and practical understanding of the foundations of programming, and that this should be taught in programming courses.

    “While most interviewees felt it was important that programming professionals had knowledge and practical understanding of the foundations of programming, and that this should be taught in programming courses”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some interviewees questioned whether all computing students would need low-level programming skills in the future.

    “some interviewees questioned whether all computing students would need low-level programming skills in the future.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Many interviewees argued the need for some form of invigilated assessment to monitor potential use of the tool, and a number mentioned an exam in a controlled environment.

    “Many interviewees argued the need for some form of invigilated assessment to monitor potential use of the tool, and a number mentioned an exam in a controlled environment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The danger is that students can use the tools and believe they are learning.

    “The danger is that students can use the tools and believe they are learning.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Over-reliance on AI tools for solving small problems in introductory programming classes means that students would not learn the fundamental elements of programming and not be prepared for higher level courses with larger tasks that Copilot would not be able to answer.

    “Over-reliance on AI tools for solving small problems in introductory programming classes means that students would not learn the fundamental elements of programming and not be prepared for higher level courses with larger tasks that Copilot would not be able to answer.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Four of the interviewees had played with the tools generally and six had explored the capabilities of the tools with their own coursework.

    “Four of the interviewees had played with the tools generally and six had explored the capabilities of the tools with their own coursework, for example by generating course content and testing the capabilities of the tools to generate and explain solutions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Findingown data

    One interviewee proposed that AI tools should be introduced as early as possible in computing programs.

    “One interviewee proposed that AI tools should be introduced as early as possible.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Interviewees agreed that the advent of AI tools will dramatically change programming education, and this was seen as a daunting prospect.

    “Interviewees agreed that the advent of the AI tools will dramatically change programming education and this was a daunting prospect.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The main idea proposed by interviewees to meet the challenge of rethinking assessment was using unique and individualized projects, possibly incorporating the use of AI tools.

    “The main idea proposed to meet this challenge was using unique and individualized projects, possibly incorporating the use of AI tools.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The main threats from AI tools related to teaching were the teachers' lack of preparedness and not knowing how students would use the tools.

    “The main threats from AI tools related to teaching were the teachers' lack of preparedness and not knowing how students would use the tools.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    An important consideration among interviewees was how the computing industry will use AI tools.

    “An important consideration was how the computing industry will use AI tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The 12 interviewees, four female and eight male, came from six institutions in Australia, Finland and New Zealand.

    “The 12 interviewees, four female and eight male, came from six institutions in Australia, Finland and New Zealand.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The fact that generated code is not always correct or appropriate was seen as a positive because it means students need to read and understand the code to adapt or fix it, and they would learn from doing this.

    “The fact that generated code is not always correct or appropriate was seen as a positive as this means that students need to read the code and understand it in order to work out how to adapt or fix it, and they would learn from doing this.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The interviewees recognized a need to rethink assessment design in computing education.

    “Overall, it was recognized that there is a need to rethink assessment design.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Several interviewees saw unauthorized use of AI tools for assessment tasks as a form of contract cheating.

    “Several interviewees saw this as a form of contract cheating:”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A particular feature of Copilot is that it can give students a starting point if they are stuck, providing a structure and sequence of code.

    “A particular feature of Copilot is that it can give students a starting point if they are stuck, providing a structure and sequence of code.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    A challenge is that contract cheating via an AI tool is difficult to detect.

    “A challenge is that contract cheating via an AI tool is difficult to detect.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Some interviewees suggested taking an educative approach by ensuring clear guidelines for the use of AI tools and requiring students to acknowledge any part of their submitted assessment work that was produced with the assistance of an AI tool.

    “Some interviewees suggested taking an educative approach by ensuring clear guidelines for the use of AI tools and requiring students to acknowledge any part of their submitted assessment work that was produced with the assistance of an AI tool.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Projects were seen by interviewees as an authentic way of assessing and as providing opportunities for students to gain experience using AI tools in a meaningful setting.

    “Projects were seen as an authentic way of assessing, and would provide opportunities for students to gain experience using the AI tools in a meaningful setting.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    An interesting use of AI tools proposed by one interviewee was an automated system to give feedback to students when they seek help.

    “An interesting use of AI tools proposed by one interviewee was an automated system to give feedback to students when they seek help.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    A student may achieve a correct solution to a problem through using an AI tool, but then have no idea how to solve a different problem without the tool.

    “A student may achieve a correct solution to a problem through using an AI tool, but then have no idea how to solve a different problem without the tool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    An innovative idea proposed by one interviewee was the creation of personalized tasks related to the students' own interests, which could help motivate the students to complete the tasks.

    “An innovative idea proposed by one interviewee was the creation of personalized tasks that are related to the students' own interests, which could help motivate the students to complete the tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The issue of student trust in AI tools was raised by several interviewees who were concerned that students would not always be able to judge the correctness of a solution produced by an AI tool and could be misled or confused by a wrong solution.

    “The issue of student trust in AI tools was raised by several interviewees who were concerned that students would not always be able to judge the correctness of a solution produced by an AI tool and could be misled or confused by a wrong solution.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Another major challenge identified by interviewees was how to teach and assess programming.

    “Another major challenge was 'how to teach and assess'.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    With the rapid advances in AI tools, changing tasks to those where AI tools are not useful was seen as difficult to achieve.

    “however, with the rapid advances in AI tools, this was seen as difficult to achieve.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Similar concerns about AI tools are shared by most of our colleagues internationally.

    “We acknowledge that similar concerns are shared by most of our colleagues internationally”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    There are calls to reconsider learning goals at course and curriculum level, and teachers are looking for novel teaching methods and assessment practices that would accommodate the change.

    “There are calls to reconsider learning goals at course and curriculum level, and teachers are looking for novel teaching methods and assessment practices that would accommodate the change.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Many interviewees emphasized that assessment should focus more on the process of programming than on the final submitted work, which is what is currently evaluated either manually or automatically in exercises, projects, and exams.

    “On a course level, many of our interviewees emphasized that assessment should focus more on the process of programming than on the final submitted work, which is what is currently evaluated either manually or automatically in exercises, projects, and exams.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Much of students' work during programming is invisible to teachers; while we can get some snapshots of what is happening in classroom sessions, we see very little of what is happening when students are working on their own.

    “The challenge has been that much of students' work during programming is invisible to teachers, and while we can get some snapshots of what is happening in classroom sessions, we see very little of what is happening when students are working on their own.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Many teachers have some experience in using AI tools based on personal exploration, but there are still no established best practices or research-based pedagogies highlighting when and how the tools should be integrated with different courses.

    “Many have some experience in using the tools based on personal exploration, but there are still no established best practices or research-based pedagogies highlighting when and how the tools should be integrated with different courses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    While formally much of current programming education states that we teach reading, writing, tracing, testing, and debugging programs, most assessment methods focus on the submitted work and its correctness and quality.

    “It is clear that while formally much of current programming education states that we teach reading, writing, tracing, testing, and debugging programs, most assessment methods focus on the submitted work and its correctness and quality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Many teachers envisage new opportunities to improve education with AI tools, providing support for both students and teachers, and this view is receiving much attention in current LLM-related research in CER.

    “However, many teachers also envisage new opportunities to improve education with these tools, providing support for both students and teachers, and this view is also receiving much attention in the current LLM-related research in CER.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    A speaker at the ITiCSE 2023 conference apologized that the results in his paper were no longer valid because a new version of GPT was more powerful than the one used, making the results dated.

    “As an example, a speaker at the ITiCSE 2023 conference started his presentation by apologizing that the results in the paper were unfortunately not longer valid, as a new version of GPT was clearly more powerful than the one used in the paper, making the results dated at best.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Tasks used for assessment are particularly problematic.

    “Tasks used for assessment are particularly problematic.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Little has been done to explore how AI tools could support the analysis of rich programming process data to identify situations where students get stuck, tinker with their program, or have clear misconceptions.

    “However, little has been done to explore how AI tools could support the analysis of such extremely rich data to identify situations where students get stuck, tinker with their program, or have clear misconceptions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The interviews revealed concerns about threats to academic integrity and challenges for the learning of novice students if they use AI tools to generate submissions and not to learn programming.

    “The interviews revealed concerns about threats to academic integrity and challenges for the learning of novice students if they use the tools to generate submissions and not to learn programming.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    There was a general concern among interviewees that most or all tasks used in CS1 can be readily solved by AI tools.

    “There was a general concern that most or all tasks used in CS1 can be readily solved by the AI tools.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Our interview data is rich in discussion of new opportunities and how programming education could or should be revised to mitigate some of the challenges and to accept that the world is now different.

    “our interview data is rich in discussion of new opportunities and how programming education could or should be revised to mitigate some of the challenges but also to accept that world is now different and we must learn to live in it.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The initial focus of discussion among computing departments has been concerns about academic integrity among students and the trivializing of many introductory course assignments, as reflected in the interviews and prior studies.

    “The initial focus of discussion among computing departments has been concerns about academic integrity among students, as well as the trivializing of many of the assignments in our introductory courses, which is reflected both in our interviews and in prior studies [19,39].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [19]

      Not checked in the free run.

    • cites [39] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Findingown data

    Most interviewees envisaged that changes will be needed to course learning outcomes because the current learning outcomes do not reflect the learning that students would be gaining through the use of AI tools.

    “Most interviewees envisaged that changes will be needed to course learning outcomes as the current learning outcomes do not reflect the learning that the students would be gaining through the use of the tool.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    At the stage when students are proficient with programming, they could use AI tools in projects in a similar way to how they would use them in their future work.

    “At this stage they could use the tools in projects in a similar way to how they would use them in their future work.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    It was proposed by interviewees that assessment tasks could be changed to those where AI tools were not useful, to avoid situations where students engage in unproductive behavior.

    “To avoid situations where students engage in such unproductive behavior, it was proposed that the tasks could be changed to those where the AI tools were not useful”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    One interviewee saw the shift toward integration and evaluation as part of the continual evolution of programming education, with a shift from low-level programming languages to higher levels of abstraction.

    “One interviewee saw this as part of the continual evolution of programming education with a shift from low-level programming languages to higher levels of abstraction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Instructors saw benefits of AI tools in providing explanations of code, helping students to find inspiration and to get feedback, and allowing access to high quality learning resources.

    “Instructors saw benefits of the AI tools in providing explanations of code, helping students to find inspiration and to get feedback, and allowing access to high quality learning resources.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Lau and Guo found that in the short term, most instructors intended to implement immediate measures to deter cheating, while the long-term plans were divided.

    “An interview of 20 instructors conducted by Lau and Guo [19] found that in the short term, most instructors intended to implement immediate measures to deter cheating, while the long-term plans were divided.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    AI tools can accurately solve homework problems and exam questions, which may lead to student overreliance.

    “One widely voiced concern is the capability of the tools to accurately solve homework problems and exam questions, which may lead to student overreliance [11,31].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11] source

      Not checked in the free run.

    • cites [31]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students said that AI tools reduced the effort of writing code, helped them to find learning materials, and allowed them to avoid busy work.

    “Students said that the AI tools reduced the effort of writing code, helped them to find learning materials, and allowed them to avoid busy work.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    All interviewees had considerable experience in teaching programming, with a range of 10 to 50 years.

    “All interviewees had considerable experience in teaching programming, with a range of 10 to 50 years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    If students acknowledge AI assistance, assessment grading can focus on the code produced by the student and there would be no plagiarism to deal with.

    “This would allow the assessment grading to focus on the code produced by the student and there would be no plagiarism to deal with.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Lau and Guo interviewed 20 instructors across nine countries, presenting a hypothetical scenario in which AI tools worked perfectly and asking how they planned to adapt.

    “Lau and Guo [19] interviewed 20 instructors across nine countries, presenting a hypothetical scenario in which AI tools worked perfectly and asking how they planned to adapt.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Twelve programming instructors were purposively selected from universities in Australia, Finland and New Zealand to capture a broad range of teaching experiences, programming contexts, and institutional practices.

    “Twelve programming instructors were purposively selected from universities in Australia, Finland and New Zealand to capture a broad range of teaching experiences, programming contexts, and institutional practices.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Novice programmers sometimes struggle to use Copilot, for example spending a lot of time unsuccessfully trying to coerce Copilot to produce specific code.

    “Novice programmers also sometimes struggle to use Copilot; for example, spending a lot of time unsuccessfully trying to coerce Copilot to produce specific code [31].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GitHub Copilot can solve around half of introductory programming exercises directly, and another 30% with some additional prompt engineering.

    “researchers have looked into the capabilities of these tools, noting that they are able to solve around half of introductory programming exercises directly, and another 30% with some additional 'prompt engineering'”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Zastudil et al. interviewed 12 students and six instructors, all from the same institution, investigating experiences and preferences for AI tools in computing classrooms.

    “Zastudil et al. [39] interviewed 12 students and six instructors, all from the same institution, investigating experiences and preferences for AI tools in computing classrooms.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students voiced concerns about over-reliance on and trustworthiness of AI tools, and noted that the availability of such tools might increase plagiarism.

    “They also voiced concerns about over-reliance on and trustworthiness of the AI tools, and noted that the availability of such tools might increase plagiarism.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains.

    “Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The study by Lau and Guo involved a hypothetical LLM that worked flawlessly, serving as an 'oracle', a level that current LLMs still appear far from attaining.

    “The study, however, involved a hypothetical LLM that worked flawlessly, serving as an 'oracle', a level that current LLMs still appear far from attaining.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

    • Rests on an assumption: Current LLMs are far from attaining a flawless 'oracle' level of performance.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    Contract cheating via an AI tool has a negative impact on the education environment and is unfair to students who spend the time to complete the task as intended but get a comparable mark to the student who used an AI tool.

    “This form of cheating has a negative impact on the education environment and is unfair to students who spend the time to complete the task as intended but get a comparable mark to the student who used an AI tool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several interview and survey studies have attempted to gauge instructors' perceptions and attitudes toward AI-enabled tools in the broader educational context.

    “Several interview and survey studies have attempted to gauge instructors' perceptions and attitudes toward AI-enabled tools in the broader educational context [2,5,16].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [2]

      Not checked in the free run.

    • cites [5]

      Not checked in the free run.

    • cites [16]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The data were analyzed following Braun and Clarke's six-step process for thematic analysis.

    “The data were analyzed following Braun and Clarke's six-step process for thematic analysis [4].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Before the interview, participants were shown a brief video of Copilot to ensure they had a shared understanding of its features and functionality.

    “Before the interview, participants were shown a brief video of Copilot to ensure they had a shared understanding of its features and functionality.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Instructors expressed concerns about over-reliance and plagiarism, and pointed out issues with the trustworthiness of the systems.

    “The instructors, too, expressed concerns about over-reliance and plagiarism, and pointed out issues with the trustworthiness of the systems.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Codex could solve around 80% of introductory programming (CS1) problems that had been included in past exams, a performance that would put Codex in the top quartile of students in the course.

    “The study found that Codex could solve around 80% of introductory programming (CS1) problems that had been included in past exams, a performance that would put Codex in the top quartile of students in the course.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Literature looking into student and instructor perspectives of large language models within computing education research is still scarce.

    “Literature looking into student and instructor perspectives of large language models within CER is still scarce, though there are a few notable exceptions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Most participants (9/12) had experience in teaching both introductory and advanced courses, while one had only taught introductory courses and two had only taught advanced courses (CS2 and above).

    “Most participants (9/12) had experience in teaching both introductory and advanced courses, while one had only taught introductory courses and two had only taught advanced courses (CS2 and above).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Some instructors were in favor of banning AI tools to maintain a focus on teaching programming fundamentals, whereas others advocated for the integration of AI tools into courses to prepare students for future job markets in which AI would be widely used.

    “Some instructors were in favor of banning AI tools to maintain a focus on teaching programming fundamentals, whereas others advocated for the integration of AI tools into courses to prepare students for future job markets in which AI would be widely used.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All interviews were audio-recorded with the prior consent of the participants and then transcribed verbatim to preserve the richness of the data.

    “All interviews were audio-recorded with the prior consent of the participants and then transcribed verbatim to preserve the richness of the data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    AI tools can be used to produce high-quality learning resources, including programming exercises, explanations of code, and on-demand feedback to students.

    “On the other hand, recent work has shown that AI tools can be used to produce high-quality learning resources, including programming exercises [32], explanations of code [24,28], and even on-demand feedback to students [27].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [32] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    AI tools could be used to generate code examples and code explanations for code reading skills.

    “Considering code reading skills, AI tools could be used to generate code examples [6] and code explanations [32].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [6]

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Perhaps only a small number of students need to learn 'real programming skills', implying that computer science would transition from common academic knowledge to an area of deep expertise.

    “Perhaps only a small number of students need to learn 'real programming skills', implying that computer science would transition from common academic knowledge to an area of deep expertise.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    AI tools could summarize students' code and documents for initial screening, and suggest comments on them which the teacher could accept, reject or modify, thus speeding up the giving of formative feedback.

    “The tools could summarize students' code and documents for initial screening, and suggest comments on them which the teacher could accept, reject or modify, thus speeding up the giving of formative feedback.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Some work has used fine-grained programming process logs from instrumented IDEs to uncover student work patterns and to visualize student programming processes.

    “Some work has looked into using fine-grained programming process logs gathered from instrumented IDEs to uncover student work patterns [21-23, 26, 40] and to visualize student programming processes [10,14,35,38].”
    Quote in the paperNot a substantial claim0 of 9 sources checked
    • cites [21]

      Not checked in the free run.

    • cites [22] source

      Not checked in the free run.

    • cites [23] source

      Not checked in the free run.

    • cites [26] source

      Not checked in the free run.

    • cites [40] source

      Not checked in the free run.

    • cites [10] source

      Not checked in the free run.

    • cites [14] source

      Not checked in the free run.

    • cites [35] source

      Not checked in the free run.

    • cites [38] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    From the teachers' perspective, AI tools can be used to generate not only examples but complete programming assignments with model solutions and test cases, thus reducing their work and supporting the production of personalized exercises for students.

    “From the teachers' perspective, AI tools can be used to generate not only examples but complete programming assignments with model solutions and test cases [32], thus reducing their work and supporting the production of personalized exercises for students.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Overall, we are still at an adaptation phase, and perhaps because of this, the future of education seems unclear.

    “Overall, we are still at an adaptation phase, and, perhaps because of this, the future of education seems unclear.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The emphasis in teaching would necessarily shift from code writing toward integration of code, code evaluation, and testing.

    “The emphasis in teaching would necessarily shift from code writing toward integration of code, code evaluation and testing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    It is easy to imagine AI-driven applications that would generate deliberately buggy code and guide students to find and fix errors, recording their actions and providing personal feedback to support their process.

    “It is easy to imagine AI-driven applications that would generate deliberately buggy code and guide students to find and fix errors, recording their actions and providing personal feedback to support their process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Large language models are advancing very fast, and results published now can be dated in a few months.

    “Additionally, large language models are advancing very fast, and results published now can be dated in a few months.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AI has already been used to improve error messages.

    “At a simpler level, AI has already been used, for example, to improve error messages [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    We may be in a phase where the skills taught in service programming courses are replaced by learning to use AI tools to carry out the necessary programming, similar to how statistical software is used without considering the underlying calculations.

    “Now, we may be in a phase where such skills are replaced by learning to use AI tools to carry out the necessary programming, in the same way that we use statistical software to perform tests without considering how the calculations are actually carried out.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Service courses in programming for non-CS degree programs have often been motivated by gaining a sufficient understanding of programming concepts and process to facilitate collaboration with professional software developers, as well as learning practical skills to write small programs to manipulate their own data.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    A rich research field is opening for us, along with a vast demand for working solutions and tools, not only in universities but also in school-level programming education.

    “A rich research field is opening for us, along with a vast demand for working solutions and tools, not only in universities but also in school level programming education.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    AI-generated code examples and explanations could be used as ready-made materials for code review sessions and could inspire new types of assignments that are automatically generated and possibly automatically assessed using techniques like 'questions about learner's code'.

    “These could be used as ready-made materials for code review sessions for students, but could also inspire new types of assignment that are automatically generated and possibly automatically assessed using, for example, the 'questions about learner's code' technique [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    AI has the potential, based on large sets of log data, to build constructive personal feedback which could be integrated into IDEs.

    “AI has the potential, based on large sets of log data, to build constructive personal feedback which could be integrated into IDEs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Even with 12 interviewees, we started seeing similar thoughts being expressed, which could signal that some level of saturation was reached.

    “However, even with this number of interviewees, we started seeing similar thoughts being expressed, which could signal that some level of saturation was reached.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    There is a risk that student use of AI tools could have a negative impact on their learning and could violate academic integrity rules if used for assessment.

    “The risk is that student use of the AI tools could have a negative impact on their learning and could violate academic integrity rules if used for assessment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    It is possible that the thoughts expressed by these academics do not accurately represent those of the larger population of computing educators.

    “It is possible that the thoughts expressed by these academics do not accurately represent those of the larger population of computing educators.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    It is likely that the views expressed by the instructors in our study reflect their thoughts on AI with the capabilities available at the time of the interviews (early 2023), and their thoughts might differ now that the capabilities have grown.

    “Similarly, it is likely that the views expressed by the instructors in our study reflect their thoughts on AI with the capabilities that were available at the time of the interviews (early 2023), and their thoughts might differ now that the capabilities have already grown.”
    Quote in the paperNot a substantial claim
92 of 92

  • The paper relies on this without support.

    Pedagogical change from AI code generation tools is inevitable.

    “If progress is to be made toward the inevitable lasting pedagogical change, there is a need for research that explores the instructor voice”

  • The paper relies on this without support.

    Current LLMs are far from attaining a flawless 'oracle' level of performance.

    “a level that current LLMs still appear far from attaining”

  • The paper relies on this without support.

    Service courses in programming for non-CS degree programs have often been motivated by gaining a sufficient understanding of programming concepts and process to facilitate collaboration with professional software developers, as well as learning practical skills to write small programs to manipulate their own data.

    “So far, service courses in programming for non-CS degree programs have often been motivated by gaining a sufficient understanding of programming concepts and process to facilitate collaboration with professional software developers, as well as learning practical skills to write small programs to manipulate their own data.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These outcomes compromise fairness, bias academic performance, and erode the integrity of assessment systems, bringing potential harm not only to individuals but also to institutions and the wider community.

    “These outcomes compromise fairness, bias academic performance, and erode the integrity of assessment systems, bringing potential harm not only to individuals but also to institutions and the wider community [316].”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [316]

    Section 4.2 reports instructors’ concerns that unauthorized AI use makes students’ understanding difficult to assess and is unfair when students who use AI receive marks comparable to those who complete the work themselves. The paper describes a negative impact on the educational environment, but does not address harm to the wider community; it presents these effects as perceived risks, not measured outcomes. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • GenAI technologies could inadvertently lead to academic laziness and overreliance if not carefully managed.

    “These technologies could inadvertently lead to academic laziness and overreliance if not carefully managed [11,14].”

    From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [11]

The citation lists haven’t been fetched yet

References 28 · Cited by 54

Research works printed but not linked: 4

  1. Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanp"a"a, and Juha Sorva. 2023. Exploring the responses of large language models to beginner programmers' help requests. arXiv preprint arXiv:2306.05715 (2023). · arXiv
  2. Jaeho Jeon and Seongyong Lee. 2023. Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies (2023), 1--20.
  3. Sam Lau and Philip J Guo. 2023. From “ban it till we wnderstand it” to “resistance is futile”: how university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In ICER 2023.
  4. Juho Leinonen. 2019. Keystroke Data in Programming Courses. Ph.,D. Dissertation. University of Helsinki.

Authors
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon
Affiliations
  • Monash University (AU)
  • University of Auckland (NZ)
  • Aalto University (FI)
  • Unaffiliated, Wadalba, Australia
ORCID
Venue
Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
Publisher
ACM
Published
2024-03-07
Pages
1223-1229
DOI
10.1145/3626252.3630880
OpenAlex
W4392564770
Semantic Scholar
1f7228550f2177c83d5fb06a7fef555fe83b9bfb
Cited by
54
References
28
Citations by year
2026: 12 · 2025: 32 · 2024: 10
FWCI
23.69
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
Funders
Ulla Tuominen Foundation
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
large language models, instructor perceptions, computing education, programming education, pedagogical change, research-practice dialogue

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review