CoChat
Sign Up
Cited by 75Artificial Intelligence in EducationTeaching and Learning ProgrammingProblem and Project Based Learning

Incorporating Generative AI into Software Development Education

Open access
Petrovska, Olga, Clift, Lee, Moller, Faron, Pearsall, Rebecca·2024·Proceedings of the 8th Conference on Computing Education Practice
DOI 10.1145/3633053.3633057
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper explores how Generative AI can be incorporated into software development education. We present examples of formative and summative assessments, which explore various aspects of ChatGPT, including its coding capabilities, its ability to construct arguments as well as ethical issues of using ChatGPT and similar tools in education and the workplace. Our work is inspired by the insights from surveys that show that the learners on our Degree Apprenticeship Programme have a great interest in learning about and exploiting emerging AI technology. Similarly, our industrial partners have a clear interest for their employees to be formally prepared to use GenAI in their software engineering roles. In this vein, it is proposed that embedding the use of GenAI tools in a careful and creative way - by developing assessments which encourage learners to critically evaluate AI output - can be beneficial in helping learners understand the subject material being taught without the risk of the AI tools “doing the homework”.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper describes embedding ChatGPT/GenAI into formative and summative assessments in an Applied Software Engineering Degree Apprenticeship Programme, using surveys of learners and industry professionals plus experimental coding and argumentation tasks. It reports generally positive learner and industry attitudes, with learners able to identify both strengths and limitations of GenAI, while noting concerns about over-reliance, trustworthiness, privacy, and ethics.

Key findings

  • Surveys covered first-year learners (12 respondents), final-year learners (12), and industry professionals (13). All but two first-year respondents agreed ChatGPT generated a working Java program requiring little to no modification; among final-year learners, 50% said generated output needed heavy modifications, everyone felt it was worse than their own app, and average estimated development time fell from 17.1 hours without ChatGPT to 10 hours with it. Only 2 of 13 industry respondents reported attempts to integrate GenAI, while 30.8% considered it definitely useful and 61.5% potentially useful; over 80% of all participants showed willingness to be trained or have staff trained.
  • In the summative assessment with 17 first-year learners, 94% generated Java code with their first prompt that compiled, and 89% felt it was equal to or better than their own code; 53% felt ChatGPT refactored code correctly.
  • For Pep/8 Assembly, 29% generated some compilable code, no learner felt it was better than their own, and only 2 learners (11%) felt it was equal; 71% either could not generate compilable Pep/8 code or got Python code instead. For a text argument on cryptocurrency sustainability, 76% were critical of ChatGPT’s arguments, rising to 81% after refinement.
  • 62.5% felt ChatGPT could make limited or well-reasoned factual arguments, 68% felt it could generate reasonable code, and 62.5% responded positively about workplace use; the 37.5% who did not feel it would be useful had prior neutral or bad code-generation experiences, and almost all learners had concerns about privacy and ethics.

Methods

  • The study used three anonymous online surveys: first-year learners (12), final-year learners (12), and industry professionals (13), followed by experimental formative and summative assessments in the Applied Software Engineering Degree Apprenticeship Programme.
  • Formative assessments had first-year learners analyse pre-generated ChatGPT Java code for a Stack class against their own code, while final-year learners prompted ChatGPT to generate a Dart Snooker Scoring app, analysed it against their own solution, rated it on a Likert scale, marked it with the original rubric, and estimated time savings.
  • The summative assessment involved 17 first-year learners completing tasks on Java and Pep/8 Assembly code generation, argumentation about cryptocurrency sustainability and “A is better than B,” and ethical/professional implications; results were analysed with a coding scheme to gauge sentiment and GenAI usability.

Limitations

  • The paper does not explicitly state a limitations section; it describes the assessments as experimental and future work as incorporating GenAI into more modules.
  • Evident from the text: the samples are small and drawn from a single Degree Apprenticeship Programme, with 12 first-year learners, 12 final-year learners, 13 industry professionals, and 17 learners in the summative assessment.
  • Evident from the text: the study relies on self-reported survey and assessment responses, lacks a control or comparison condition, and reports short-term experimental assessments; the paper also notes an example of over-reliance where one learner failed to notice generated code was not in Pep/8.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3633053.3633057

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 8 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Compared to generating Java code, first-year learners struggled to get ChatGPT to generate Pep/8 Assembly code.

    “Compared to generating Java code, the learners struggled to get ChatGPT to generate Pep/8 Assembly code.”
    Quote in the paperSubstantial claim
  • Findingown data

    Learners on the Degree Apprenticeship Programme have a great interest in learning about and exploiting emerging AI technology.

    “Our work is inspired by the insights from surveys that show that the learners on our Degree Apprenticeship Programme have a great interest in learning about and exploiting emerging AI technology.”
    Quote in the paperSubstantial claim
  • Findingown data

    A majority of the surveyed participants see a benefit in formal training on the use of GenAI tools as an aid in software engineering.

    “A majority of the surveyed participants see a benefit in formal training on the use of GenAI tools as an aid in software engineering.”
    Quote in the paperSubstantial claim
  • Findingown data

    Everyone in the final-year survey believed that they would have benefitted from using ChatGPT during their coursework assignment, specifically by saving time.

    “At the same time, everyone believed that they would have benefitted from using ChatGPT during their coursework assignment, specifically by saving time.”
    Quote in the paperNot a substantial claim
  • Findingown data

    There was no correlation between people's exposure to software development and the way in which the ChatGPT output was rated.

    “Interestingly, there was no correlation between people's exposure to software development and the way in which the output was rated.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 19
  • The quote only partly states this claim. 3

  • May combine two claims.

    Findingown data

    Final-year respondents believe ChatGPT would be useful for refactoring or writing initial small bits of code, but not a whole program.

    “They believe it would be useful for refactoring or writing initial small bits of code, but not a whole program.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The remaining 71% of first-year learners either couldn't generate compilable Pep/8 code or their prompts were not specific enough and generated Python code instead.

    “The remaining 71% of learners either couldn't generate compilable Pep/8 code or their prompts were not specific enough and generated Python code instead.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Respondents noted that ChatGPT's output did not include comments or follow coding conventions.

    “did not include comments or follow coding conventions;”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Many learners reported that the arguments made by ChatGPT were either vague or that it wasn't a balanced discussion, and the generated discussion was one-sided.

    “Many of them reported that the arguments made were either vague or that it wasn't a balanced discussion, and the generated discussion was one-sided.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    When asked to reflect on their experience, 62.5% of first-year learners felt that ChatGPT could make either limited or well-reasoned factual arguments, in many cases providing reasonable references and discussing a topic in depth.

    “When asked to reflect on their experience, 62.5% of learners felt that ChatGPT could make either limited or well-reasoned factual arguments, in many cases providing reasonable references and discussing a topic in depth.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The coding scheme used to analyse the summative assessment results reliably gauges sentiment and GenAI usability.

  • May combine two claims.

    Findingown data

    Most final-year respondents managed to get a running program of varied quality, with 50% stating that it needed heavy modifications.

    “Most of the respondents managed to get a running program of varied quality, with 50% of the respondents stating that it needed heavy modifications.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Many learners felt that the generated text wasn't sufficient for a constructive argument, and would require additional work and editing.

    “Many learners felt that the generated text wasn't sufficient for a constructive argument, and would require additional work and editing.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When refactoring their generated code, 53% of first-year learners felt that it refactored correctly, and ChatGPT could implement additional features, or modify existing ones.

    “When refactoring their generated code, 53% of learners felt that it refactored correctly, and ChatGPT could implement additional features, or modify existing ones.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Learners' ratings of ChatGPT output quality are a valid measure of its actual capabilities.

  • May combine two claims.

    Findingown data

    In other cases, code either did not compile after refactoring or learners felt that the newly refactored code didn't reflect the prompt they had given.

    “In other cases, code either did not compile after refactoring or learners felt that the newly refactored code didn't reflect the prompt they had given.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When generating Java code via ChatGPT, 94% of first-year learners generated code with their first prompt which compiled, and 89% of them felt that it was equal to or better than the code they wrote themselves.

    “When generating Java code via ChatGPT, 94% of learners generated code with their first prompt which compiled, and 89% of them felt that it was equal to or better than the code they wrote themselves in their attempt.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Learners' ratings of ChatGPT output quality are a valid measure of its actual capabilities.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Improved efficiency, new ideas and solutions, and a pre-generated 'skeleton' code were mentioned among the benefits of ChatGPT as a support tool.

    “Improved efficiency, new ideas and solutions, and a pre-generated 'skeleton' code were mentioned among the benefits.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    When tasked with generating a text-based argument regarding the sustainability of cryptocurrencies, most first-year learners (76%) were critical of the arguments made by ChatGPT.

    “When tasked with generating a text-based argument regarding the sustainability of cryptocurrencies, most learners (76%) were critical of the arguments made by ChatGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Everyone in the final-year survey felt that the generated output was worse than their own previously developed app and that ChatGPT struggled with specific features, requiring further prompts to improve the code.

    “Everyone felt that the generated output was worse than their own previously developed app and that ChatGPT struggled with specific features, requiring further prompts to improve the code.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is a clear trend among industry professionals of seeing GenAI as useful in improving a work life cycle, with 30.8% considering it definitely useful and 61.5% considering it potentially useful.

    “However, there is a clear trend of seeing GenAI as useful in improving a work life cycle, with 30.8% of the respondents considering it definitely useful and 61.5% of the respondents considering it potentially useful.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Three-quarters of the final-year learners feel that ChatGPT would be useful in their full-time job by saving time and offering alternative solutions to programming tasks.

    “Three-quarters of the final-year learners feel that ChatGPT would be useful in their full-time job by saving time and offering alternative solutions to programming tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Industrial partners have a clear interest for their employees to be formally prepared to use GenAI in their software engineering roles.

    “Similarly, our industrial partners have a clear interest for their employees to be formally prepared to use GenAI in their software engineering roles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average development time without ChatGPT was estimated by final-year participants to be 17.1 hours, and the estimate for development time with ChatGPT was 10 hours.

    “The average development time without ChatGPT was estimated to be 17.1 hours, and the estimate for development time with ChatGPT was 10 hours.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Learners' self-reported estimates of development time with and without ChatGPT are accurate.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT's output was described as 'more efficient and better styled'.

    “ChatGPT's output was described as”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    When asked if they felt that GenAI should be used within the workplace, 62.5% of first-year learners responded positively, stating they feel that when used correctly, this could increase productivity and make code generation easier.

    “When asked if they felt that GenAI should be used within the workplace, 62.5% responded positively, stating they feel that when used correctly, this could increase productivity and make code generation easier, a similar statistic to the one seen in the previously conducted surveys.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Many learners felt that the writing style of ChatGPT was very artificial.

    “However, many learners felt that the writing style was very artificial.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Half of first-year respondents agreed that ChatGPT would have been beneficial as a support tool in their programming module, while a third opted for 'maybe'.

    “When asked about the benefits of ChatGPT as a support tool, half agreed that it would have been beneficial as a support tool in their programming module, while a third opted for the 'maybe' option.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    29% of first-year learners were able to generate some compilable Pep/8 code, although no learner felt that the generated code was better than what they could write themselves, and only 2 learners (11%) felt that it was equal to what they would write.

    “Results show that 29% of learners were able to generate some compilable Pep/8 code, although no learner felt that the generated code was better than what they could write themselves, and only 2 learners (11%) felt that it was equal to what they would write.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Almost all first-year learners had a level of concern when considering using GenAI in the workplace, mostly around privacy and ethical considerations.

    “Almost all learners had a level of concern when considering using GenAI in the workplace, mostly around privacy and ethical considerations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Apprentices expressed enjoyment in discussing and using GenAI in their summative assessments, and especially the more reluctant users applauded the opportunity to explore technology which they had either dismissed or felt was beyond their skill level.

    “They expressed enjoyment in discussing and using GenAI in their summative assessments, and -especially the more reluctant usersapplauded the opportunity to explore technology which they had either dismissed or felt was beyond their skill level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Peng et al. observed a 55.8% reduction in development time in their controlled experiment on the impact of GitHub Copilot on software developer's productivity.

    “Peng et al. [13] explored the impact of GitHub Copiloton software developer's productivity, observing a 55.8% reduction in development time in their controlled experiment.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Due to ChatGPT being a generalised LLM, the behaviour of failing to refactor correctly is not surprising and can become more common the more complex the refactoring request.

    “Due to ChatGPT being a generalised LLM, this behaviour is not surprising and can become more common the more complex the refactoring request.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Embedding the use of GenAI tools in a careful and creative way-by developing assessments which encourage learners to critically evaluate AI output-can be beneficial in helping learners understand the subject material being taught without the risk of the AI tools 'doing the homework'.

    “it is proposed that embedding the use of GenAI tools in a careful and creative way -by developing assessments which encourage learners to critically evaluate AI output -can be beneficial in helping learners understand the subject material being taught without the risk of the AI tools”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Embedding GenAI tools in assessments in a careful and creative way prevents the AI tools from 'doing the homework'.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Biderman and Raff discovered that plagiarism was not detected by MOSS for Java code generated by GPT-J, although they do not have evidence that this would be the case for tasks in more advanced programming courses.

    “Biderman and Raff [2] explored the performance of Java code generated by GPT-J on MOSS (measure of software similarity tool) and discovered that plagiarism was not detected, although they do not have evidence that this would be the case for tasks in more advanced programming courses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Barke et al. observed occasional over-reliance of first-time users in their study of how programmers interact with Copilot.

    “Barke et al. [1] conducted a study with 20 programmers to explore how they interact with Copilot, and observed occasional over-reliance of first-time users.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationown data

    With such a large majority of learners finding code generation useful, this supports the argument for using GenAI as a teaching aid.

    “With such a large majority of learners finding code generation useful, this supports the argument for using GenAI as a teaching aid.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Learners' ratings of ChatGPT output quality are a valid measure of its actual capabilities.
    • Rests on an assumption: ChatGPT's performance on the specific tasks in the assessments is indicative of its general capabilities in software development education.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Denny et al. found that Copilot solved half of CS1 Python programming tasks on its very first attempt.

    “Denny et al. [4] investigated Copilot's performance solving CS1 Python programming tasks and found that it solved half of them on its very first attempt.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    43% of those who used GenAI tools assume outright that the information it provides is factual and 38% consider it to be unbiased.

    “Disturbingly, 43% of those who used it assume outright that the information it provides is factual and 38% consider it to be unbiased.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    From this response, it can be understood that many learners may not feel comfortable to use ChatGPT for large amounts of nuanced text generation, as it is reported to do a worse job than what many of them could do by themselves.

    “From this response, it can be understood that many learners may not feel comfortable to use ChatGPT for large amounts of nuanced text generation, as it is reported to do a worse job than what many of them could do by themselves.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Finnie-Ansley et al. found that OpenAI's Codex solves the majority of CS2 programming exercises in Python.

    “Finnie-Ansley et al. [6] found that OpenAI's Codex solves the majority of CS2 programming exercises in Python.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The failure of one learner to notice that the code was not in Pep/8 is an example of over-reliance.

    “In fact, one of the learners even failed to notice that the code was not in Pep/8, which is an example of over-reliance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Zastudil et al. observed a positive attitude with regard to integrating GenAI into the educational process as well as concerns about over-reliance, trustworthiness and academic integrity.

    “Zastudil et al. [20] interviewed computer science students and instructors to understand their interaction with GenAI and their expectations regarding its use in education.They observed a positive attitude with regard to integrating GenAI into the educational process as well as concerns about over-reliance, trustworthiness and academic integrity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    According to Deloitte's Digital Consumer Trends 2023 survey, 52% of people in the UK are aware of GenAI tools, and half of these have used it.

    “According to Deloitte's Digital Consumer Trends 2023 survey [12], 52% of people in the UK are aware of such GenAI tools, and half of these have used it.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Results show that when employed in formative and summative assessments, students can use GenAI to complete tasks, and can easily identify positives and negatives regarding this new technology.

    “Results show that when employed in formative and summative assessments, students can use GenAI to complete tasks, and can easily identify positives and negatives regarding this new technology.”
    Quote in the paperSubstantial claim
38 of 38

  • The paper relies on this without support.

    Learners' self-reported estimates of development time with and without ChatGPT are accurate.

    “we asked participants to estimate the amount of time that they spent on the programming task initially, and to provide an estimate on how long it would take them if they were to use ChatGPT.”

  • The paper relies on this without support.

    Learners' ratings of ChatGPT output quality are a valid measure of its actual capabilities.

    “evaluate the quality of the pre-generated code using a 5 point Likert scale.”

  • The paper relies on this without support.

    The coding scheme used to analyse the summative assessment results reliably gauges sentiment and GenAI usability.

    “The results of the assessment were analysed through the creation and use of a coding scheme, to gauge sentiment towards different tasks, and GenAI usability.”

  • The paper relies on this without support.

    Embedding GenAI tools in assessments in a careful and creative way prevents the AI tools from 'doing the homework'.

    “it is proposed that embedding the use of GenAI tools in a careful and creative way -by developing assessments which encourage learners to critically evaluate AI output -can be beneficial in helping learners understand the subject material being taught without the risk of the AI tools”

  • The paper relies on this without support.

    ChatGPT's performance on the specific tasks in the assessments is indicative of its general capabilities in software development education.

    “Results show that when employed in formative and summative assessments, students can use GenAI to complete tasks, and can easily identify positives and negatives regarding this new technology.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arxiv:2107.03374 [cs.LG] · arXiv
  2. Giuseppe Destefanis Silvia Bartolucci and Marco Ortu. 2023. A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions. (2023). arxiv:2305.09402 [cs.SE] · arXiv
  3. Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora Ethan Guo Collin Burns Samir Puranik Horace He Dawn Song and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. (2021). arxiv:2105.09938 [cs.SE] · arXiv
  4. Deloitte LLP. 2023. Digital Consumer Trends 2023. Retrieved October 3, 2023 from https://www2.deloitte.com/uk/en/pages/technology-media-and-telecommunications/articles/digital-consumer-trends.html
  5. Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. (2023). arxiv:2302.06590 [cs.SE] · arXiv

Cited by

Sorted by citations

Related

Authors
Petrovska, O., Clift, L., Moller, F., & Pearsall, R.
Affiliations
  • Swansea University (GB)
  • The Open University (GB)
ORCID
Venue
Proceedings of the 8th Conference on Computing Education Practice
Publisher
ACM
Published
2024-01-05
Pages
37-40
DOI
10.1145/3633053.3633057
OpenAlex
W4390427686
Semantic Scholar
f51489c94d71271a98512c8b214aaa599cc0c059
Cited by
75
References
14
Citations by year
2026: 23 · 2025: 39 · 2024: 13
FWCI
19.28
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Computing Education Practice (CEP) (publishedVersion, cc-by)
  • Cronfa (Swansea University) (submittedVersion, cc-by)
  • Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) (submittedVersion)
Funders
Swansea University
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Problem and Project Based Learning
Keywords
generative AI, software engineering education, ChatGPT, formative assessment, summative assessment, AI ethics, critical evaluation of AI outputs, degree apprenticeships, industrial training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review