CoChat
Sign Up
PreprintCited by 9Artificial Intelligence in EducationTeaching and Learning ProgrammingArtificial Intelligence in Healthcare and Education

The Robots are Here: Navigating the Generative AI Revolution in Computing Education

Open access
Prather, James, Denny, Paul, Leinonen, Juho, Becker, Brett A., Albluwi, Ibrahim, Craig, Michelle, Keuning, Hieke, Kiesler, Natalie +7 more·2023·arXiv
DOI 10.48550/arxiv.2310.00658arXiv 2310.00658
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.00658

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 148 · Cited by 9

Research works printed but not linked: 127

  1. n.d.]. Authorship and contributorship. https://www.cambridge.org/core/ services/authors/publishing-ethics/research-publishing-ethics-guidelines- for-journals/authorship-and-contributorship#ai-contributions-to-research- content Accessed: 08 July 2023.
  2. n.d.]. Best Practice Guidelines on Research Integrity and Publishing Ethics. https://authorservices.wiley.com/ethics-guidelines/index.html#5 Accessed: 08 July 2023.
  3. n.d.]. Publishing Ethics | Elsevier Policy. https://beta.elsevier.com/about/ policies-and-standards/publishing-ethics?trial=true#4-duties-of-authors Ac- cessed: 08 July 2023.
  4. 2019. AAAI Code of Professional Ethics and Conduct. https://aaai.org/about- aaai/ethics-and-diversity/. Accessed: 08 July 2023.
  5. 2020. IEEE Code of Ethics. https://www.ieee.org/about/corporate/governance/ p7-8.html Accessed: 08 July 2023.
  6. 2023. ACM Code of Ethics. https://www.acm.org/code-of-ethics Accessed: 08 July 2023.
  7. 2023. ACM Policy on Authorship. https://www.acm.org/publications/policies/ new-acm-policy-on-authorship Accessed: 08 July 2023.
  8. ACM/IEEE-CS Joint Task Force on Computing Curricula. 2013. Computer Science Curricula 2013. Technical Report. ACM Press and IEEE Computer Society Press. https://doi.org/10.1145/2534860 · DOI
  9. Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Maha- jan, and Mohsen Dorodchi. 2023. Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). 709-726.
  10. Ibrahim Albluwi. 2019. Plagiarism in Programming Assessments: A Systematic Review. ACM Trans. Comput. Educ. 20, 1, Article 6 (Dec. 2019), 28 pages. https: //doi.org/10.1145/3371156 · DOI · Paper page
  11. Joe Michael Allen, Frank Vahid, Alex Edgcomb, Kelly Downey, and Kris Miller. 2019. An Analysis of Using Many Small Programs in CS1 (SIGCSE '19). ACM, NY, NY, USA, 585-591. https://doi.org/10.1145/3287324.3287466 · DOI · Paper page
  12. Pedro Alves and Bruno Pereira Cipriano. 2023. The centaur programmer -How Kasparov's Advanced Chess spans over to the software development of the future. arXiv:2304.11172 [cs.HC] · arXiv
  13. Sara Amani, Lance White, Trini Balart, Laksha Arora, Dr. Kristi J. Shryock, Dr. Kelly Brumbelow, and Dr. Karan L. Watson. 2023. Generative AI Percep- tions: A Survey to Measure the Perceptions of Faculty, Staff, and Students on Generative AI Tools in Academia. arXiv:2304.14415 [cs.HC] · arXiv
  14. Mikko Apiola, Sonsoles López-Pernas, and Mohammed Saqr. 2023. Past, Present and Future of Computing Education Research: A Global Perspective. Springer Nature.
  15. Lena Armstrong, Jayne Everson, and Amy J. Ko. 2023. Navigating a Black Box: Students' Experiences and Perceptions of Automated Hiring. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (Chicago, IL, USA) (ICER '23 V1). Association for Computing Machinery, New York, NY, USA, 148-158. https://doi.org/10.1145/3568813.3600123 · DOI
  16. Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Arjun Guha, Molly Q Feldman, and Carolyn Jane Anderson. 2023. StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code. arXiv preprint arXiv:2306.04556 (2023). · arXiv
  17. Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil War- riem, and Prajish Prasad. 2023. Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Fin- land) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 292-298. https://doi.org/10.1145/3587102.3588852 · DOI · Paper page
  18. Yeting Bao and Hadi Hosseini. 2023. Mind the Gap: The Illusion of Skill Ac- quisition in Computational Thinking. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 778-784. https://doi.org/10.1145/3545945.3569749 · DOI
  19. Brett Becker. 2017. Artificial Intelligence in Education: What Is It, Where Is It Now, Where Is It Going. Ireland's Yearbook of Education 2018 (2017), 42-46.
  20. Brett Becker, James Prather, Paul Denny, Andrew Luxton-Reilly, James Finnie- Ansley, and Eddie Antonio Santos. 2023. Programming Is Hard -Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation (SIGCSE '23). ACM.
  21. Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bou- vier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter- Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler Er- ror Messages Considered Unhelpful: The Landscape of Text-Based Program- ming Error Message Research (ITiCSE-WGR '19). ACM, NY, NY,USA, 177-210. https://doi.org/10.1145/3344429.3372508 · DOI · Paper page
  22. Brett A. Becker and Thomas Fitzpatrick. 2019. What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 1011-1017. https://doi.org/10.1145/3287324.3287485 · DOI · Paper page
  23. Brett A. Becker and Keith Quille. 2019. 50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapo- lis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 338-344. https://doi.org/10.1145/3287324.3287432 · DOI · Paper page
  24. John B Biggs and Kevin F Collis. 2014. Evaluating the Quality of Learning: The SOLO Raxonomy (Structure of the Observed Learning Outcome). Academic Press.
  25. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The Values Encoded in Machine Learning Research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 173-184. https://doi.org/10.1145/3531146. 3533083 · DOI · Paper page
  26. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs.LG] · arXiv
  27. Robert W Brennan and Jonathan Lesage. 2022. Exploring the Implications of Openai Codex on Education for Industry 4.0. In International Workshop on Service Orientation in Holonic and Multi-Agent Manufacturing. Springer, 254-266.
  28. Neil C. C. Brown, Amjad Altadmri, Sue Sentance, and Michael Kölling. 2018. Blackbox, Five Years On: An Evaluation of a Large-Scale Programming Data Col- lection Project. In Proceedings of the 2018 ACM Conference on International Com- puting Education Research (Espoo, Finland) (ICER '18). Association for Comput- ing Machinery, New York, NY, USA, 196-204. https://doi.org/10.1145/3230977. 3230991 · DOI
  29. Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A Large Scale Repository of Novice Programmers' Activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (Atlanta, Georgia, USA) (SIGCSE '14). Association for Computing Machinery, New York, NY, USA, 223-228. https://doi.org/10.1145/2538862.2538924 · DOI
  30. Peter Brusilovsky, Barbara J. Ericson, Cay S. Horstmann, Christian Servin, Frank Vahid, and Craig Zilles. 2023. The Future of Computing Education Materi- als. https://csed.acm.org/wp-content/uploads/2023/03/Educational-Materials- First-Draft-1.pdf First Draft, to be published in the CS2023: ACM/IEEE- CS/AAAI Computer Science Curricula.
  31. Christopher Bull and Ahmed Kharrufa. 2023. Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it. IEEE Software (2023).
  32. Cecilia Ka Yuk Chan and Louisa H. Y. Tsi. 2023. The AI Revolution in Education: Will AI Replace or Assist Teachers in Higher Education? arXiv:2305.01185 [cs.CY] · arXiv
  33. Bruno Pereira Cipriano and Pedro Alves. 2023. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 61-67. https://doi.org/10.1145/3587102.3588814 · DOI
  34. Alison Clear, Allen Parrish, John Impagliazzo, Pearl Wang, Paolo Ciancarini, Ernesto Cuadros-Vargas, Stephen Frezza, Judith Gal-Ezer, Arnold Pears, Shingo Takada, Heikki Topi, Gerrit van der Veer, Abhijat Vichare, Les Waguespack, and Ming Zhang. 2020. Computing Curricula 2020 Paradigms for Global Computing Education. ACM, New York.
  35. Stephen Cooper, Wanda Dann, and Randy Pausch. 2003. Teaching Objects-First in Introductory Computer Science. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (Reno, Navada, USA) (SIGCSE '03). Association for Computing Machinery, New York, NY, USA, 191-195. https: //doi.org/10.1145/611892.611966 · DOI
  36. Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jiang. 2023. Github copilot ai pair pro- grammer: Asset or liability? Journal of Systems and Software (2023), 111734.
  37. Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Zhen Ming (Jack) Jiang. 2022. GitHub Copilot AI Pair Programmer: Asset or Liability? https://doi.org/10.48550/arXiv.2206.15331 arXiv:2206.15331 [cs] · DOI · Paper page
  38. Nell B. Dale. 2006. Most Difficult Topics in CS1: Results of an Online Survey of Educators. SIGCSE Bull. 38, 2 (jun 2006), 49-53. https://doi.org/10.1145/ 1138403.1138432 · DOI
  39. Adrian de Freitas, Joel Coffman, Michelle de Freitas, Justin Wilson, and Troy Weingart. 2023. FalconCode: A Multiyear Dataset of Python Code Samples from an Introductory Computer Science Course. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 938-944. https://doi.org/10.1145/3545945.3569822 · DOI · Paper page
  40. Liliya A Demidova, Elena G Andrianova, Peter N Sovietov, and Artyom V Gor- chakov. 2023. Dataset of Program Source Codes Solving Unique Programming Exercises Generated by Digital Teaching Assistant. Data 8, 6 (2023), 109.
  41. Paul Denny, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Ba- naszkiewicz. 2019. Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer. In Proceedings of the 2019 ACM Conference on International Computing Education Research (Toronto ON, Canada) (ICER '19). Association for Computing Machinery, New York, NY, USA, 259-267. https://doi.org/10.1145/3291279.3339402 · DOI
  42. Paul Denny, Brett A. Becker, Juho Leinonen, and James Prather. 2023. Chat Overflow: Artificially Intelligent Models for Computing Education -RenAIs- sance or ApocAIypse?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). ACM, NY, NY, USA, 3-4. https://doi.org/10.1145/3587102.3588773 Video: www.youtube.com/watch?v=KwVcRXQc3IU. · DOI
  43. Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. 2023. Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources. arXiv:2306.10509 [cs.HC] · arXiv
  44. Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Lan- guage. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1136-1142. https://doi.org/10.1145/3545945. 3569823 · DOI · Paper page
  45. Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators. arXiv:2307.16364 [cs.HC] · arXiv
  46. Paul Denny, James Prather, Brett A. Becker, Zachary Albrecht, Dastyni Loksa, and Raymond Pettit. 2019. A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming. In Proceedings of the 19th Koli Calling Interna- tional Conference on Computing Education Research (Koli, Finland) (Koli Calling '19). Association for Computing Machinery, New York, NY, USA, Article 11, 10 pages. https://doi.org/10.1145/3364510.3366170 · DOI
  47. Catherine D'ignazio and Lauren F Klein. 2020. Data Feminism. MIT press.
  48. Felix Dobslaw and Peter Bergh. 2023. Experiences with Remote Examination Formats in Light of GPT-4. arXiv preprint arXiv:2305.02198 (2023). · arXiv
  49. Augie Doebling and Ayaan M. Kazerouni. 2021. Patterns of Academic Help- Seeking in Undergraduate Computing Students. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research (Joensuu, Finland) (Koli Calling '21). Association for Computing Machinery, New York, NY, USA, Article 13, 10 pages. https://doi.org/10.1145/3488042.3488052 · DOI
  50. Thomas Dohmke, Marco Iansiti, and Greg Richards. 2023. Sea Change in Soft- ware Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle. arXiv:2306.15033 [econ.GN] · arXiv
  51. Stefania Druga and Nancy Otero. 2023. Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for Families. arXiv:2305.10417 [cs.HC] · arXiv
  52. Joseph V. Elarde and Fatt-Fei Chong. 2011. Introductory Computing Course Content: Educator and Student Perspectives. In Proceedings of the 2011 Confer- ence on Information Technology Education (West Point, New York, USA) (SIG- ITE '11). Association for Computing Machinery, New York, NY, USA, 55-60. https://doi.org/10.1145/2047594.2047610 · DOI
  53. Neil A. Ernst and Gabriele Bavota. 2022. AI-Driven Development Is Here: Should You Worry? IEEE Software 39, 2 (mar 2022), 106-110. https://doi.org/10.1109/ ms.2021.3133805 · DOI · Paper page
  54. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE '22). Association for Computing Machinery, New York, NY, USA, 10-19. https: //doi.org/10.1145/3511861.3511863 · DOI · Paper page
  55. James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI Wants to Know If This Will Be on the Exam: Testing OpenAI's Codex on CS2 Programming Exercises (ACE '23). ACM, NY, NY, USA, 97-104. https://doi.org/10.1145/3576123.3576134 · DOI · Paper page
  56. Kathi Fisler. 2014. The Recurring Rainfall Problem (ICER '14). ACM, NY, NY, USA, 35-42. https://doi.org/10.1145/2632320.2632346 · DOI
  57. Norbert Forman, József Udvaros, and Mihály Szilárd Avornicului. 2023. ChatGPT: A new study tool shaping the future for high school students. International Journal of Advanced Natural Sciences and Engineering Researches 7, 4 (May 2023), 95-102. https://doi.org/10.59287/ijanser.562 · DOI
  58. Pavel Gherciu. 2022. Net impact of large language models trained on code. In Conferinţa tehnico-ştiinţifică a studenţilor, masteranzilor şi doctoranzilor, Vol. 1. 189-192.
  59. Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, Vol. 31.
  60. Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests. arXiv preprint arXiv:2306.05715 (2023). · arXiv
  61. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. NeurIPS (2021).
  62. Maurice Isserman. 2003. Plagiarism: A Lie of the Mind. The Chronicle Review 49 (2003). Issue 34. http://chronicle.com
  63. Brandon Jaipersaud, Paul Zhang, Jimmy Ba, Andrew Petersen, Lisa Zhang, and Michael R. Zhang. 2023. Decomposed Prompting to Answer Questions on a Course Discussion Board. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Ning Wang, Genaro Rebolledo- Mendez, Vania Dimitrova, Noboru Matsuda, and Olga C. Santos (Eds.). Springer Nature Switzerland, Cham, 218-223.
  64. Deborah G Johnson and Mario Verdicchio. 2023. Ethical AI is not about AI. Commun. ACM 66, 2 (2023), 32-34.
  65. Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 455, 23 pages. https://doi.org/10.1145/3544548.3580919 · DOI · Paper page
  66. Arshia Khan and Janna Madden. 2018. Active learning: a new assessment model that boost confidence and learning while reducing test anxiety. International Journal of Modern Education and Computer Science 10, 12 (2018), 1.
  67. Natalie Kiesler. 2020. On Programming Competence and Its Classification. In Proceedings of the 20th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '20). Association for Computing Machinery, New York, Article 1, 10 pages. https://doi.org/10.1145/3428029. 3428030 · DOI
  68. Natalie Kiesler. 2022. Kompetenzförderung in der Programmierausbildung durch Modellierung von Kompetenzen und informativem Feedback. Dissertation. Johann Wolfgang Goethe-Universität, Frankfurt am Main. Fachbereich Informatik und Mathematik.
  69. Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023. Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. https: //doi.org/10.48550/arXiv.2309.00029 arXiv:2309.00029 [cs.AI] · DOI
  70. Natalie Kiesler and Daniel Schiffner. 2022. On the Lack of Recognition of Software Artifacts and IT Infrastructure in Educational Technology Research. In 20. Fachtagung Bildungstechnologien (DELFI), Peter A. Henning, Michael Striewe, and Matthias Wölfel (Eds.). Gesellschaft für Informatik e.V., Bonn, 201-206. https://doi.org/10.18420/delfi2022-034 · DOI
  71. Natalie Kiesler and Daniel Schiffner. 2023. Large Language Models in Intro- ductory Programming Education: ChatGPT's Performance and Implications for Assessments. In CoRR abs/2308.08572. https://doi.org/10.48550/arXiv.2308.08572 arXiv:2308.08572 [cs.SE] · DOI
  72. Natalie Kiesler and Daniel Schiffner. 2023. Why We Need Open Data in Computer Science Education Research. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education Vol. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA. https: //doi.org/10.1145/3587102.3588860 · DOI
  73. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Ad- vances in Neural Information Processing Systems 32 (2019).
  74. Celine Latulipe, N Bruce Long, and Carlos E Seminario. 2015. Structuring flipped classes with lightweight teams and gamification. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 392-397.
  75. Sam Lau and Philip J. Guo. 2023. From "Ban It TillWe Understand It" to "Re- sistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot (ICER '23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3568813.3600138 · DOI · Paper page
  76. Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course. arXiv preprint arXiv:2210.14494 (2022). · arXiv
  77. Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938 [cs.CY] · arXiv
  78. Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages (SIGCSE 2023). ACM, NY, NY, USA, 563-569. https://doi.org/10.1145/3545945.3569770 · DOI · Paper page
  79. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092-1097.
  80. Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Program- ming Classes. arXiv:2308.06921 [cs.CY] · arXiv
  81. Raymond Lister, Elizabeth S. Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, Beth Simon, and Lynda Thomas. 2004. A Multi-National Study of Reading and Tracing Skills in Novice Programmers. SIGCSE Bull. 36, 4 (jun 2004), 119-150. https://doi.org/10.1145/1041624.1041673 · DOI · Paper page
  82. Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Gian- nakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). Association for Computing Machinery, New York, NY, USA, 55-106. https://doi.org/10.1145/3293881.3295779 · DOI
  83. Qianou Ma, Tongshuang Wu, and Kenneth Koedinger. 2023. Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming. arXiv preprint arXiv:2306.05153 (2023). · arXiv
  84. Stephen MacNeil, Joanne Kim, Juho Leinonen, Paul Denny, Seth Bernstein, Brett A. Becker, Michel Wermelinger, Arto Hellas, Andrew Tran, Sami Sarsa, James Prather, and Viraj Kumar. 2023. The Implications of Large Language Models for CS Teachers and Students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1255. https: //doi.org/10.1145/3545947.3573358 · DOI
  85. Stephen MacNeil, Celine Latulipe, Bruce Long, and Aman Yadav. 2016. Exploring lightweight teams in a distributed learning environment. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 193-198.
  86. Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 931-937. https://doi.org/10.1145/3545945.3569785 · DOI · Paper page
  87. Stephen MacNeil, Andrew Tran, Juho Leinonen, Paul Denny, Joanne Kim, Arto Hellas, Seth Bernstein, and Sami Sarsa. 2023. Automatically Generating CS Learning Materials with Large Language Models. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1176. https://doi.org/10.1145/3545947.3569630 · DOI
  88. Jordan K. Matelsky, Felipe Parodi, Tony Liu, Richard D. Lange, and Konrad P. Kording. 2023. A large language model-assisted education tool to provide feedback on open-ended responses. arXiv:2308.02439 [cs.CY] · arXiv
  89. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2023. In-IDE Generation-based Information Support with a Large Language Model. arXiv:2307.08177 [cs.SE] · arXiv
  90. Maciej Pankiewicz and Ryan S. Baker. 2023. Large Language Models (GPT) for automating feedback on programming assignments. arXiv:2307.00150 [cs.HC] · arXiv
  91. Randy Pausch, Tommy Burnette, AC Capeheart, Matthew Conway, Dennis Cosgrove, Rob DeLine, Jim Durbin, Rich Gossweiler, Shuichi Koga, and Jeff White. 1995. Alice: Rapid prototyping system for virtual reality. IEEE Computer Graphics and Applications 15, 3 (1995), 8-11.
  92. Fynn Petersen-Frey, Marcus Soll, Louis Kobras, Melf Johannsen, Peter Kling, and Chris Biemann. 2022. Dataset of Student Solutions to Algorithm and Data Structure Programming Assignments. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 956-962.
  93. Carrie Anne Philbin. 2023. Exploring the Potential of Artificial Intelligence Program Generators in Computer Programming Education for Students. ACM Inroads 14, 3 (aug 2023), 30-38. https://doi.org/10.1145/3610406 · DOI
  94. Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majum- dar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. arXiv:2302.04662 [cs.PL] · arXiv
  95. Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv preprint arXiv:2303.13528 (2023). · arXiv
  96. Russell A Poldrack, Thomas Lu, and Gašper Beguš. 2023. AI-assisted coding: Experiments with GPT-4. arXiv:2304.13187 [cs.AI] · arXiv
  97. Lori Pollock. 2019. A Collaborative Practicum Targeting Communication Skills for Computer Science Researchers. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 845-851. https: //doi.org/10.1145/3287324.3287454 · DOI
  98. James Prather, Brett A. Becker, Michelle Craig, Paul Denny, Dastyni Loksa, and Lauren Margulieux. 2020. What Do We Think We Think We Are Doing? Metacognition and Self-Regulation in Programming (ICER '20). ACM, NY, NY, USA, 2-13. https://doi.org/10.1145/3372782.3406263 · DOI
  99. James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice Pro- grammers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER '18). Association for Computing Machinery, New York, NY, USA, 41-50. https://doi.org/10.1145/3230977.3230981 · DOI · Paper page
  100. Victor-Alexandru Pădurean, Georgios Tzannetos, and Adish Singla. 2023. Neural Task Synthesis for Visual Programming. arXiv:2305.18342 [cs.LG] · arXiv
  101. Rajendra Raj, Mihaela Sabin, John Impagliazzo, David Bowers, Mats Daniels, Felienne Hermans, Natalie Kiesler, Amruth N. Kumar, Bonnie MacKellar, Renée McCauley, Syed Waqar Nabi, and Michael Oudshoorn. 2021. Professional Competencies in Computing Education: Pedagogies and Assessment. In Pro- ceedings of the 2021 Working Group Reports on Innovation and Technology in Computer Science Education (Virtual Event, Germany) (ITiCSE-WGR '21). As- sociation for Computing Machinery, New York, NY, USA, 133-161. https: //doi.org/10.1145/3502870.3506570 · DOI
  102. Parsa Rajabi, Parnian Taghipour, Diana Cukierman, and Tenzin Doleck. 2023. Exploring ChatGPT's impact on post-secondary education: A qualitative study. In Western Canadian Conference on Computing Education (WCCCE'23), May 04-05, 2023. Simon Fraser University.
  103. Arun Raman and Viraj Kumar. 2022. Programming Pedagogy and Assessment in the Era of AI/ML: A Position Paper. In Proceedings of the 15th Annual ACM India Compute Conference (Jaipur, India) (COMPUTE '22). Association for Computing Machinery, New York, NY, USA, 29-34. https://doi.org/10.1145/3561833.3561843 · DOI
  104. Raghu Raman, Santanu Mandal, Payel Das, Tavleen Kaur, Sanjanasri JP, and Prema Nedungadi. 2023. University students as early adopters of ChatGPT: Innovation Diffusion Study. arXiv:10.21203/rs.3.rs-2734142/v1 [cs.HC] https: //doi.org/10.21203/rs.3.rs-2734142/v1 · DOI
  105. Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Eve- lyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and Yasmin Kafai. 2009. Scratch: Programming for all. Commun. ACM 52 (11 2009), 60-67. Issue 11. https://doi.org/10.1145/1592761.1592779 · DOI · Paper page
  106. Steven I Ross, Michael Muller, Fernando Martinez, Stephanie Houde, and Justin D Weisz. 2023. A Case Study in Engineering a Conversational Programming Assistant's Persona. In Joint Proceedings of the ACM IUI Workshops 2023, March 2023, Sydney, Australia.
  107. Gery W. Ryan and H. Russell Bernard. 2003. Techniques to Identify Themes. Field Methods 15, 1 (2003), 85-109. https://doi.org/10.1177/1525822X02239569 arXiv:https://doi.org/10.1177/1525822X02239569 · DOI · Paper page
  108. Dana Saito-Stehberger. 2022. Examples of Culturally Responsive Teaching in Computational Thinking Curriculum. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 1055. https: //doi.org/10.1145/3478432.3499247 · DOI
  109. Pamela Samuelson. 2020. AI Authorship? Commun. ACM 63, 7 (jun 2020), 20-22. https://doi.org/10.1145/3401718 · DOI · Paper page
  110. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Gener- ation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research -Volume 1 (Lugano and Virtual Event, Switzerland) (ICER '22). ACM, NY NY, USA, 27-43. https://doi.org/10.1145/3501385.3543957 · DOI · Paper page
  111. Jaromir Savelka, Arav Agarwal, Christopher Bogart, Yifan Song, and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses? arXiv preprint arXiv:2303.09325 (2023). · arXiv
  112. Carsten Schulte and Jens Bennedsen. 2006. What Do Teachers Teach in Intro- ductory Programming?. In Proceedings of the Second International Workshop on Computing Education Research (Canterbury, United Kingdom) (ICER '06). Association for Computing Machinery, New York, NY, USA, 17-28. https: //doi.org/10.1145/1151588.1151593 · DOI
  113. Mazyar Seraj, Eva-Sophie Katterfeldt, Kerstin Bub, Serge Autexier, and Rolf Drechsler. 2019. Scratch and Google Blockly: How Girls' Programming Skills and Attitudes Are Influenced. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research. ACM, New York. https://doi.org/ 10.1145/3364510.3364515 · DOI
  114. Judy Sheard, Angela Carbone, Raymond Lister, Beth Simon, Errol Thompson, and Jacqueline L. Whalley. 2008. Going SOLO to Assess Novice Programmers. SIGCSE Bull. 40, 3 (jun 2008), 209-213. https://doi.org/10.1145/1597849.1384328 · DOI
  115. Judy Sheard, Simon, Matthew Butler, Katrina Falkner, Michael Morgan, and Amali Weerasinghe. 2017. Strategies for Maintaining Academic Integrity in First-Year Computing Courses (ITiCSE '17). ACM, NY, NY, USA, 244-249. https: //doi.org/10.1145/3059009.3059064 · DOI · Paper page
  116. Lee S. Shulman. 2005. Signature Pedagogies in the Professions. Daedalus 134, 3 (2005), 52-59. http://www.jstor.org/stable/20027998
  117. Simon, Judy Sheard, Michael Morgan, Andrew Petersen, Amber Settle, and Jane Sinclair. 2018. Informing Students about Academic Integrity in Programming. In Proceedings of the 20th Australasian Computing Education Conference (Brisbane, Queensland, Australia) (ACE '18). Association for Computing Machinery, New York, NY, USA, 113-122. https://doi.org/10.1145/3160489.3160502 · DOI
  118. Adish Singla. 2023. Evaluating ChatGPT and GPT-4 for Visual Programming. arXiv:2308.02522 [cs.LG] · arXiv
  119. Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Christopher Bogart, Jaromir Savelka, and Majd Sakr. 2023. Harnessing llms in curricular design: Using gpt-4 to support authoring of learning objectives. arXiv preprint arXiv:2306.17459 (2023). · arXiv
  120. Tianjia Wang, Daniel Vargas Dıaz, Chris Brown, and Yan Chen. 2023. Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
  121. Tianjia Wang, Daniel Vargas-Diaz, Chris Brown, and Yan Chen. 2023. To- wards Adapting Computer Science Courses to AI Assistants' Capabilities. arXiv preprint arXiv:2306.03289 (2023). · arXiv
  122. Patricia Widjojo and Christoph Treude. 2023. Addressing Compiler Errors: Stack Overflow or Large Language Models? arXiv:2307.10793 [cs.SE] · arXiv
  123. Alistair Willis, Patricia Charlton, and Tony Hirst. 2020. Developing Students' Written Communication Skills with Jupyter Notebooks. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '20). Association for Computing Machinery, New York, NY, USA, 1089-1095. https://doi.org/10.1145/3328778.3366927 · DOI
  124. Claes Wohlin, Marcos Kalinowski, Katia Romero Felizardo, and Emilia Mendes. 2022. Successful combination of database search and snowballing for identi- fication of primary studies in systematic literature studies. Information and Software Technology 147 (2022), 106908.
  125. Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large language models meet NL2Code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers). 7443-7464.
  126. Cynthia Zastudil, Magdalena Rogalska, Christine Kapp, Jennifer Vaughn, and Stephen MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. IEEE Frontiers in Education (FIE) (2023). https: //doi.org/10.48550/arXiv.2308.04309 · DOI
  127. Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2022. Repairing Bugs in Python Assignments Using Large Language Models. arXiv preprint arXiv:2209.14876 (2022). · arXiv

Authors
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., MacNeil, S., Peterson, A., Pettit, R., Reeves, B. N., & Savelka, J.
Affiliations
  • University College Dublin (IE)
  • Abilene Christian University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.00658
arXiv
2310.00658
OpenAlex
W4387323762
Cited by
9
References
148
Citations by year
2026: 2 · 2024: 6 · 2023: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Repository KITopen (Karlsruhe Institute of Technology) (submittedVersion, cc-by)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
large language models, generative AI in education, computing education, pedagogy, curriculum adaptation, assessment, ACM Code of Ethics, AI ethics, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review