Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Author(s): Prather, James; Leinonen, Juho; Kiesler, Natalie; Benario, Jamie Gorson; Lau, Sam; MacNeil, Stephen; Norouzi, Narges; Opel, Simone; Pettit, Vee; Porter, Leo; Reeves, Brent N; Savelka, Jaromir; Smith, David H; Strickroth, Sven; Zingaro, Daniel
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Terry, G., Hayfield, N., Clarke, V., & Braun, V. (2017) · The SAGE Handbook of Qualitative Research in Psychology · doi:10.4135/9781526405555.n2 · Cited by 2,448
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493 · cited 1×
“Public perception of GenAI is mixed, partially because it is a “black box” and that lack of transparency often increases fear [21], which is one reason why GenAI should be designed to increase transparency to end users [142].”
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 1×
“Beginning in early 2022, computing education researchers showed that these models had incredible accuracy solving programming problems and exam questions in multiple courses and contexts [6, 40, 41, 48, 68, 119, 125, 126].”
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1×
“Specifically, one study found that software developers are using GenAI tools at many points in the programming process, including creating, modifying, debugging, and explaining code [15].”
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · cited 8×
“Another survey of educators was conducted by Prather et al. [111].”
InfluentialSelf-citationStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
Predicting academic performance: a systematic literature review
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. N. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295783 · Cited by 355
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316 · cited 2×
“Computing education is undergoing a seismic shift due to the advances in generative AI (GenAI) [22, 36, 111].”
Self-citationGenerative AI for Customizable Learning Experiences
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024) · Sustainability · doi:10.3390/su16073034 · Cited by 274
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · cited 6×
“Although some have advocated for banning GenAI entirely, that also seems impractical given its free availability to students outside the classroom [80].”
InfluentialSelf-citationUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Self-citationA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · cited 1×
“Similarly, in an observational study, researchers found that although a larger percentage of students are able to complete programming tasks with the aid of GenAI tools, students faced new metacognitive challenges [113].”
Self-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · cited 1×
“Similarly, in an observational study, researchers found that although a larger percentage of students are able to complete programming tasks with the aid of GenAI tools, students faced new metacognitive challenges [113].”
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · cited 2×
“In-deed, GenAI can provide other kinds of advanced and customized help and qualitative feedback to students working on computing assignments [10, 11, 67, 81, 89, 94].”
Self-citationCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
Prompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187 · cited 1×
“…[86, 133], WorkedGen [57], LLM Hint Factory [158], Tutor Kai [52], CodeTutor [90], GPTeach [96], Charlie the Coding Cow [103], KOGI [78], NetLogo Chat [24], Promptly [33, 34], IPSSC [85], CS50 Duck [87], Gilt [100], CodeTailor [50], and a variety of others that lacked names [76, 97, 101, 156].”
Self-citationTeaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185
Brauner, P., Hick, A., Philipsen, R., & Ziefle, M. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1113903 · Cited by 143 · cited 1×
“Public perception of GenAI is mixed, partially because it is a “black box” and that lack of transparency often increases fear [21], which is one reason why GenAI should be designed to increase transparency to end users [142].”
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143
Trust in Generative AI among students: An Exploratory Study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Ragavan, S. S., & Alipour, M. A. (2023) · arXiv · doi:10.48550/arxiv.2310.04631 · Cited by 143
Trust in Generative AI among Students: An exploratory study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Srinivasa Ragavan, S., & Alipour, M. A. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630842 · Cited by 143
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · cited 1×
“Beginning in early 2022, computing education researchers showed that these models had incredible accuracy solving programming problems and exam questions in multiple courses and contexts [6, 40, 41, 48, 68, 119, 125, 126].”
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Self-citationLyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662036 · Cited by 131
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124 · cited 1×
“Typically, productivity is measured in terms of the acceptance rates of coding suggestions [163].”
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2022) · arXiv · doi:10.48550/arxiv.2210.14306 · Cited by 114 · cited 1×
“New assessment methods [61] and ways to measure user interaction with GenAI [98] are required, and although some are arguing for making assignments “LLM-proof” [20], which seems to be an impractical goal given the rapid improvement in these models.”
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · cited 1×
“Beginning in early 2022, computing education researchers showed that these models had incredible accuracy solving programming problems and exam questions in multiple courses and contexts [6, 40, 41, 48, 68, 119, 125, 126].”
Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education
Jin, H., Lee, S., Shin, H., & Kim, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642349 · Cited by 99
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97 · cited 2×
“Regardless, an increasing number of instructors are adopting (or at least not banning) GenAI in their courses compared to prior studies [49, 111].”
Self-citationDoes ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96
CS1-LLM: Integrating LLMs into CS1 Instruction
Vadaparty, A., Zingaro, D., Smith IV, D. H., Padala, M., Alvarado, C., Gorson Benario, J., & Porter, L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653584 · Cited by 77 · cited 1×
“Some are pointing to its introduction as a way to equip students to move faster through the curriculum than was ever possible before [148].”
Self-citationAmani, S., White, L., Balart, T., Arora, L., Shryock, K. J., Brumbelow, K., & Watson, K. L. (2023) · arXiv · doi:10.48550/arxiv.2304.14415 · Cited by 76
Better Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69
(2024) · doi:10.1145/3664646.3664773 · Cited by 67
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
Self-citationCan Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
Exploring ChatGPT’s impact on post-secondary education: A qualitative study
Rajabi, P., Taghipour, P., Cukierman, D., & Doleck, T. (2023) · Proceedings of the 25th Western Canadian Conference on Computing Education · doi:10.1145/3593342.3593360 · Cited by 57
Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56 · cited 1×
“Beginning in early 2022, computing education researchers showed that these models had incredible accuracy solving programming problems and exam questions in multiple courses and contexts [6, 40, 41, 48, 68, 119, 125, 126].”
Self-citationStudents’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming
Zhang, Z., Dong, Z., Shi, Y., Price, T., Matsuda, N., & Xu, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30372 · Cited by 54
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
Self-citationChildren on ChatGPT Readability in an Educational Context: Myth or Opportunity?
Murgia, E., Pera, M. S., Landoni, M., & Huibers, T. (2023) · Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3563359.3596996 · Cited by 54 · cited 1×
“K-12 teachers outside of computing education are having similar conversations to those occurring in higher education [14, 18, 99, 109, 116, 121].”
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Self-citationScholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 1×
“Specifically, one study found that software developers are using GenAI tools at many points in the programming process, including creating, modifying, debugging, and explaining code [15].”
The Psychological Effects of AI-Assisted Programming on Students and Professionals
Valový, M., & Buchalcevova, A. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00050 · Cited by 23
Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing
Denny, P., Sarsa, S., Hellas, A., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.04715 · Cited by 21 · cited 1×
“Other early work focused on the ways in which GenAI can provide support to computing educators [16, 37, 81, 82, 94, 124, 127].”
Self-citationSingh, A., Brooks, C., Wang, X., Li, W., Kim, J., & Wilson, D. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636853 · Cited by 18
AI-Generated Code Not Considered Harmful
Kendon, T., Wu, L., & Aycock, J. (2023) · Proceedings of the 25th Western Canadian Conference on Computing Education · doi:10.1145/3593342.3593349 · Cited by 13 · cited 1×
“New assessment methods [61] and ways to measure user interaction with GenAI [98] are required, and although some are arguing for making assignments “LLM-proof” [20], which seems to be an impractical goal given the rapid improvement in these models.”
Bopp, C., Foerst, A., & Kellogg, B. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630941 · Cited by 11
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv
Cited by
GenAI in Indian higher education: faculty at the crossroads of adoption
Razi-ur-Rahim, M., Uddin, F., Chauhan, J., & Faisal, K. P. (2026) · Asian Education and Development Studies · doi:10.1108/aeds-04-2025-0194 · Cited by 9
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Self-citationZhu, S., & Liu, Z. (2026) · Proceedings of the 2026 International Conference on Big Data and Informatization Education · doi:10.1145/3806980.3806985
Mapeamento Sistemático sobre o Uso de IAs Generativas no Ensino de Programação
Ferreira de Sousa, J., & Silveira, F. R. D. V. (2026) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2026.7169
Paterson, J., Oyelere, S. S., Dubay, C., & Izu, C. (2025) · Working Group Reports of the ACM Global Computing Education Conference 2025 · doi:10.1145/3815397.3815774
Metadata
- Authors
- Prather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., Reeves, B. N., Savelka, J., Smith, D. H., Strickroth, S., & Zingaro, D.
- Venue
- 2024 Working Group Reports on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2025-01-22
- Pages
- 300-338
- OpenAlex
- W4406803762
- Cited by
- 96
- References
- 61
- FWCI
- 112.13
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.