The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Intelligent tutoring systems for programming education: a systematic review
Abstract
A variety of intelligent tutoring systems have been created for the purpose of teaching computer programming. Most published literature focuses on systems that have been developed to teach programming within tertiary courses. A majority of systems have been developed to teach introductory programming concepts; other systems tutor more specific aspects of programming like scope or recursion. Literature reports that these systems address many of the difficulties associated with teaching programming to novices; however, individual systems vary greatly, and there is a large range of supplementary features developed in these systems. Most intelligent programming tutors involve some form of interactive programming exercises, but the use of supplementary features like plans, quizzes and worked solutions vary greatly between different systems. This systematic review reports key information about existing systems and the prevalence of different features within them. An overview of how supplementary features are integrated into these systems is given, along with implications for how intelligent programming tutors could be improved by supporting a wider range of supplementary features.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Intelligent feedback and hints can be embedded into programming tasks, helping with specific semantic or syntactic issues (Crow et al., 2018)”
From A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour (2024) · cites this paper as Crow et al., 2018
Citations
Cites
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“Backward snowballing as outlined in ’Guidelines for snowballing in systematic literature studies and a replication in software engineer-ing’ [28] was carried out for papers cited within the literature that met the inclusion criteria; this was done to ensure that research and IPTs that were…”
MethodResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 1×
“3160492 has been well established with experiments involving human tutors [3].”
BackgroundThe Scratch Programming Language and Environment
Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,410
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930 · cited 1×
“Experiments within the field of intelligent tutoring have shown similar effect size benefits to human tutoring [11].”
BackgroundELM-ART: An Adaptive Versatile System for Web-Based Instruction
Weber, G., & Brusilovsky, P. L. (2001) · Cited by 542
PROUST: Knowledge-Based Program Understanding
Johnson, W. L., & Soloway, E. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232210 · Cited by 284
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258
The automated tutoring of introductory computer programming
Anderson, J. R., & Skwarecki, E. (1986) · Communications of the ACM · doi:10.1145/6592.6593 · Cited by 225
Hooshyar, D., Ahmad, R. B., Yousefi, M., Yusop, F. D., & Horng, S. (2015) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12099 · Cited by 105
Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback
Gerdes, A., Heeren, B. J., Jeuring, J. T., & van Binsbergen, L. T. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0080-x · Cited by 102
Intelligent Tutor, Environment and Manual for Introductory Programming
Brusilovsky, P. L. (1992) · Programmed Learning and Educational Technology · doi:10.1080/0954730920290104 · Cited by 72
Developing an Intelligent Tutoring System For Students Learning To Program in C++
Abu-Naser, S. S. (2008) · Information Technology Journal · doi:10.3923/itj.2008.1055.1060 · Cited by 71
Developing intelligent programming tutors for novice programmers
Pillay, N. (2003) · ACM SIGCSE Bulletin · doi:10.1145/782941.782986 · Cited by 60
Generation of problems, answers, grade, and feedback---case study of a fully automated tutor
Kumar, A. N. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163408 · Cited by 60
J-Latte: a Constraint-Based Tutor for Java
Holland, J., Mitrović, A., & Martin, B. (2009) · UC Research Repository (University of Canterbury) · Cited by 59
A Review of AI-Supported Tutoring Approaches for Learning Programming
Le, N.-T., Strickroth, S., Groß, S., & Niels Pinkwart, C. (2013) · Studies in computational intelligence · doi:10.1007/978-3-319-00293-4_20 · Cited by 52
How Effective are Intelligent Tutoring Systems in Computer Science Education?
Nesbit, J. C., Adesope, O., Liu, Q., & Ma, W. (2014) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2014.38 · Cited by 52
A Classification of Adaptive Feedback in Educational Systems for Programming
Le, N.-T. (2016) · Systems · doi:10.3390/systems4020022 · Cited by 48
An intelligent tutoring system for introductory C language course
Song, J., Hahn, S. H., Tak, K., & Kim, J. H. (1997) · Computers & Education · doi:10.1016/s0360-1315(97)00003-1 · Cited by 43
Coached program planning: dialogue-based support for novice program design
Lane, H. C., & VanLehn, K. A. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611955 · Cited by 39
Initial results of using an intelligent tutoring system with Alice
Cooper, S., Nam, Y. J., & Si, L. (2012) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2325296.2325332 · Cited by 32
Incorporating an intelligent tutoring system into CS1
Soh, L. (2006) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1121341.1121494 · Cited by 18
Nesbit, J. C., Liu, L., Liu, Q., & Adesope, O. (2015) · ASEE Annual Conference & Exposition · doi:10.18260/p.25090 · Cited by 18
Intelligent tutoring system for CS-I and II laboratory
Yoo, J. P., Pettey, C. C., Yoo, S., Hankins, J., Li, C., & Seo, S. J. (2006) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/1185448.1185482 · Cited by 14
A design-first curriculum and Eclipse™ tools
Moritz, S. H., Blank, G. D., Parvez, S. M., & Fang, W. (2007) · Journal of computing sciences in colleges · doi:10.5555/1181849.1181856 · Cited by 4
Integrating Intelligent Feedback into Block Programming Environments
Price, T. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787748 · Cited by 2
Research works printed but not linked: 1
- Omar AlZoubi , Davide Fossati , Barbara Di Eugenio , and Nick Green , [n. d.]. ChiQat-Tutor : An integrated environment for learning recursion . In Proc. of the Second Workshop on AI-supported Education for Computer Science (AIEDCS)(at ITS 2014 ). Honolulu, HI ( 2014 ). Omar AlZoubi, Davide Fossati, Barbara Di Eugenio, and Nick Green, [n. d.]. ChiQat-Tutor: An integrated environment for learning recursion. In Proc. of the Second Workshop on AI-supported Education for Computer Science (AIEDCS)(at ITS 2014). Honolulu, HI (2014).
Cited by
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00436-z · Cited by 852
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
“, using web-based examples), the customised solutions offered by Codex may lead students to perceive the solutions as being more credible, similar to an intelligent tutoring system [6].”
MethodSelf-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationWang, H., Tlili, A., Huang, R., Cai, Z., Li, M., Cheng, Z., Yang, D., Li, M., Zhu, X., & Fei, C. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11555-x · Cited by 239
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
Metadata
- Authors
- Crow, T., Luxton-Reilly, A., & Wuensche, B.
- Affiliations
- University of Auckland (NZ)
- Venue
- Proceedings of the 20th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2018-01-30
- Pages
- 53-62
- OpenAlex
- W2783082347
- Cited by
- 166
- References
- 27
- Citations by year
- 2026: 15 · 2025: 25 · 2024: 27 · 2023: 20 · 2022: 24 · 2021: 23 · 2020: 10 · 2019: 17 · 2018: 5
- FWCI
- 15.57
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Australasian Computing Education Conference (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- intelligent tutoring systems, programming education, novice programmers, worked solutions, quizzes, plans, scope, recursion, introductory programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.