A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning.
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
© 2013 American Psychological Association.This meta-analysis synthesizes research on the effectiveness of intelligent tutoring systems (ITS) for college students. Thirty-five reports were found containing 39 studies assessing the effectiveness of 22 types of ITS in higher education settings. Most frequently studied were AutoTutor, Assessment and Learning in Knowledge Spaces, eXtended Tutor-Expert System, and Web Interface for Statistics Education. Major findings include (a) Overall, ITS had a moderate positive effect on college students' academic learning (g = .32 to g = .37); (b) ITS were less effective than human tutoring, but they outperformed all other instruction methods and learning activities, including traditional classroom instruction, reading printed text or computerized materials, computer-assisted instruction, laboratory or homework assignments, and no-treatment control; (c) ITS's effectiveness did not significantly differ by different ITS, subject domain, or the manner or degree of their involvement in instruction and learning; and (d) effectiveness in earlier studies appeared to be significantly greater than that in more recent studies. In addition, there is some evidence suggesting the importance of teachers and pedagogy in ITS-assisted learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another meta-analysis calculated a point estimate for intelligent tutoring systems in college-level business education of ḡ = 0.16 (Steenbergen-Hu & Cooper, 2014).”
From Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education (2017) · cites this paper as Steenbergen-Hu & Cooper, 2014
In the citing paper
“and that ITS were less effective that human tutoring”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“Steenbergen-Hu and Cooper (2014) found that ITS had a moderate effect of students' learning”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“but ITS outperformed all other instruction methods (such as traditional classroom instruction, reading printed or digital text, or homework assignments).”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“most of the strategies and functions that our participants desired to improve the usability issues of ChatGPT, such as hints, feedback, prompt suggestions, a learning map and navigation system, and AI-Helper agents, resemble the traits that ITSs have long had and are already proven to be effective.”
From The Promise and Peril of ChatGPT in Higher Education: Opportunities, Challenges, and Design Implications (2024) · cites this paper as [80]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · cited 1×
“We further conducted Duval and Tweedie’s (2000) trim and fill procedures and found that the adjusted average effect sizes remained the same as the observed ones, which were g .”
The Role of Goal Orientation in Self-Regulated Learning
Pintrich, P. R. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50043-3 · Cited by 5,173 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853 · cited 1×
“Graesser et al.’s (2011) review of ITS posed a number of questions that need answering.”
A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students
Pintrich, P. R. (2004) · Educational Psychology Review · doi:10.1007/s10648-004-0006-x · Cited by 3,269 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 7×
“Bloom (1984) reported that expert human tutors can help students achieve learning gains as large as two sigmas. Although not as effective as what Bloom (1984) found, a recent meta-review by VanLehn (2011) found that human tutoring had a positive impact of d .79 on students’ learning. Many ITS are designed to follow th…”
InfluentialVanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323 · cited 4×
“VanLehn (2011) reviewed randomized experiments that compared the effectiveness of human tutoring, computer tutoring, and no tutoring.”
InfluentialCognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · cited 1×
“For example, Cognitive Tutors are built on a cognitive theory called adaptive control of thought (ACT, or ACT-R in its updated form; Anderson et al., 1995).”
Sample criteria for testing outlying observations
Grubbs, F. E. (1950) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729885 · Cited by 1,631 · cited 4×
“Grubbs (1950) tests detected no effect-size outliers.”
InfluentialBuilding Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 1,376
Intelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
Cited by
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
“• RQ3: What are the potential issues associated with ChatGPT, and how can they be addressed?”
BackgroundVision, challenges, roles and research issues of Artificial Intelligence in Education
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2020.100001 · Cited by 1,435
Variables associated with achievement in higher education: A systematic review of meta-analyses.
Schneider, M., & Preckel, F. (2017) · Psychological Bulletin · doi:10.1037/bul0000098 · Cited by 1,083
Artificial Intelligence trends in education: a narrative overview
Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.08.233 · Cited by 956
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930
“The meta-analytic review by Steenbergen-Hu and Cooper (2014) examined 35 evaluations of ITS effectiveness in colleges.”
MethodExplainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Intelligent tutoring systems and learning outcomes: A meta-analysis
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014) · Journal of Educational Psychology · doi:10.1037/a0037123 · Cited by 840
“Steenbergen-Hu and Cooper (2014) conducted a meta-analysis of 39 studies evaluating the use of ITS for college students' academic learning The studies appeared between 1990 and 2011and reported on 22 different types of ITS While most types of ITS in the analysis were evaluated by only a single…”
BackgroundMethodInfluentialArtificial intelligence innovation in education: A twenty-year data-driven historical analysis
Guan, C., Mou, J., & Jiang, Z. (2020) · International Journal of Innovation Studies · doi:10.1016/j.ijis.2020.09.001 · Cited by 665
Huang, A. Y. Q., Lu, O. H. T., & Yang, S. J. H. (2023) · Computers & Education · doi:10.1016/j.compedu.2022.104684 · Cited by 605
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103897 · Cited by 574
Transformative effects of ChatGPT on modern education: Emerging Era of AI Chatbots
Gill, S. S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Parlikad, A. K., et al. (2024) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.06.002 · Cited by 488
Imran, M., & Almusharraf, N. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13605 · Cited by 421
“In a meta-analysis-based study of the effectiveness of ITS on college learners ’ academic performance, Steenbergen-Hu and Cooper (2014) concluded that ITS has a moderately positive impact on college students ’ academic achievement.”
BackgroundA review of AI teaching and learning from 2000 to 2020
Ng, D. T. K., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. W. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11491-w · Cited by 411
Empowering Educators to Be AI-Ready
Luckin, R., Cukurova, M., Kent, C., & du Boulay, B. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100076 · Cited by 422
Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education
Belland, B. R., Walker, A. E., Kim, N. J., & Lefler, M. (2017) · Review of Educational Research · doi:10.3102/0034654316670999 · Cited by 346
“Another meta-analysis calculated a point estimate for intelligent tutoring systems in college-level business education of ḡ = 0.16 (Steenbergen-Hu & Cooper, 2014).”
BackgroundMethodInfluentialDigital support for academic writing: A review of technologies and pedagogies
Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., & Rapp, C. (2019) · Computers & Education · doi:10.1016/j.compedu.2018.12.005 · Cited by 298
Developing Personalized Education: A Dynamic Framework
Tetzlaff, L., Schmiedek, F., & Brod, G. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09570-w · Cited by 266
Kim, N. J., Belland, B. R., & Walker, A. E. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9419-1 · Cited by 220
Artificial Intelligence as an Effective Classroom Assistant
du Boulay, B. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.93 · Cited by 213
Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025252 · Cited by 193
“47 when the comparator was textbooks or workbooks.(15) Ma et al(18) reported a mean ES of 0.”
BackgroundA Meta-Analysis of Scaffolding Effects in Online Learning in Higher Education
Doo, M. Y., Bonk, C., & Heo, H. (2020) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v21i3.4638 · Cited by 188
“(Steenbergen-Hu & Cooper, 2014, p. 342) The large effect size in our research may be related to the research designs of individual scaffolding studies.”
BackgroundInfluentialZheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.2015693 · Cited by 183
“Furthermore, only Steenbergen-Hu and Cooper (2014) conducted a meta-analysis of the e ff ect of intelligent tutoring systems (ITS), and they found that ITS had a moderate e ff ect on college students ’ learning achievement.”
Intelligent Tutoring Systems: A Comprehensive Historical Survey with Recent Developments
Alkhatlan, A., & Kalita, J. (2019) · International Journal of Computer Applications · doi:10.5120/ijca2019918451 · Cited by 155
Kannan, J., & Munday, P. (2018) · Círculo de Lingüística Aplicada a la Comunicación · doi:10.5209/clac.62495 · Cited by 157
Looking Beyond the Hype: Understanding the Effects of AI on Learning
Bauer, E., Greiff, S., Graesser, A. C., Scheiter, K., & Sailer, M. (2025) · Educational Psychology Review · doi:10.1007/s10648-025-10020-8 · Cited by 153
“…found medium to large learning benefits for ITS, which provide feedback and targeted hints, compared to self-reliant learning without support, and small to medium gains when comparing ITS to non-adaptive or less adaptive computer-assisted learning (Ma et al., 2014; Steenbergen-Hu & Cooper, 2014).”
Reconsidering the voice effect when learning from a virtual human
Craig, S. D., & Schroeder, N. L. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.003 · Cited by 179
“Advances in artificial intelligence have made intelligent tutoring systems effective for learning (Ma, Adesope, Nesbit, & Liu, 2014; Steenbergen-Hu & Cooper, 2014), video game engines now make it possible to create high fidelity educational games (Epic Games Inc., 2015; Unity Technologies, 2015),…”
BackgroundXu, Z., Wijekumar, K. (., Ramirez, G., Hu, X., & Irey, R. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12758 · Cited by 143
Christopoulos, A., & Sprangers, P. (2021) · Cogent Education · doi:10.1080/2331186x.2021.1964690 · Cited by 239
“Relevant systematic reviews and meta-analyses (Li & Ma, 2010; Ma et al., 2014; Steenbergen-Hu & Cooper, 2014) have found positive overall effects on the potential of ITS to provide adaptive, personalized learning pathways thus, recommending the conduct of continues research to improve and advance…”
BackgroundThe Changing Nature of Educational Assessment
Bennett, R. E. (2015) · Review of Research in Education · doi:10.3102/0091732x14554179 · Cited by 191
“Evidence suggests that such tutors can facilitate learning, though this conclusion appears to be more supportable at the university than at the K–12 level (Pane, Griffin, McCaffrey, & Karam, 2013; Steenbergen-Hu & Cooper, 2013, 2014).”
BackgroundHanham, J., Lee, C. B., & Teo, T. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104252 · Cited by 141
Reinhold, F., Hoch, S., Werner, B., Richter-Gebert, J., & Reiss, K. M. (2019) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.101264 · Cited by 201
Guo, L., Wang, D., Gu, F., Li, Y., Wang, Y., & Zhou, R. (2021) · Asia Pacific Education Review · doi:10.1007/s12564-021-09697-7 · Cited by 124
“…review type has been meta-analyses based on quantitative systematic reviews, most of which have compared the learning effects of different ITS systems, the teaching methods, or the learning conditions (VanLehn, 2011; Graham et al., 2015; Steenbergen-Hu & Cooper, 2014; Ma et al., 2014; Kulik, 2015).”
BackgroundInfluentialA Systematic Literature Review of Intelligent Tutoring Systems With Dialogue in Natural Language
Paladines, J., & Ramirez, J. (2020) · IEEE Access · doi:10.1109/access.2020.3021383 · Cited by 121
Authoring Tools for Designing Intelligent Tutoring Systems: a Systematic Review of the Literature
Dermeval, D., Paiva, R., Bittencourt, I. I., Vassileva, J., & Borges, D. (2018) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0157-9 · Cited by 119
“As reported by du Boulay (2016), there have been a number of recent positive reviews in support of the effectiveness of ITSs (Kulik and Fletcher 2016; Ma et al. 2014; Steenbergen-Hu and Cooper 2013, 2014; VanLehn 2011).”
MethodResultMohamed, A. M., Shaaban, T. S., Bakry, S. H., Guillén-Gámez, F. D., & Strzelecki, A. (2025) · Innovative Higher Education · doi:10.1007/s10755-024-09747-z · Cited by 115
“AI tutor - ing systems can improve students’ academic performance (Nemorin et al., 2023; Steenbergen-Hu & Cooper, 2014 ).”
Evaluating Artificial Intelligence in Education for Next Generation
Joshi, S., Rambola, R. K., & Churi, P. (2021) · Journal of Physics: Conference Series · doi:10.1088/1742-6596/1714/1/012039 · Cited by 102
“Some focus on subjects with mathematical rules, some teach reading or writing, and still, others try to teach general topic skills, such as self-paced reading strategies [24] [25] [29].”
BackgroundDesigning Adaptive Instruction for Teams: a Meta-Analysis
Sottilare, R. A., Burke, C. S., Salas, E., Sinatra, A. M., Johnston, J. H., & Gilbert, S. B. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0146-z · Cited by 119
“A recent review of artificial intelligence in education (AIED) meta-analyses by du Boulay (2016) noted investigations by VanLehn (2011), Ma et al. (2014), Kulik and Fletcher (2015), Steenbergen-Hu and Cooper (2013, 2014), and Pane et al. (2014).”
BackgroundElectronixTutor: an intelligent tutoring system with multiple learning resources for electronics
Graesser, A. C., Hu, X., Nye, B. D., VanLehn, K. A., Kumar, R. V., Heffernan, C., Heffernan, N. T., Woolf, B. P., Olney, A. M., Rus, V., et al. (2018) · International Journal of STEM Education · doi:10.1186/s40594-018-0110-y · Cited by 124
“…meta-analyses show positive effect sizes that vary from d = 0.05 (Dynarsky et al. 2007; Steenbergen-Hu and Cooper 2014) to d = 1.08 (Dodds and Fletcher 2004), but most hover between d = 0.40 and d = 0.80 (Kulik and Fletcher 2015; Ma et al. 2014; Steenbergen-Hu and Cooper 2013; VanLehn 2011).”
BackgroundInfluentialWang, X., Huang, R., Sommer, M., Pei, B., Shidfar, P., Rehman, M. S., Ritzhaupt, A. D., & Martin, F. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240459 · Cited by 198
“…positive effects on student learning outcomes in K-12 Mathematics (Steenbergen-Hu & Cooper, 2014), K-12 Reading (Xu et al., 2019), higher education (Steenbergen-Hu & Cooper, 2014) and in all context and discipline (Ma et al., 2014) but slightly lower effect sizes than the fi ndings in this study.”
InfluentialArtificial Intelligence and the Student Experience: An Institutional Perspective
Khare, K., Stewart, B. G., & Khare, A. (2018) · IAFOR Journal of Education · doi:10.22492/ije.6.3.04 · Cited by 115
“Studies by Steenbergen-Hu and Cooper (2013; 2014) and VanLehn, (2011) have indicated that AI tutors are as good or better than human tutors, however the evidence is unclear, and the experimental designs used to date do not provide an unambiguous answer.”
BackgroundPerformance Comparison of an AI-Based Adaptive Learning System in China
Cui, W., Xue, Z., & Thai, K.-P. (2018) · 2018 Chinese Automation Congress (CAC) · doi:10.1109/cac.2018.8623327 · Cited by 93
Artificial intelligence in sport management education: Playing the AI game with ChatGPT
Keiper, M. C., Fried, G. B., Lupinek, J. M., & Nordstrom, H. (2023) · Journal of Hospitality Leisure Sport & Tourism Education · doi:10.1016/j.jhlste.2023.100456 · Cited by 149
Sailer, M., Maier, R., Berger, S., Kastorff, T., & Stegmann, K. (2024) · Learning and Individual Differences · doi:10.1016/j.lindif.2024.102446 · Cited by 192
Bang, H. J., Li, L., & Flynn, K. (2023) · Early Childhood Education Journal · doi:10.1007/s10643-022-01332-3 · Cited by 91
Park, H., & Ahn, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642785 · Cited by 84
“Some have regarded ITSs as superior to earlier forms of learning technology, such as computer-assisted instruction or computer-based training [22, 80, 87].”
BackgroundInfluentialFlow Theory to Promote Learning in Educational Systems: Is it Really Relevant?
dos Santos, W. O., Bittencourt, I. I., Isotani, S., Dermeval, D., Marques, L. B., & Silveira, I. F. (2018) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2018.26.02.29 · Cited by 95
Why do students use ChatGPT? Answering through a triangulation approach
Niloy, A. C., Bari, M. A., Sultana, J., Chowdhury, R., Raisa, F. M., Islam, A., Mahmud, S., Jahan, I., Sarkar, M., Akter, S., et al. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100208 · Cited by 144
Kümmel, E., Moskaliuk, J., Creß, U., & Kimmerle, J. (2020) · Education Sciences · doi:10.3390/educsci10030078 · Cited by 128
Hwang, S. (2022) · Sustainability · doi:10.3390/su142013185 · Cited by 80
Xhakaj, F., Aleven, V. A. W. M. M., & McLaren, B. M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66610-5_23 · Cited by 83
“ITSs have been shown to enhance student learning [8, 11, 19].”
BackgroundA meta-analysis of the effectiveness of ALEKS on learning
Fang, Y., Ren, Z., Hu, X., & Graesser, A. C. (2019) · Educational Psychology · doi:10.1080/01443410.2018.1495829 · Cited by 79
“Steenbergen-Hu and Cooper (2014) found the effect size of ITS on college students ’ learning was smaller in traditional instruction than no instruction.”
BackgroundResultInfluentialEscape from the Skinner Box: The case for contemporary intelligent learning environments
du Boulay, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12860 · Cited by 79
“…of formal education, seven meta-reviews have compared the quality of the learning outcomes with ILES as compared to using more traditional methods (Kulik & Fletcher, 2016; Ma, Adesope, Nesbit, & Liu, 2014; Nesbit, Adesope, Liu, & Ma, 2014; Steenbergen-Hu & Cooper, 2013, 2014; VanLehn, 2011).”
MethodYüce, A., Abubakar, A. M., & İlkan, M. (2019) · Online Information Review · doi:10.1108/oir-11-2017-0340 · Cited by 99
Artificial Intelligence in Language Learning: Practices and Prospects
Kuddus, K. (2022) · doi:10.1002/9781119792437.ch1 · Cited by 70
Belland, B. R., Walker, A. E., & Kim, N. J. (2017) · Review of Educational Research · doi:10.3102/0034654317723009 · Cited by 103
“…with lecture, but when it is clear that an intervention is better than control, as is the case with scaffolding (Belland et al., 2017, 2015; Ma et al., 2014; Steenbergen-Hu & Cooper, 2013, 2014; VanLehn, 2011), it makes sense to determine which versions work better under which circumstances.”
BackgroundMethodInfluentialIntelligent tutoring systems work as a math gap reducer in 6th grade after-school program
Huang, X., Craig, S. D., Xie, J., Graesser, A. C., & Hu, X. (2016) · Learning and Individual Differences · doi:10.1016/j.lindif.2016.01.012 · Cited by 90
Persisting Issues in Pedagogical Agent Research
Schroeder, N. L., & Gotch, C. M. (2015) · Journal of Educational Computing Research · doi:10.1177/0735633115597625 · Cited by 71
“In some cases, these agents may also qualify as intelligent tutoring systems, which meta-analytic reviews have found to be effective for learning (e.g., Steenbergen-Hu & Cooper, 2014).”
BackgroundThe potential of learning with (and not from) artificial intelligence in education
Chichekian, T., & Benteux, B. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.903051 · Cited by 60
“These results are not surprising given that three metaanalytic reviews (Ma et al., 2014; Steenbergen-Hu and Cooper, 2014; Kulik, 2015) had also revealed the effectiveness of the ITS-related learning to be reflective of improved test score measurements.”
ResultΛαμπρόπουλος, Γ. (2023) · Springer series on cultural computing · doi:10.1007/978-3-031-27166-3_8 · Cited by 89
Nazaretsky, T., Cukurova, M., Ariely, M., & Alexandron, G. (2021) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/dzqju · Cited by 70
Yi, L., Liu, D., Jiang, T., & Xian, Y. (2024) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-024-10499-7 · Cited by 103
“This effect size ( g = 0.343) is comparable with that of Hwang (2022) ( g = 0.351), smaller than Steenbergen-Hu and Cooper (2014) ( g = 0.65), larger than Fang et al. (2018) and Steenbergen-Hu and Cooper (2013), who found that ITS didn’t bring significant influence on K-12 students’ mathematical…”
InfluentialThe End is the Beginning is the End: The closed-loop learning analytics framework
Sailer, M., Ninaus, M., Huber, S. E., Bauer, E., & Greiff, S. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108305 · Cited by 93
Tlili, A., Saqer, K., Salha, S. H., & Huang, R. (2025) · Information Development · doi:10.1177/02666669241304407 · Cited by 117
“The obtained effect of the present study is higher than other studies in the literature who also conducted a meta-analysis related to the effect of ITS on learning achievement before 2013 and found a small effect (Steenbergen-Hu & Cooper, 2014).”
Kim, N. J., Belland, B. R., Lefler, M., Andreasen, L., Walker, A. E., & Axelrod, D. B. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09502-3 · Cited by 73
“…students with special education needs (Swanson and Deshler 2003; Swanson and Lussier 2001) and scaffolding embedded within intelligent tutoring systems used by various age levels (Kulik and Fletcher 2016; Ma et al. 2014; Steenbergen-Hu and Cooper 2013; Steenbergen-Hu and Cooper 2014; VanLehn 2011).”
BackgroundInfluentialStichwort – Formatives Assessment
Schütze, B., Souvignier, E., & Hasselhorn, M. (2018) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-018-0838-7 · Cited by 93
Systematic reviews and meta-analyses of higher education research
Tight, M. (2018) · European Journal of Higher Education · doi:10.1080/21568235.2018.1541752 · Cited by 74
Personalized learning in iSTART: Past modifications and future design
McCarthy, K. S., Watanabe, M., Dai, J., & McNamara, D. S. (2020) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1716201 · Cited by 75
How Effective are Intelligent Tutoring Systems in Computer Science Education?
Nesbit, J. C., Adesope, O. O., Liu, Q., & Ma, W. (2014) · 2014 IEEE 14th International Conference on Advanced Learning Technologies · doi:10.1109/icalt.2014.38 · Cited by 51
“41 standard deviations in favor of the ITS condition [7; 8].”
BackgroundApplications of Artificial Intelligence to higher education: possibilities, evidence, and challenges
Pedró, F. (2020) · IUL Research · doi:10.57568/iulres.v1i1.43 · Cited by 102
An Interview Reflection on “Intelligent Tutoring Goes to School in the Big City”
Koedinger, K. R., & Aleven, V. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0082-8 · Cited by 51
A Qualitative Analysis of Implementing E-Learning during the COVID-19 Lockdown
Lozano, C. P., Bergés, M. S., Gil, M. L., & Bustamante, J. C. (2021) · Sustainability · doi:10.3390/su13063317 · Cited by 70
Educational Game and Intelligent Tutoring System: A Classroom Study and Comparative Design Analysis
Long, Y., & Aleven, V. A. W. M. M. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3057889 · Cited by 63
Zhang, D., Wu, J. G., & Fu, Z. (2024) · System · doi:10.1016/j.system.2024.103501 · Cited by 87
Liu, S., Guo, X., Hu, X., & Zhao, X. (2024) · Electronics · doi:10.3390/electronics13244876 · Cited by 75
Copur-Gencturk, Y., Li, J., & Atabas, S. (2024) · American Educational Research Journal · doi:10.3102/00028312241248514 · Cited by 47
“Such systems have been used in other domains and have been found effective in improving students’ academic performance (for a review, see Steenbergen-Hu & Cooper, 2014; VanLehn, 2011).”
Positive Artificial Intelligence in Education (P-AIED): A Roadmap
Bittencourt, I., Chalco, G., Santos, J., Fernandes, S. C. S., Silva, J., Batista, N., Hutz, C., & Isotani, S. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00357-y · Cited by 39
Revolutionizing Language Learning: How ChatGPT and AI are changing the Way We Learn Languages
Solak, E. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.732 · Cited by 47
“Steenbergen-Hu and Cooper (2014) suggested that AI has a minimal impact on students' learning.”
BackgroundRecent Meta-reviews and Meta–analyses of AIED Systems
du Boulay, B. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0060-1 · Cited by 45
“In a parallel study of university students, Steenbergen-Hu and Cooper (2014) found more positive effects for ITSs as compared to conventional instruction.”
Scaling Academic Writing Instruction: Evaluation of a Scaffolding Tool (Thesis Writer)
Rapp, C., & Kauf, P. (2018) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0162-z · Cited by 45
“In their meta-analysis, Steenbergen-Hu and Cooper (2014) assessed the effectiveness of various ITS on students’ academic learning.”
BackgroundBrady, M., Devitt, A., & Kiersey, R. A. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12742 · Cited by 58
Designing PairBuddy—A Conversational Agent for Pair Programming
Robe, P., & Kuttal, S. K. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3498326 · Cited by 62
“ITSs are known to increase student performance well beyond conventional classes [131, 153, 174, 240], and in some cases, outperform human tutoring [230, 231, 239].”
BackgroundConijn, R., Martinez-Maldonado, R., Knight, S., Buckingham Shum, S., Van Waes, L., & van Zaanen, M. (2022) · Computer Assisted Language Learning · doi:10.1080/09588221.2020.1839503 · Cited by 44
Hu, Y.-H., Fu, J. S., & Yeh, H. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2022.2160467 · Cited by 65
“Meanwhile, other scholars (Steenbergen-Hu & Cooper, 2014) conducted a meta-analysis of 39 related studies involving Intelligent Tutoring System (ITS), which is the same as ITR, and found that ITS has a certain impact on students ’ learning e ff ects but is not as e ff ective as human teachers.”
ChatGPT in Education: An Effect in Search of a Cause
Weidlich, J., Gašević, D., Drachsler, H., & Kirschner, P. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70105 · Cited by 52
Deeper Learning With Advances in Discourse Science and Technology
Graesser, A. C. (2015) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215600888 · Cited by 62
“Some of these intelligent systems have pedagogical agents that interact with the learner in natural language and have produced learning gains on difficult material on par with human tutors (Steenbergen-Hu & Cooper, 2014; VanLehn, 2011; VanLehn et al., 2007).”
BackgroundStat-Knowlab. Assessment and Learning of Statistics with Competence-based Knowledge Space Theory
de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-020-00223-1 · Cited by 50
“The number of published studies has grown to such a point that reviews (see, e.g., VanLehn 2011) and meta-analyses (see, e.g., Ma et al. 2014, Steenbergen-Hu and Cooper 2013, 2014) have became possible.”
BackgroundPrincesses are bigger than elephants: Effect size as a category error in evidence‐based education
Simpson, A. P. (2018) · British Educational Research Journal · doi:10.1002/berj.3474 · Cited by 47
“For example, Steenbergen-Hu and Cooper’s (1994) meta-analysis for intelligent tutorial system interventions includes studies where the comparison activity was human tutoring (average d=-0.”
Gazi, S. E., & Ibrahimi, A. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1206-3.ch014 · Cited by 38
Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks
Mitri, D. D., Schneider, J., Specht, M. M., & Drachsler, H. (2019) · Sensors · doi:10.3390/s19143099 · Cited by 41
“In traditional academic learning, ITS has been proved to be nearly as good as human tutors [20], outperforming other instruction methods and learning activities, including traditional classroom instruction, reading printed text or electronic materials, computer-assisted instruction, laboratory or homework assignments…”
BackgroundEvolution of an Intelligent Deductive Logic Tutor Using Data-Driven Elements
Mostafavi, B., & Barnes, T. M. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0112-1 · Cited by 37
“…student instruction is one of the most effective educational strategies, and students who receive individualized instruction often perform significantly better on performance evaluations than students who receive classroom instruction (Ma et al. 2014; VanLehn 2011; Steenbergen-Hu and Cooper 2014).”
BackgroundHow Teachers Use Data to Help Students Learn: Contextual Inquiry for the Design of a Dashboard
Xhakaj, F., Aleven, V. A. W. M. M., & McLaren, B. M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45153-4_26 · Cited by 43
“A number of meta-reviews indicate that ITS can enhance student learning in actual classrooms, compared to other forms of instruction [16, 19, 23, 24, 27].”
BackgroundSchez-Sobrino, S., Gmez-Portes, C., Vallejo, D., Glez-Morcillo, C., & Redondo, M. Á. (2020) · Applied Sciences · doi:10.3390/app10041518 · Cited by 46
“Several works conducted a literature review where they collected the results after evaluating an ITS whose conclusions confirmed the effectiveness of using this type of tool in the classroom [10,23].”
BackgroundWyatt‐Smith, C., Lingard, B., & Heck, E. (2021) · Routledge eBooks · doi:10.4324/9781003045793 · Cited by 67
Engagement Detection and Its Applications in Learning: A Tutorial and Selective Review
Booth, B. M., Bosch, N., & D’Mello, S. K. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3309560 · Cited by 54
“, [195], [196], and [197]) but they also result in increased levels of boredom [150], [198], [199], which may hinder the development of sustained engagement in the long term.”
BackgroundA Common Framework for Artificial Intelligence in Higher Education (AAI-HE Model)
Jantakun, T., Jantakun, K., & Jantakoon, T. (2021) · International Education Studies · doi:10.5539/ies.v14n11p94 · Cited by 57
Artificial Intelligence in Higher Education: A Bibliometric Approach
Kavitha, K., Joshith, V. P., Rajeev, N. P., & Asha, S. (2024) · European Journal of Educational Research · doi:10.12973/eu-jer.13.3.1121 · Cited by 69
Λαμπρόπουλος, Γ. (2025) · Applied Sciences · doi:10.3390/app15063223 · Cited by 69
Artificial Neural Networks and other AI Applications for Business Management Decision Support
Walczak, S. M. (2016) · International Journal of Sociotechnology and Knowledge Development · doi:10.4018/ijskd.2016100101 · Cited by 32
Metadata
- Authors
- Steenbergen-Hu, S., & Cooper, H.
- Venue
- Journal of Educational Psychology
- Published
- 2014-05
- Volume
- 106
- Issue
- 2
- Pages
- 331-347
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2316441559
- Cited by
- 355
- References
- 68
- Citations by year
- 2026: 48 · 2025: 55 · 2024: 50 · 2023: 28 · 2022: 24 · 2021: 31 · 2020: 31 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 15 · 2015: 12 · 2014: 4
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Journal of Educational Psychology (publishedVersion)
- DukeSpace (Duke University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- intelligent tutoring systems, university students, academic learning, higher education, computer-assisted instruction, AutoTutor, ALKIS, effect size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.