The DOI matches its Crossref record; 127 of the 127 references it cites were found in a registry or an index.
Intelligent tutoring systems and learning outcomes: A meta-analysis
Abstract
Intelligent Tutoring Systems (ITS) are computer programs that model learners’ psychological states to provide individualized instruction. They have been developed for diverse subject areas (e.g., algebra, medicine, law, reading) to help learners acquire domain-specific, cognitive and metacognitive knowledge. A meta-analysis was conducted on research that compared the outcomes from students learning from ITS to those learning from non-ITS learning environments. The meta-analysis examined how effect sizes varied with type of ITS, type of comparison treatment received by learners, type of learning outcome, whether knowledge to be learned was procedural or declarative, and other factors. After a search of major bibliographic databases, 107 effect sizes involving 14,321 participants were extracted and analyzed. The use of ITS was associated with greater achievement in comparison with teacher-led, large-group instruction (g .42), non-ITS computer-based instruction (g .57), and textbooks or workbooks (g .35). There was no significant difference between learning from ITS and learning from individualized human tutoring (g –.11) or small-group instruction (g .05). Significant, positive mean effect sizes were found regardless of whether the ITS was used as the principal means of instruction, a supplement to teacher-led instruction, an integral component of teacher-led instruction, or an aid to homework. Significant, positive effect sizes were found at all levels of education, in almost all subject domains evaluated, and whether or not the ITS provided feedback or modeled student misconceptions. The claim that ITS are relatively effective tools for learning is consistent with our analysis of potential publication bias.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Every learner has unique learning contexts based on their prior knowledge about the topic, social background, economic well-being and emotional state [41].”
From Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021 (2022) · cites this paper as [41]
In the citing paper
“Despite being limited in terms of the domain that a particular intelligent tutoring system addresses, they have proven to be effective in providing relevant content to students, interacting with students [6] and improving students’ academic performance [18, 41].”
From Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021 (2022) · cites this paper as [41]
Citations
Cites
(1988) · Cited by 16,582
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,377
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
(2000) · Cited by 9,198
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
An introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 1×
“Another notable contribution of VanLehn's review was that he reported an effect size of d ϭ 0.79 for human tutoring compared with no-treatment controls, a value much smaller than the frequently cited, twosigma effect sizes reported by Bloom (1984).”
ResultInfluentialHays, W. (1983) · Contemporary Psychology · doi:10.1037/022267 · Cited by 2,399
Flagged first, then by citations
Cited by
Sorted by citations
Vision, challenges, roles and research issues of Artificial Intelligence in Education
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2020.100001 · Cited by 1,435
“Several meta-analytic studies have demonstrated the effectiveness of intelligent tutoring systems to promote learning outcomes (Ma, Adesope, Nesbit, & Liu, 2014; Steenbergen-Hu & Cooper, 2014; VanLehn, 2011).”
BackgroundAI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking
Gerlich, M. (2025) · Societies · doi:10.3390/soc15010006 · Cited by 1,248
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
Artificial Intelligence trends in education: a narrative overview
Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.08.233 · Cited by 956
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930
“Finally, the meta-analysis by Ma et al. (2014) analyzed 107 findings from 73 separate reports.”
MethodExplainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Huang, A. Y. Q., Lu, O. H. T., & Yang, S. J. H. (2023) · Computers & Education · doi:10.1016/j.compedu.2022.104684 · Cited by 605
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558257 · Cited by 605
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103897 · Cited by 574
“The assumption of a novelty effect is not in line with the result of Ma et al. (2014), who found that interventions with durations of less than four weeks denote less-positive student learning outcomes.”
BackgroundResultInfluentialStephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1619259 · Cited by 524
“ITSs provide individualized instruction and have the potential for significantly higher achievement outcomes than other modes of instruction (except small-group human tutoring and individual human tutoring when results were comparable) (Ma, Adesope, Nesbit, & Liu, 2014).”
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548
A review of AI teaching and learning from 2000 to 2020
Ng, D. T. K., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. W. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11491-w · Cited by 556
Differentiated Instruction in Secondary Education: A Systematic Review of Research Evidence
Smale-Jacobse, A. E., Meijer, A., Helms‐Lorenz, M., & Maulana, R. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02366 · Cited by 560
“Review studies show that such adaptive ICT applications may considerably improve student achievement (Ma et al., 2014; Van der Kleij et al., 2015; Kulik and Fletcher, 2016; Shute and Rahimi, 2017).”
BackgroundArtificial Intelligence in Education (AIEd): a high-level academic and industry note 2021
Chaudhry, M. A., & Kazim, E. (2022) · AI and Ethics · doi:10.1007/s43681-021-00074-z · Cited by 425
“According to Luckin [37] from University College London, ‘AI would provide a fairer, richer assessment system that would evaluate students across a longer period of time and from an evidence-based, value-added perspective’.”
BackgroundInfluentialSynthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education
Belland, B. R., Walker, A. E., Kim, N. J., & Lefler, M. (2017) · Review of Educational Research · doi:10.3102/0034654316670999 · Cited by 346
“63 in humanities and social science (Ma et al., 2014).”
BackgroundMethodInfluentialPower to the Teachers: An Exploratory Review on Artificial Intelligence in Education
Lameras, P., & Arnab, S. (2021) · Information · doi:10.3390/info13010014 · Cited by 291
“Ma et al. [79] conducted a meta-analysis that compared the outcomes on ITSs that were assimilated by students for developing subject domain understandings to those from non-ITS learning environments.”
ResultCukurova, M. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13514 · Cited by 302
The Cognitive-Affective-Social Theory of Learning in digital Environments (CASTLE)
Schneider, S., Beege, M., Nebel, S., Schnaubert, L., & Rey, G. D. (2021) · Educational Psychology Review · doi:10.1007/s10648-021-09626-5 · Cited by 395
Wang, S., Christensen, C., Cui, W., Tong, R., Yarnall, L., Shear, L., & Feng, M. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2020.1808794 · Cited by 277
“For example, developers have created these systems to present content, pose questions, assign tasks, provide hints, answer questions, or suggest improvements in the behavior or attitudes of learners (Ma et al., 2014).”
BackgroundInducing Self-Explanation: a Meta-Analysis
Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F., & Winne, P. H. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9434-x · Cited by 286
“Computer-based instruction (CBI) was defined as offering response feedback with no student modeling, and intelligent tutoring systems (ITS) were defined as modeling student knowledge to adapt feedback, task selection, or prompts (Ma et al. 2014).”
Self-citationFazlollahi, A. M., Bakhaidar, M., Alsayegh, A., Yilmaz, R., Winkler-Schwartz, A., Mirchi, N., Langleben, I., Ledwos, N., Sabbagh, A. J., Bajunaid, K., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.49008 · Cited by 285
“1) Sex Men 9 (39) 10 (43) 10 (42) Women 14 (61) 13 (57) 14 (58) Undergraduate medical training level Medicine Preparatorya 9 (39) 10 (43) 7 (29) First year 8 (35) 8 (35) 9 (38) Second year 6 (26) 5 (22) 8 (33) Institution McGill University 14 (61) 8 (35) 10 (42) University of Montreal 3 (13) 7 (30) 7 (29) University o…”
Developing Personalized Education: A Dynamic Framework
Tetzlaff, L., Schmiedek, F., & Brod, G. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09570-w · Cited by 321
“ITS are, by definition, computer programs that model learners ’ psychological states to provide personalized instruction (Ma et al. 2014).”
BackgroundInfluentialPalomares, I., Martínez-Cámara, E., Montes, R., García-Moral, P., Chiachio, M., Chiachio, J., Alonso, S., Melero, F. J., Molina, D., Fernández, B., et al. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02264-y · Cited by 276
Variability of teachers’ technology integration in the classroom: A matter of utility!
Backfisch, I., Lachner, A., Stürmer, K., & Scheiter, K. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104159 · Cited by 312
Kim, N. J., Belland, B. R., & Walker, A. E. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9419-1 · Cited by 220
“Hints are clues or suggestions to help students go forward (Melero et al. 2011).”
BackgroundMustafa, M. Y., Tlili, A., Lampropoulos, G., Huang, R., Jandrić, P., Zhao, J., Salha, S., Xu, L., Panda, S., Kinshuk, et al. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00350-5 · Cited by 235
“…models to offer student-centered, personalized, and self-paced learning opportunities to help students acquire knowledge (Graesser et al., 2012; Ma et al., 2014) by tailoring the instructional methods, approaches, and activities according to each individual learner (Mousavinasab et al., 2021).”
Artificial Intelligence as an Effective Classroom Assistant
du Boulay, B. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.93 · Cited by 213
Sepasgozar, S. M. E., Karimi, R., Farahzadi, L., Moezzi, F., Shirowzhan, S., Ebrahimzadeh, S. M., Hui, F. K. P., & Aye, L. (2020) · Applied Sciences · doi:10.3390/app10093074 · Cited by 224
La Inteligencia Artificial como recurso educativo durante la formación inicial del profesorado
del Puerto, D. A., & Esteban, P. G. (2022) · RIED Revista Iberoamericana de Educación a Distancia · doi:10.5944/ried.25.2.32332 · Cited by 416
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-23204-7_14 · Cited by 189
Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025252 · Cited by 193
“This is significant since, in the meta-analyses of Steenbergen-Hu and Cooper,(15) Ma et al,(18) and Kulik and Fletcher,(17) AEEs seem to be more effective in postsecondary students(17) 18 and for learning subjects related to biology, physiology and social science.(18) Moreover, previous meta-analyses focused on the ef…”
BackgroundImmersive, interactive virtual field trips promote science learning
Mead, C., Buxner, S. R., Bruce, G., Taylor, W. L., Semken, S. C., & Anbar, A. D. (2019) · Journal of Geoscience Education · doi:10.1080/10899995.2019.1565285 · Cited by 183
“Adaptive learning has been shown to be more effective than comparable nonadaptive designs (Kulik & Fletcher, 2016; Ma et al., 2014; VanLehn, 2011).”
BackgroundIntelligent Tutoring Systems: A Comprehensive Historical Survey with Recent Developments
Alkhatlan, A., & Kalita, J. K. (2019) · International Journal of Computer Applications · doi:10.5120/ijca2019918451 · Cited by 243
“[22] was conducted in 2014 for the purpose of comparing the learning outcomes for those who learn by using ITSs and those who learn in non-ITS learning environments.”
BackgroundEffectiveness of Mobile Devices on Vocabulary Learning: A Meta-Analysis
Mahdi, H. S. (2017) · Journal of Educational Computing Research · doi:10.1177/0735633117698826 · Cited by 160
“On the other hand, a random-effects model operates under the assumption that there is more than one true effect and that the ESs could be different from study to the other (Ma, Adesope, Nesbit, & Liu, 2014).”
BackgroundA meta-analysis of signaling principle in multimedia learning environments
Alpizar, D., Adesope, O. O., & Wong, R. M. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09748-7 · Cited by 157
“The use of educational videos (see Guo et al. 2014), homework assessment, tutoring programs (see Cole and Todd 2003; Ma et al. 2014), and online discussion forums (see Mazzolini and Maddison 2003; Thomas 2002) highlight how the use of technology is impacting our society today (Department of…”
BackgroundSelf-citationReconsidering the voice effect when learning from a virtual human
Craig, S. D., & Schroeder, N. L. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.003 · Cited by 151
Wang, H., Tlili, A., Huang, R., Cai, Z., Li, M., Cheng, Z., Yang, D., Li, M., Zhu, X., & Fei, C. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11555-x · Cited by 147
“…student diagnose module which collects and updates the information about students’ knowledge, skills, behaviors, responses, learning styles, etc. (Ma et al., 2014); (3) instructional module which focuses on the strategies and methods of teaching and delivering customized learning content…”
BackgroundMethodResultInfluentialXu, Z., Wijekumar, K. K., Ramírez, G., Hu, X., & Irey, R. C. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12758 · Cited by 181
Christopoulos, A., & Sprangers, P. (2021) · Cogent Education · doi:10.1080/2331186x.2021.1964690 · Cited by 145
“Computer-based instruction has yielded positive results in areas such as student motivation (Turk & Akyuz, 2016), satisfaction (Özyurt et al., 2014), performance (Ma et al., 2014), and learning outcomes (Van der Kleij et al., 2015).”
BackgroundThe Normalized Programming State Model
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787710 · Cited by 144
Self-citationWang, T., & Cheng, E. C. K. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100031 · Cited by 141
Looking Beyond the Hype: Understanding the Effects of AI on Learning
Bauer, E., Greiff, S., Graesser, A. C., Scheiter, K., & Sailer, M. (2025) · Educational Psychology Review · doi:10.1007/s10648-025-10020-8 · Cited by 153
“These examples illustrate how LLMs can be integrated with ITS components and pedagogical principles, but also highlight challenges in balancing flexibility, content quality, pedagogical quality, and personalized instruction.”
InfluentialOnTask: Delivering Data-Informed, Personalized Learning Support Actions
Pardo, A., Bartimote-Aufflick, K., Shum, S. B., Dawson, S., Gao, J. P., Gašević, D., Leichtweis, S., Liu, D., Martínez‐Maldonado, R., Mirriahi, N., et al. (2018) · Journal of Learning Analytics · doi:10.18608/jla.2018.53.15 · Cited by 140
Reinhold, F., Hoch, S., Werner, B., Richter-Gebert, J., & Reiss, K. M. (2019) · Learning and Instruction · doi:10.1016/j.learninstruc.2019.101264 · Cited by 179
“In addition, these differences between low-achieving and highachieving students lead to new questions that need further investigation and cannot be answered based on our studies: as appropriate feedback as well as adaptive task difficulty have shown to be beneficial for the learning process in different studies (Arroy…”
BackgroundResultInfluentialGuo, L., Wang, D., Gu, F., Li, Y., Wang, Y., & Zhou, R. (2021) · Asia Pacific Education Review · doi:10.1007/s12564-021-09697-7 · Cited by 124
Modelling Human Teaching Tactics and Strategies for Tutoring Systems: 14 Years On
du Boulay, B., & Luckin, R. (2015) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0053-0 · Cited by 124
“In terms of understanding recent progress, largely in the cognitive rather than the motivational aspects of modelling teaching, we note a meta-analysis of the learning outcomes of using Intelligent Tutoring Systems (Ma et al. 2014).”
BackgroundLearner modelling: systematic review of the literature from the last 5 years
Abyaa, A., Idrissi, M. K., & Bennani, S. (2019) · Educational Technology Research and Development · doi:10.1007/s11423-018-09644-1 · Cited by 131
A Systematic Literature Review of Intelligent Tutoring Systems With Dialogue in Natural Language
Paladines, J., & Ramírez, J. (2020) · IEEE Access · doi:10.1109/access.2020.3021383 · Cited by 143
“As a result, we have considered the ITS types mentioned in [4]: model-tracing tutors (MTTs), constraint-based modeling (CBM), Bayesian network modeling (BNM), and expectation and misconceptions tailoring (EMT).”
Artificial Intelligence in Education: Mathematics Teachers’ Perspectives, Practices and Challenges
Tashtoush, M. A., Wardat, Y., AlAli, R. M., & Saleh, S. G. (2024) · Iraqi Journal for Computer Science and Mathematics · doi:10.52866/ijcsm.2024.05.01.004 · Cited by 161
“A literature review on AI in education reveals that it has been extensively implemented in various sectors, including higher education [7], K-12 education [1], student assessment [15], robotics [16], data mining [17], and intelligent tutoring systems (ITS) [18].”
BackgroundAuthoring Tools for Designing Intelligent Tutoring Systems: a Systematic Review of the Literature
Dermeval, D., Paiva, R., Bittencourt, I. I., Vassileva, J., & Borges, D. C. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0157-9 · Cited by 134
“As reported by du Boulay (2016), there have been a number of recent positive reviews in support of the effectiveness of ITSs (Kulik and Fletcher 2016; Ma et al. 2014; Steenbergen-Hu and Cooper 2013, 2014; VanLehn 2011).”
BackgroundMethodResultThe Science of Workplace Instruction: Learning and Development Applied to Work
Kraiger, K., & Ford, J. K. (2020) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-012420-060109 · Cited by 162
Enhancing problem-solving skills with smart personal assistant technology
Winkler, R., Söllner, M., & Leimeister, J. M. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104148 · Cited by 140
“3 the last five to ten years, a lot of researchers were able to show that step-based computer tutoring systems were nearly as effective as average human tutors and more effective compared to no tutoring condition and teacher-led classroom instruction (Ma et al. 2014; Nesbit et al. 2014; Vanlehn 2011).”
BackgroundMcTigue, E. M., Solheim, O. J., Zimmer, W. K., & Uppstad, P. H. (2020) · Reading Research Quarterly · doi:10.1002/rrq.256 · Cited by 113
Scaffolding Computational Thinking With ChatGPT
Liao, J., Zhong, L., Zhe, L., Xu, H., Liu, M., & Xie, T. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3392896 · Cited by 144
“Melero et al. [38] and Ma et al. [39] proposed the concept of systematic instruction scaffolding as the systematic use of technological tools or environmental designs, such as learning management systems and intelligent tutoring systems including computers or AI.”
BackgroundChen, J. J., & Lin, J. C. (2023) · Contemporary Issues in Early Childhood · doi:10.1177/14639491231169813 · Cited by 106
Development and Adoption of an Adaptive Learning System
Khosravi, H., Sadiq, S. W., & Gašević, D. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366900 · Cited by 123
“using an ITS or individualised human tutoring [33].”
MethodIncorporation of Artificial Intelligence Tutoring Techniques in Mathematics
Pappas, M. A., & Drigas, A. S. (2016) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v6i4.6063 · Cited by 103
“Each constraint consists of three components: 1) a relevance condition indicating when the constraint is applicable, 2) a satisfaction condition and 3) a feedback message for student’s errors (Ma et al, 2014)”
BackgroundEvaluating Artificial Intelligence in Education for Next Generation
Joshi, S., Rambola, R. K., & Churi, P. P. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1714/1/012039 · Cited by 139
“Some focus on subjects with mathematical rules, some teach reading or writing, and still, others try to teach general topic skills, such as self-paced reading strategies [24] [25] [29].”
BackgroundDesigning Adaptive Instruction for Teams: a Meta-Analysis
Sottilare, R. A., Burke, C. S., Eduardo Salas, A., Sinatra, A. M., Johnston, J. H., & Gilbert, S. B. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0146-z · Cited by 101
“A recent review of artificial intelligence in education (AIED) meta-analyses by du Boulay (2016) noted investigations by VanLehn (2011), Ma et al. (2014), Kulik and Fletcher (2015), Steenbergen-Hu and Cooper (2013, 2014), and Pane et al. (2014).”
BackgroundElectronixTutor: an intelligent tutoring system with multiple learning resources for electronics
Graesser, A. C., Hu, X., Nye, B. D., VanLehn, K. A., Kumar, R. V., Heffernan, C., Heffernan, N. T., Woolf, B. P., Olney, A. M., Rus, V., et al. (2018) · International Journal of STEM Education · doi:10.1186/s40594-018-0110-y · Cited by 97
“…meta-analyses show positive effect sizes that vary from d = 0.05 (Dynarsky et al. 2007; Steenbergen-Hu and Cooper 2014) to d = 1.08 (Dodds and Fletcher 2004), but most hover between d = 0.40 and d = 0.80 (Kulik and Fletcher 2015; Ma et al. 2014; Steenbergen-Hu and Cooper 2013; VanLehn 2011).”
BackgroundInfluentialWang, X., Huang, R., Sommer, M., Pei, B., Shidfar, P., Rehman, M. S., Ritzhaupt, A. D., & Martin, F. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240459 · Cited by 170
“Our fi ndings are different from the meta-analysis conducted Ma et al. (2014) that found large effect size in learning humanities.”
InfluentialOn the promise of personalized learning for educational equity
Dumont, H., & Ready, D. D. (2023) · npj Science of Learning · doi:10.1038/s41539-023-00174-x · Cited by 141
“Personalized learning does fi nd support in the recent literature, with some studies reporting that such approaches, especially those supported by adaptive learning technologies, are more effective in raising student achievement than traditional non-adaptive instruction 34,37,39,42,43 .”
BackgroundChallenges and opportunities for classroom-based formative assessment and AI: a perspective article
Hopfenbeck, T. N., Zhang, Z., Sun, S. Z., Robertson, P., & McGrane, J. A. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1270700 · Cited by 125
“…and intelligent tutoring systems enables every student’s attainment to be individually and more precisely assessed, which facilitates more appropriate and targeted feedback based on their individual learning stage and trajectory (Ma et al., 2014; Tomasik et al., 2018; Mousavinasab et al., 2021).”
BackgroundScaffolding for Optimal Challenge in K–12 Problem-Based Learning
Kim, N. J., Belland, B. R., & Axelrod, D. B. (2018) · Interdisciplinary Journal of Problem-based Learning · doi:10.7771/1541-5015.1712 · Cited by 106
Lernen und Lehren mit digitalen Medien: Eine Standortbestimmung
Scheiter, K. (2021) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-021-01047-y · Cited by 126
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2017) · ACM Transactions on Computing Education · doi:10.1145/3120259 · Cited by 86
Self-citationPai, K.-C., Kuo, B., Liao, C.-H., & Liu, Y.-M. (2020) · Educational Psychology · doi:10.1080/01443410.2020.1731427 · Cited by 84
EnglishBot: An AI-Powered Conversational System for Second Language Learning
Ruan, S., Jiang, L., Xu, Q., Liu, Z., Davis, G. M., Brunskill, E., & Landay, J. A. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450648 · Cited by 113
“can be as effective as human tutoring [63], and that use of an ITS in a study was associated with greater achievement in comparison with teacher-led large-group instruction, non-ITS computer-based instruction, and textbooks or workbooks [41].”
BackgroundA feedback model for data-rich learning experiences
Pardo, A. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2017.1356905 · Cited by 82
“Recent studies have shown the positive effect of these systems with respect to learning outcomes (Ma et al. 2014) for which feedback is an essential element.”
BackgroundRiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities
Khosravi, H., Kitto, K., & Williams, J. J. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.63.12 · Cited by 108
“44 and d = 54 which gives support to the claim that using RiPPLE does indeed provide measurable learning gains; however, the effect is considered to be at a medium level (Ma et al., 2014) and less effective compared to many of the studies that were discussed in Section 2.”
BackgroundResultHwang, S. (2022) · Sustainability · doi:10.3390/su142013185 · Cited by 107
Xhakaj, F., Aleven, V., & McLaren, B. M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66610-5_23 · Cited by 81
“ITSs have been shown to enhance student learning [8, 11, 19].”
BackgroundChow, J. C., & Ekholm, E. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9437-7 · Cited by 80
A meta-analysis of the effectiveness of ALEKS on learning
Fang, Y., Ren, Z., Hu, X., & Graesser, A. C. (2019) · Educational Psychology · doi:10.1080/01443410.2018.1495829 · Cited by 79
“This is likely the reason, why empirical studies using ALEKS were excluded from recent ITS meta-analyses (Kulik & Fletcher, 2015; Ma, Adesope, Nesbit, & Liu, 2014; Steenbergen-Hu & Cooper, 2013; VanLehn, 2011).”
BackgroundResultInfluentialEscape from the Skinner Box: The case for contemporary intelligent learning environments
du Boulay, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12860 · Cited by 80
“…of formal education, seven meta-reviews have compared the quality of the learning outcomes with ILES as compared to using more traditional methods (Kulik & Fletcher, 2016; Ma, Adesope, Nesbit, & Liu, 2014; Nesbit, Adesope, Liu, & Ma, 2014; Steenbergen-Hu & Cooper, 2013, 2014; VanLehn, 2011).”
BackgroundMethodInfluentialTeachers’ technology use for teaching: Comparing two explanatory mechanisms
Backfisch, I., Scherer, R., Siddiq, F., Lachner, A., & Scheiter, K. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103390 · Cited by 98
Fernández-Herrero, J. (2024) · Education Sciences · doi:10.3390/educsci14080839 · Cited by 107
“However, they note that [93] found that studies using ITSs in the humanities and social sciences had a significantly higher weighted mean effect size.”
Factors Influencing Students' Behavioral Intention to Continue Artificial Intelligence Learning
Chai, C. S., Lin, P.-Y., Jong, M. S.-Y., Dai, Y., Chiu, T. K. F., & Huang, B. (2020) · 2020 International Symposium on Educational Technology (ISET) · doi:10.1109/iset49818.2020.00040 · Cited by 106
Dong, L., Tang, X., & Wang, X. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100400 · Cited by 116
Embracing imperfection in learning analytics
Kitto, K., Shum, S. B., & Gibson, A. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170413 · Cited by 75
“For example, Intelligent Tutoring Systems (ITS) have been shown to positively affect student learning outcomes when compared to conventional educational experiences [33, 34, 51].”
BackgroundAI and the Future of Learning: Expert Panel Report
Roschelle, J., Lester, J., & Fusco, J. (2020) · doi:10.51388/20.500.12265/106 · Cited by 76
“More generally, meta-analyses of ITS approaches have found positive impacts on student learning (Van Lehn, 2011; Ma et al., 2014; Kulik & Fletcher, 2016).”
BackgroundThe effect of chatbots on learning: a meta-analysis of empirical research
Alemdağ, E. (2023) · Journal of Research on Technology in Education · doi:10.1080/15391523.2023.2255698 · Cited by 87
“For example, Ma et al. (2014) first defined ITSs as a computer program that provides tutoring functions, creates multidimensional models for each student’s psychological state, and adapts their functions based on these models.”
InfluentialA survey of adaptive context-aware learning environments
Hasanov, A., Laine, T. H., & Chung, T. (2019) · Journal of Ambient Intelligence and Smart Environments · doi:10.3233/ais-190534 · Cited by 81
“Several studies that were conducted in the field of adaptive and context-aware learning show positive effects of the usage of ACALEs on learning and teaching across domains and levels, including but not limited to effectiveness, efficiency, interaction, support, immersion, and motivation [40,44,56,57,78,79,87].”
BackgroundGraesser, A. C., Sabatini, J., & Li, H. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-113042 · Cited by 101
Bernard, R. M., Borokhovski, E., Schmid, R. F., Waddington, D. I., & Pickup, D. I. (2019) · Campbell Systematic Reviews · doi:10.1002/cl2.1017 · Cited by 110
An Interaction Design for Machine Teaching to Develop AI Tutors
Weitekamp, D. P., Harpstead, E., & Koedinger, K. R. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376226 · Cited by 70
“In several studies ITSs have been shown to beneft learners when used alone or in combination with traditional instruction [7, 24, 25, 12].”
BackgroundImplementation of technology-supported personalized learning—its impact on instructional quality
Schmid, R., Pauli, C., Стеблер, Р., Reusser, K., & Petko, D. (2022) · The Journal of Educational Research · doi:10.1080/00220671.2022.2089086 · Cited by 91
“Meta-analyses of intelligent tutoring systems (ITS) indicate positive effects on student outcomes when compared to non-individualized forms of teacher-guided instruction or non-ITS educational software (Kulik & Fletcher, 2016; Ma et al., 2014).”
BackgroundMeasuring the effect of ARS on academic performance: A global meta-analysis
Castillo-Manzano, J. I., Castro‐Nuño, M., Valpuesta, L. L., Díaz, M. T. S., & Ovando, R. Y. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.02.007 · Cited by 75
Dutt, S., Ahuja, N. J., & Kumar, M. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10713-x · Cited by 68
“ITS application provides comfort to the learner to learn and perform with ability, availability of time and interest, and that too without the human teacher intervention (Ma et al., 2014).”
BackgroundÜZÜMCÜ, Ö., & AÇILMIŞ, H. (2023) · Technology Knowledge and Learning · doi:10.1007/s10758-023-09687-1 · Cited by 98
“It is also defined as operating like human intelligence for learning, cognitive problem-solving, and model recognition (Ma et al., 2014).”
Belland, B. R., Walker, A. E., & Kim, N. J. (2017) · Review of Educational Research · doi:10.3102/0034654317723009 · Cited by 90
“Meta-analyses have also been performed among specific subtypes of scaffolding, such as that in intelligent tutoring systems (Ma et al., 2014; Steenbergen-Hu & Cooper, 2013, 2014; VanLehn, 2011), dynamic assessment (Swanson & Lussier, 2001), and scaffolding for students with learning disabilities (Swanson & Deshler, 20…”
BackgroundMethodInfluentialLeveraging Big Data to Help Each Learner and Accelerate Learning Science
Winne, P. H. (2017) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146811711900305 · Cited by 68
Providing feedback on computer-based algebra homework in middle-school classrooms
Fyfe, E. R. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.05.082 · Cited by 71
McLaren, B. M., Richey, J. E., Nguyen, H. A., & Hou, X. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104366 · Cited by 77
Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program
Huang, X., Craig, S. D., Xie, J., Graesser, A. C., & Hu, X. (2016) · Learning and Individual Differences · doi:10.1016/j.lindif.2016.01.012 · Cited by 74
Collins, B. R., Black, E. W., & Rarey, K. E. (2024) · Clinical Anatomy · doi:10.1002/ca.24178 · Cited by 61
“An earlier meta-analysis involving 107 effect sizes found that intelligent tutoring systems improve student achievement with effect sizes ranging from 0.35 to 0.57 depending on the control comparison (Ma et al., 2014).”
InfluentialThe potential of learning with (and not from) artificial intelligence in education
Chichekian, T., & Benteux, B. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.903051 · Cited by 75
“Ma et al. (2014) indicated that whenever a process variable was reported in a study, it was often only meaningful in the context of the learning task.”
BackgroundMethodResultInfluentialΛαμπρόπουλος, Γ. (2023) · Springer series on cultural computing · doi:10.1007/978-3-031-27166-3_8 · Cited by 71
IDE-Based Learning Analytics for Computing Education
Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017) · ACM Transactions on Computing Education · doi:10.1145/3105759 · Cited by 67
Broadbent, J., Panadero, E., Lodge, J. M., & de Barba, P. (2020) · Handbook of Research in Educational Communications and Technology · doi:10.1007/978-3-030-36119-8_3 · Cited by 80
Metadata
- Authors
- Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q.
- Affiliations
- Simon Fraser University (CA)
- Washington State University (US)
- Venue
- Journal of Educational Psychology
- Published
- 2014-11
- Volume
- 106
- Issue
- 4
- Pages
- 901-918
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2117655204
- Cited by
- 840
- References
- 127
- Citations by year
- 2026: 179 · 2025: 132 · 2024: 99 · 2023: 65 · 2022: 67 · 2021: 69 · 2020: 70 · 2019: 41 · 2018: 43 · 2017: 36 · 2016: 22 · 2015: 17
- FWCI
- 24.90
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Psychology (publishedVersion)
- Funders
- Social Sciences and Humanities Research Council of Canada
- Washington State University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- intelligent tutoring systems, learning outcomes, personalized learning, computer-assisted instruction, teacher-centered instruction, small group instruction, homework help, metacognitive knowledge, feedback, misconceptions, publication bias, effect size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
