A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning.
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
© 2013 American Psychological Association.This meta-analysis synthesizes research on the effectiveness of intelligent tutoring systems (ITS) for college students. Thirty-five reports were found containing 39 studies assessing the effectiveness of 22 types of ITS in higher education settings. Most frequently studied were AutoTutor, Assessment and Learning in Knowledge Spaces, eXtended Tutor-Expert System, and Web Interface for Statistics Education. Major findings include (a) Overall, ITS had a moderate positive effect on college students' academic learning (g = .32 to g = .37); (b) ITS were less effective than human tutoring, but they outperformed all other instruction methods and learning activities, including traditional classroom instruction, reading printed text or computerized materials, computer-assisted instruction, laboratory or homework assignments, and no-treatment control; (c) ITS's effectiveness did not significantly differ by different ITS, subject domain, or the manner or degree of their involvement in instruction and learning; and (d) effectiveness in earlier studies appeared to be significantly greater than that in more recent studies. In addition, there is some evidence suggesting the importance of teachers and pedagogy in ITS-assisted learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another meta-analysis calculated a point estimate for intelligent tutoring systems in college-level business education of ḡ = 0.16 (Steenbergen-Hu & Cooper, 2014).”
From Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education (2017) · cites this paper as Steenbergen-Hu & Cooper, 2014
In the citing paper
“and that ITS were less effective that human tutoring”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“Steenbergen-Hu and Cooper (2014) found that ITS had a moderate effect of students' learning”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“but ITS outperformed all other instruction methods (such as traditional classroom instruction, reading printed or digital text, or homework assignments).”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“most of the strategies and functions that our participants desired to improve the usability issues of ChatGPT, such as hints, feedback, prompt suggestions, a learning map and navigation system, and AI-Helper agents, resemble the traits that ITSs have long had and are already proven to be effective.”
From The Promise and Peril of ChatGPT in Higher Education: Opportunities, Challenges, and Design Implications (2024) · cites this paper as [80]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · cited 1×
“We further conducted Duval and Tweedie’s (2000) trim and fill procedures and found that the adjusted average effect sizes remained the same as the observed ones, which were g .”
The Role of Goal Orientation in Self-Regulated Learning
Pintrich, P. R. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50043-3 · Cited by 5,173 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853 · cited 1×
“Graesser et al.’s (2011) review of ITS posed a number of questions that need answering.”
A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students
Pintrich, P. R. (2004) · Educational Psychology Review · doi:10.1007/s10648-004-0006-x · Cited by 3,269 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 7×
“Bloom (1984) reported that expert human tutors can help students achieve learning gains as large as two sigmas. Although not as effective as what Bloom (1984) found, a recent meta-review by VanLehn (2011) found that human tutoring had a positive impact of d .79 on students’ learning. Many ITS are designed to follow th…”
InfluentialVanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323 · cited 4×
“VanLehn (2011) reviewed randomized experiments that compared the effectiveness of human tutoring, computer tutoring, and no tutoring.”
InfluentialCognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · cited 1×
“For example, Cognitive Tutors are built on a cognitive theory called adaptive control of thought (ACT, or ACT-R in its updated form; Anderson et al., 1995).”
Sample criteria for testing outlying observations
Grubbs, F. E. (1950) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729885 · Cited by 1,631 · cited 4×
“Grubbs (1950) tests detected no effect-size outliers.”
InfluentialBuilding Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 1,376
Intelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
Cited by
Items 301–355 of 355
Chevalère, J., Berthon, M., Rocher, N., Pailler, D., Mazenod, V., & Huguet, P. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13424-9 · Cited by 1
“It is well established that CAI offers robust benefits compared to conventional class - room learning situations (Chevalère et al., 2021 , 2022 ; Kulik & Kulik, 1991 ; Ma et al., 2014 ; Major et al., 2021 ; Murphy et al., 2020 ; Steenbergen-Hu & Cooper, 2014).”
Digitales adaptives Lernen in der Ökonomischen Bildung
Bode, D., Dietrich, F., Hinrichs, L., Koch, M. R., & Loerwald, D. (2025) · Springer eBooks · doi:10.1007/978-3-658-45265-0_3
Liu, Q., Prakash, H. M., & Heckler, A. F. (2025) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.21.010121
Mishra, S., & Lal, N. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-852-120251002
Shalavina, A. A., Shevtsova, Y. V., & Владимировна, П. С. (2025) · CARDIOVASCULAR THERAPY AND PREVENTION · doi:10.15829/1728-8800-2025-4471
Integrating AI and Flipped Learning: A Modern Framework for Initial Teacher Training
Falzone, Y., Leone, A., Pitrella, V., Longo, L., & Carlo, D. R. D. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94002-6_10 · Cited by 1
Akter, R., Afrin, N., Fahim, M. R. I., & Hoque, U. S. (2025) · International Journal of Research in Business and Social Science (2147-4478) · doi:10.20525/ijrbs.v14i6.4303 · Cited by 3
A Bibliometric Analysis of Visualizing Knowledge Mapping for Intelligent Tutoring Systems
Ye, X., Wang, J., & Qin, Q. (2025) · International Conference on Education and Training Technologies (ICETT) · doi:10.1109/icett66247.2025.11137303
“…is not as good as human tutoring, they are superior to all other teaching methods and learning activities, including traditional classroom instruction, reading printed texts or computerized materials, computer-assisted instruction, laboratory or homework assignments, and no-treatment controls [3].”
AI-Driven Innovation in Universities: A Multifaceted Analysis of Current and Future Trends
Dhanasekaran, B., Natarajan, S. K., Dubey, A. K., Palarimath, S., & Gulvady, S. (2025) · Studies in big data · doi:10.1007/978-3-031-97609-4_10
Fink, M. C., Heitzmann, N., Siebeck, M., Fischer, F., & Fischer, M. R. (2021) · Figshare · doi:10.6084/m9.figshare.16767596
Get Ready for the Next Pandemic With AI-Supported eLearning
Baytak, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9521-9.ch017
NAIA: A Robust Artificial Intelligence Framework for Multi-Role Virtual Academic Assistance
M., A. F. P., Q., K. J. B., C., S. D. S., & M., C. G. Q. (2025) · Systems · doi:10.3390/systems13121091
“However, ITS outperformed all other kinds of instruction methods such as standard classroom lessons, reading books (digital or print), or assignments [6].”
Two-Stage Computerized Adaptive Learning Detection
Ye, S., & TORRE, J. D. L. (2025) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986251397632
“These computerized interventions, such as answer-based, step-based, and sub-step-based tutoring, have been deemed effective for K–12 through college-level students (Kulik & Fletcher, 2016; Steenbergen-Hu & Cooper, 2014; VanLehn, 2011).”
Using Virtual Companions on Serious Games to Increase Knowledge and Immersion of Students
França, C. S., Betencourt, G., Peres, B. J., Campos, P., & Gonçalves, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-11043-5_42
PLM - an Open Learner Model for Adaptive Learning Environments – Specification
Böck, F., & Landes, D. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346719
Godoi, D. F., Mohallem, A. G. D. C., Sant´Anna, B., Pires, E. M. S. G., Petreche, F. D. G., Paschoal, L. M., Couto, T. B., & Cunha, M. L. D. R. (2026) · Einstein (São Paulo) · doi:10.31744/einstein_journal/2026suppl_1ed0001
Lörch, L., & Brod, G. (2026) · Research Square · doi:10.21203/rs.3.rs-8797891/v1
Negură, C., & Ionescu, S. C. (2025) · International Conference of Management and Industrial Engineering · doi:10.56177/12icmie2025.154
Chang, L. (2026) · Research Square · doi:10.21203/rs.3.rs-8998934/v1 · Cited by 1
LLM Practice Problem Generation in a Beginning Programming Class
Gold, K. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-95-4423-3_1
Sacré, M. (2021) · theses.fr (ABES) · doi:10.70675/5947a044zad77z404ez9371za7ec87a5e1b1 · Cited by 2
Design Tools for Gateway Interface Ecosystems: A Socio-Ecological Approach
Guttman, R. D., Herckis, L., Culyba, S., Hymes, K., & Hammer, J. (2026) · SN Computer Science · doi:10.1007/s42979-026-04931-8
Ali, S. R., Khan, M. O., Faraz, M., & Imran, S. A. (2026) · Regional lens. · doi:10.55737/rl.2025.41184
Personalized Learning in Programming: Evaluating a Chatbot Using Naive Bayes Versus IRT
Morales, M., Silva, J., Cubillos, C., & Mellado, R. (2025) · IEEE CHILEAN Conference on Electrical, Electronics Engineering, Information and Communication Technologies (CHILECON) · doi:10.1109/chilecon66915.2025.11476608
“Additionally, optimizing the interface and reducing session duration could minimize fatigue, thereby improving academic performance in conversational systems [20], [21], [22].”
Jamil, H. M. (2026) · ACM Transactions on Computing Education · doi:10.1145/3811544
“Although meta-analyses report moderate positive effects on college learning outcomes [19], evaluations are usually restricted to discrete tasks or single-course deployments.”
Kharitonov, A. E. (2025) · MODERN ADDITIONAL PROFESSIONAL PEDAGOGICAL EDUCATION · doi:10.54884/2414-1186-2025-8-4-023
Jian, X., & Li, N. (2026) · International Conference on AI in Education Technology and Applications (AIETA) · doi:10.1109/aieta69357.2026.11508949
“The system has been deployed on a cloud Web platform to provide full-cycle intelligent tutoring for Python programming courses in universities[1].”
AI-Powered learning platforms for employee upskilling and reskilling in organizations
Prabhakar, R., Singh, D. U., Husain, D. M., Rima, P., & Anil Kumar, K. (2026) · Journal of Progress in Artificial Intelligence · doi:10.68152/1726
Wu, J., Sun, H., & Xu, X. (2026) · The Asia-Pacific Education Researcher · doi:10.1007/s40299-026-01111-3
“Younger learners may benefit more from visual and contextual supports that reduce cognitive load during error correction (Sweller, 2020), whereas older learners may require less structured but more adaptive guid - ance (Steenbergen-Hu & Cooper, 2014).”
Stümke, M., Schmid, U., & Bernholt, S. (2026) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5c01626
Tracing affect in digital learning: Combining log data and introspective momentary assessments
Hilpert, J., & Janson, M. P. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105668
Cognitive Mechanisms Underlying Learning Gains in Intelligent Tutoring Systems
Ibanga, I. J., Asukwo, A. E., & Bashir, M. (2026) · Journal of Educational Technology Innovation and Applications · doi:10.56741/iistr.jetia.001801 · Cited by 1
Prompt Matters: How Pedagogical Engineering Shapes Behavior and Engagement with AI Tutors
Miller, K. (2026) · Technology Knowledge and Learning · doi:10.1007/s10758-026-09983-6
George, G. K., Kurien, G. P., & Ramakrishnan, N. (2026) · AIP conference proceedings · doi:10.1063/5.0327152
Zhang, Y., Gao, J., Huang, X., Li, S., Wu, M., Lin, Y., & Lobczowski, N. G. (2026) · Canadian Journal of Science Mathematics and Technology Education · doi:10.1007/s42330-026-00494-1
Schaaf, J., Rolfes, T., Nagy, G., & Heinze, A. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1842708
Role of AI-Powered Automation in Higher Education
Kumar, K. G., Periyasamy, S., Balamurugan, K., Sangamithra, T., Keerthana, N. V., & Udhaya, K. S. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7665-3.ch002
Schroeder, N. L., Zhang, S., Gladstone, J. R., & Bhat, S. (2026) · Journal of Educational Computing Research · doi:10.1177/07356331261467266
“We have decades of evidence about the effects and important features of intelligent tutoring systems (Kulik & Fletcher, 2016; Ma et al., 2014; Steenbergen-Hu & Cooper, 2014; VanLehn, 2011) and educational scaffolding (Belland et al., 2017; Kim et al., 2018) that can help guide the types of…”
Salvo, M. D. (2026) · Digital Education Review · doi:10.1344/der.2026.49.79-94
Włodarski, R. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805783
“Therefore, researchers and educators [16, 17, 18] call for deeper evaluation of genAI-based tutoring.”
College Student Success and Well-Being: Theories and Interventions
Bowman, N. A., & Perez, M.-V. (2026) · Oxford University Press eBooks · doi:10.1093/9780197759424.003.0025 · Cited by 1
FANG, Z., Lin, W., Hong, R., Sutarat, P., & Jin, Y. (2026) · Journal of Education and Learning Reviews · doi:10.60027/jelr.2026.e3244
Dlamini, L. V., Mosia, M. S., & Egara, F. O. (2026) · F1000Research · doi:10.12688/f1000research.187529.1
Guaman, L. M. V., & Guallo, E. T. T. (2026) · Ciencia Interdisciplinaria Internacional · doi:10.70577/cieninter.v4i3.45
Adaptive Generation of Faded Worked Examples in Vocational Education
Soennecken, T., Schütt, A., Petrak, B., & André, E. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21997-8_1
Bokhove, C., Davies, B., Cen, J., Špakula, J., & Gammack, D. (2026) · ZDM · doi:10.1007/s11858-026-01829-w
Exploring the Effects of Generative AI on Learning Using the ISAR Model
Bauer, E., Greiff, S., Graesser, A. C., Scheiter, K., & Sailer, M. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_3
Mesenhöller, J., & Böhme, K. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1875856
Li, P., Yuan, Q., Lian, J., & Lyu, Z. (2026) · Discover Computing · doi:10.1007/s10791-026-10706-9
“Existing studies have shown that generative AI tutors have the characteristics of personalization, interactivity, and real-time feedback [6, 7], which can effectively make up for the lack of personalized guidance in traditional teaching [8, 9] and improve students’ learning outcomes [10, 11].”
Topçubaşı, T., Keleş, O., Demir, S., & İŞÇİ, T. G. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1948951
Dlamini, L. V., Mosia, M. S., & Egara, F. O. (2026) · F1000Research · doi:10.12688/f1000research.187529.2
Akinwumi, P. O., Qian, M., Akande, I., & Babatope, O. A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1933178
Najem, K. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02640-y
“Meta-analytic evidence indicates learning gains broadly comparable to one-to-one human tutoring, with effect sizes between 0.40 and 0.76 depending on domain [27, 39].”
InfluentialANISULOWO, B. O., & SHAIBU, L. (2026) · Federal University Gusau Faculty of Education Journal · doi:10.64348/zije.2026632
“…effects are reported in the literature when focusing on college students, with ITS outperforming all instruction methods other than human tutoring (Steenbergen‐Hu & Cooper, 2013), and with a 50-evaluation meta-analysis reporting a median effect of 0.66 standard deviations (from the 50th…”
Influential
Metadata
- Authors
- Steenbergen-Hu, S., & Cooper, H.
- Venue
- Journal of Educational Psychology
- Published
- 2014-05
- Volume
- 106
- Issue
- 2
- Pages
- 331-347
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2316441559
- Cited by
- 355
- References
- 68
- Citations by year
- 2026: 48 · 2025: 55 · 2024: 50 · 2023: 28 · 2022: 24 · 2021: 31 · 2020: 31 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 15 · 2015: 12 · 2014: 4
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Journal of Educational Psychology (publishedVersion)
- DukeSpace (Duke University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- intelligent tutoring systems, university students, academic learning, higher education, computer-assisted instruction, AutoTutor, ALKIS, effect size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.