The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index.
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Abstract
This review describes a meta-analysis of findings from 50 controlled evaluations of intelligent computer tutoring systems. The median effect of intelligent tutoring in the 50 evaluations was to raise test scores 0.66 standard deviations over conventional levels, or from the 50th to the 75th percentile. However, the amount of improvement found in an evaluation depended to a great extent on whether improvement was measured on locally developed or standardized tests, suggesting that alignment of test and instructional objectives is a critical determinant of evaluation results. The review also describes findings from two groups of evaluations that did not meet all of the selection requirements for the meta-analysis: six evaluations with nonconventional control groups and four with flawed implementations of intelligent tutoring systems. Intelligent tutoring effects in these evaluations were small, suggesting that evaluation results are also affected by the nature of control treatments and the adequacy of program implementations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“With an effect size of g = 0.7, ChatGPT would thus be more impactful than ITS (see, e.g., ES = 0.66 in Kulik and Fletcher 2016)”
From ChatGPT in Education: An Effect in Search of a Cause (2025) · cites this paper as Kulik and Fletcher 2016
In the citing paper
“In Kulik and Fletcher (2016), ITS, with their integrated instructional features, yielded an effect size of 0.66 standard deviations in learning outcomes compared to conventional teaching.”
From ChatGPT in Education: An Effect in Search of a Cause (2025) · cites this paper as Kulik and Fletcher (2016)
In the citing paper
“For example, Carnegie Learning's MATHia platform achieved up to 36% higher knowledge retention than traditional instruction, which constitutes strong evidence for the pedagogical potential of AI in controlled settings (Kulik and Fletcher, 2016).”
From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Kulik and Fletcher, 2016
In the citing paper
“For adaptive learning systems, the evidence base was generally strong, as multiple independent studies validated vendor claims of improved learning outcomes (Kulik and Fletcher, 2016).”
From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Kulik and Fletcher, 2016
Citations
Cites
James, W. (1913) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,226
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Statistical Methods for Meta-Analysis.
Freeman, P. R., Hedges, L. V., & Olkin, I. (1986) · Biometrics · doi:10.2307/2531069 · Cited by 4,784
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985) · Science · doi:10.1126/science.228.4698.456 · Cited by 3,119
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 10×
“, Carbonell, 1970; Fletcher, 1985; Sleeman & Brown, 1982). Grounded in artificial intelligence concepts and cognitive theory, these newer systems guided learners through each step of a problem solution by creating hints and feedback as needed from expert-knowledge databases. The first-generation computer tutors have b…”
BackgroundInfluentialVanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323
Cognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · cited 1×
“He attributed the belief that ITSs produce 1–standard deviation gains to a widely cited article (Anderson, Corbett, Koedinger, & Pelletier, 1995) that summarized findings from several influential studies.”
BackgroundProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Research works printed but not linked: 19
- Anania J. (1981). The effects of quality of instruction on the cognitive and affective learning of students (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (Order No. T-28171)
- *Atkinson R. K., 2007, An experimental evaluation of three computer-based reading comprehension tutors
- Beal C. R., 2010, Journal of Interactive Online Learning, 9, 64
- Beal C. R., 2007, Journal of Interactive Online Learning, 6, 43
- Biesinger K., 2008, Journal of Computers in Mathematics and Science Teaching, 27, 5
- Burke A. J. (1980). Students’ potential for learning contrasted under tutorial and group approaches to instruction. (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (Order No. T-28810)
- *Chang K.-E., 1999, Journal of Computers in Mathematics and Science Teaching, 18, 303
- Cohen J., 1988, Statistical power analysis for the social sciences
- Craig S. D., 2004, Journal of Educational Multimedia and Hypermedia, 13, 163
- Fletcher J. D., 1982, Psychology and society: Information technology in the 1980s, 166
- *Fletcher J. D., 2012, DARPA digital Tutor: Assessment data
- Hartley S. S. (1977). Meta-analysis of the effects of individually paced instruction in mathematics (Doctoral dissertation). Available from ProQuest Dissertations & Theses database. (Order No. 7729926)
- *Hategekimana C., 2008, Proceedings of Society for Information Technology & Teacher Education International Conference 2008, 414
- *Jeremic Z., 2009, Educational Technology & Society, 12, 111
- Koedinger K. R., 2002, Proceedings of PMENA XXII (North American Chapter of the International Group for the Psychology of Mathematics Education, 1, 21
- *Le N. T., Menzel W., Pinkwart N. (2009). Evaluation of a constraint-based homework assistance system for logic programming. In Leung H., Li R., Lau R., Li Q. (Eds.), Proceedings of the 6th International Conference on Web-based Learning, Edinburgh, UK, 2007 (pp. 367–379). Berlin, Germany: Springer. Retrieved from http://www.icce2009.ied.edu.hk/pdf/C1/proceedings051-058.pdf
- *Ritter S., 2007, Supporting learning flow through integrative technologies, 13
- *Stankov S., 2004, Eighth IEEE International Conference on Intelligent Engineering Systems, 545
- *Steuck K., 1997, Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL
Cited by
Social robots for education: a review.
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., & Tanaka, F. (2018) · Science Robotics · doi:10.1126/scirobotics.aat5954 · Cited by 1,558
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
Artificial Intelligence in Education: A Systematic Literature Review
Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124167 · Cited by 977
Explainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Interacting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841
“Teaching agents play the role of human teachers and can present instructions, illustrate examples, ask questions (Wambs-ganss et al., 2020), and provide immediate feedback (Kulik & Fletcher, 2016).”
Background
Metadata
- Authors
- Kulik, J. A., & Fletcher, J. D.
- Affiliations
- University of Michigan (US)
- Institute For Defense Analyses (US)
- ORCID
- Fletcher, J. D.
- Venue
- Review of Educational Research
- Published
- 2016-03
- Volume
- 86
- Issue
- 1
- Pages
- 42-78
- ISSN
- 0034-6543 (ISSN-L)
- 1935-1046
- OpenAlex
- W1985824294
- Cited by
- 930
- References
- 100
- Citations by year
- 2026: 259 · 2025: 170 · 2024: 119 · 2023: 45 · 2022: 62 · 2021: 74 · 2020: 56 · 2019: 46 · 2018: 59 · 2017: 24 · 2016: 13
- FWCI
- 31.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Review of Educational Research (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- intelligent tutoring systems, computer-assisted instruction, educational effectiveness, test score improvement, control group design, program implementation quality, educational technology evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. https://app.cochat.ai/papers/10.3102/0034654315581420
- The paper
- Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research. https://doi.org/10.3102/0034654315581420
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.