The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning.
Open accessAbstract
© 2013 American Psychological Association.This meta-analysis synthesizes research on the effectiveness of intelligent tutoring systems (ITS) for college students. Thirty-five reports were found containing 39 studies assessing the effectiveness of 22 types of ITS in higher education settings. Most frequently studied were AutoTutor, Assessment and Learning in Knowledge Spaces, eXtended Tutor-Expert System, and Web Interface for Statistics Education. Major findings include (a) Overall, ITS had a moderate positive effect on college students' academic learning (g = .32 to g = .37); (b) ITS were less effective than human tutoring, but they outperformed all other instruction methods and learning activities, including traditional classroom instruction, reading printed text or computerized materials, computer-assisted instruction, laboratory or homework assignments, and no-treatment control; (c) ITS's effectiveness did not significantly differ by different ITS, subject domain, or the manner or degree of their involvement in instruction and learning; and (d) effectiveness in earlier studies appeared to be significantly greater than that in more recent studies. In addition, there is some evidence suggesting the importance of teachers and pedagogy in ITS-assisted learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another meta-analysis calculated a point estimate for intelligent tutoring systems in college-level business education of ḡ = 0.16 (Steenbergen-Hu & Cooper, 2014).”
From Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education (2017) · cites this paper as Steenbergen-Hu & Cooper, 2014
In the citing paper
“and that ITS were less effective that human tutoring”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“Steenbergen-Hu and Cooper (2014) found that ITS had a moderate effect of students' learning”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“but ITS outperformed all other instruction methods (such as traditional classroom instruction, reading printed or digital text, or homework assignments).”
From Systematic review of research on artificial intelligence applications in higher education – where are the educators? (2019) · cites this paper as Steenbergen-Hu and Cooper (2014)
In the citing paper
“most of the strategies and functions that our participants desired to improve the usability issues of ChatGPT, such as hints, feedback, prompt suggestions, a learning map and navigation system, and AI-Helper agents, resemble the traits that ITSs have long had and are already proven to be effective.”
From The Promise and Peril of ChatGPT in Higher Education: Opportunities, Challenges, and Design Implications (2024) · cites this paper as [80]
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Duval, S., & Tweedie, R. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00455.x · Cited by 14,099 · cited 1×
“We further conducted Duval and Tweedie’s (2000) trim and fill procedures and found that the adjusted average effect sizes remained the same as the observed ones, which were g .”
The Role of Goal Orientation in Self-Regulated Learning
Pintrich, P. R. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50043-3 · Cited by 5,173 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853 · cited 1×
“Graesser et al.’s (2011) review of ITS posed a number of questions that need answering.”
A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students
Pintrich, P. R. (2004) · Educational Psychology Review · doi:10.1007/s10648-004-0006-x · Cited by 3,269 · cited 1×
“The last two questions are particularly important because research has found that high and low selfregulating learners usually exhibit different learning characteristics (Pintrich, 2000, 2004).”
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 7×
“Bloom (1984) reported that expert human tutors can help students achieve learning gains as large as two sigmas. Although not as effective as what Bloom (1984) found, a recent meta-review by VanLehn (2011) found that human tutoring had a positive impact of d .79 on students’ learning. Many ITS are designed to follow th…”
InfluentialVanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323 · cited 4×
“VanLehn (2011) reviewed randomized experiments that compared the effectiveness of human tutoring, computer tutoring, and no tutoring.”
InfluentialCognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · cited 1×
“For example, Cognitive Tutors are built on a cognitive theory called adaptive control of thought (ACT, or ACT-R in its updated form; Anderson et al., 1995).”
Sample criteria for testing outlying observations
Grubbs, F. E. (1950) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729885 · Cited by 1,631 · cited 4×
“Grubbs (1950) tests detected no effect-size outliers.”
InfluentialBuilding Intelligent Interactive Tutors : student-centered strategies for revolutionizing e-learning
Woolf, B. P. (2009) · HAL · Cited by 1,376
Intelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011) · Review of Educational Research · doi:10.3102/0034654310393361 · Cited by 1,029
Aleven, V. A. W. M. M., & Koedinger, K. R. (2002) · Cognitive Science · doi:10.1207/s15516709cog2602_1 · Cited by 895 · cited 1×
“AutoTutor was built by incorporating explanation-based constructivist theories of learning into the practices of human tutoring, which usually involve collaborative constructive activities (Aleven & Koedinger, 2002; Chi, de Leeuw, Chiu, & LaVancher, 1994).”
Eliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
The Behavior of Tutoring Systems
VanLehn, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 749 · cited 2×
“In this regard, CAI, CBT, and Web-based homework are different from ITS in that they lack an inner loop (VanLehn, 2006).”
The handbook of research synthesis and meta-analysis, 2nd ed.
Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009) · Cited by 541
Self-citationThompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 538 · cited 1×
“Instead, we adopted a meta-analytic thinking approach for interpreting the results (cf. Cumming & Finch, 2001; Thompson, 2002, 2006).”
Cumming, G., & Finch, S. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614002 · Cited by 528 · cited 1×
“Instead, we adopted a meta-analytic thinking approach for interpreting the results (cf. Cumming & Finch, 2001; Thompson, 2002, 2006).”
Doignon, J., & Falmagne, J. (1998) · Cited by 451
Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor
Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(2)02 · Cited by 406
AutoTutor: A simulation of a human tutor
Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., & Kreuz, R. (1999) · Cognitive Systems Research · doi:10.1016/s1389-0417(99)00005-4 · Cited by 387
Learning from performance errors.
Ohlsson, S. (1996) · Psychological Review · doi:10.1037//0033-295x.103.2.241 · Cited by 387 · cited 1×
“Constraint-based tutors are based on the constraint-based model (CBM), derived from Ohlsson’s theory of learning from performance errors (Ohlsson, 1992, 1996).”
When Are Tutorial Dialogues More Effective Than Reading?
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rosé, C. P. (2007) · Cognitive Science · doi:10.1080/03640210709336984 · Cited by 378 · cited 5×
“Of the 35 reports, 33 each contained one independent study, one report (VanLehn et al., 2007) contained four studies, and another report (Koch & Gobell, 1999) contained two studies.”
InfluentialHow features of educational technology applications affect student reading outcomes: A meta-analysis
Cheung, A. C. K., & Slavin, R. E. (2012) · Educational Research Review · doi:10.1016/j.edurev.2012.05.002 · Cited by 307 · cited 4×
“There is evidence that the differences mentioned above have an impact on the magnitude of effect sizes (Cheung & Slavin, 2012; Sosa et al., 2011).”
InfluentialiSTART: Interactive strategy training for active reading and thinking
McNamara, D. S., Levinstein, I. B., & Boonthum, C. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195567 · Cited by 284
A Large‐Scale Evaluation of an Intelligent Discovery World: Smithtown
Shute, V. J., & Glaser, R. (1990) · Interactive Learning Environments · doi:10.1080/1049482900010104 · Cited by 279
The role of method in treatment effectiveness research: Evidence from meta-analysis.
Wilson, D. B., & Lipsey, M. W. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.413 · Cited by 261 · cited 1×
“In cases for which only inference results were reported but no means and standard deviations were available, g was estimated from the inferential statistics, such as t, F, or p-values (Wilson & Lipsey, 2001).”
Cognitive Computer Tutors: Solving the Two-Sigma Problem
Corbett, A. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44566-8_14 · Cited by 239 · cited 2×
“…Berger, Healy, & Romero, 2003; Assessment and Learning in Knowledge Spaces [ALEKS]: Doignon & Falmagne, 1999), physics (e.g., Andes, Atlas, and Why/Atlas: VanLehn et al., 2002, 2007), and computer science (e.g., dialogue-based ITS: Lane & VanLehn, 2005; ACT Programming Tutor: Corbett, 2001).”
The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing
VanLehn, K., Jordan, P. W., Rosé, C. P., Bhembe, D., Böttner, M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringenberg, M., Roque, A., et al. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-47987-2_20 · Cited by 230 · cited 2×
“…Berger, Healy, & Romero, 2003; Assessment and Learning in Knowledge Spaces [ALEKS]: Doignon & Falmagne, 1999), physics (e.g., Andes, Atlas, and Why/Atlas: VanLehn et al., 2002, 2007), and computer science (e.g., dialogue-based ITS: Lane & VanLehn, 2005; ACT Programming Tutor: Corbett, 2001).”
Constraint-Based Student Modeling
Ohlsson, S. (1994) · Student Modelling: The Key to Individualized Knowledge-Based Instruction · doi:10.1007/978-3-662-03037-0_7 · Cited by 227 · cited 1×
“Constraint-based tutors are based on the constraint-based model (CBM), derived from Ohlsson’s theory of learning from performance errors (Ohlsson, 1992, 1996).”
Evaluation of a Constraint-Based Tutor for a Database Language
Mitrović, A., & Ohlsson, S. (1999) · UC Research Repository (University of Canterbury) · Cited by 218
“Intelligent tutoring systems,”
Graesser, A. C., Hu, X., & Sottilare, R. (2018) · International Handbook of the Learning Sciences · doi:10.4324/9781315617572-24 · Cited by 182
Steenbergen-Hu, S., & Cooper, H. (2013) · Journal of Educational Psychology · doi:10.1037/a0032447 · Cited by 175 · cited 3×
“Steenbergen-Hu and Cooper (2013) generated the overall effect sizes from 31 studies that compared ITS with regular classroom instruction.”
Self-citationTeaching scientific thinking skills: Students and computers coaching each other
Reif, F., & Scott, L. A. (1999) · American Journal of Physics · doi:10.1119/1.19130 · Cited by 152 · cited 1×
“…point out that the results were not statistically significant (see Table 2) and that there were only three studies that compared ITS-assisted learning with human tutoring (i.e., Reif & Scott, 1999; Stankov, Glavinić, & Grubišić, 2004; VanLehn et al., 2007, Study 1) in the current meta-analysis.”
Fifteen years of constraint-based tutors: what we have achieved and where we are going
Mitrović, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9105-9 · Cited by 151 · cited 1×
“The core idea of CBM is to deliver the domain knowledge as a set of constraints, which can be used to analyze students’ solutions and provide feedback on errors (Mitrovic, 2012).”
Dynarski, M., Agodini, R., Heaviside, S., Carey, N., Campuzano, L., Means, B., Murphy, R. F., Penuel, W. R., Javitz, H., Emery, D., et al. (2007) · National Center for Education Evaluation and Regional Assistance · Cited by 149
Effectiveness of Computer-Assisted Instruction in Statistics: A Meta-Analysis
Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. (2010) · Review of Educational Research · doi:10.3102/0034654310378174 · Cited by 129
Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009) · National Center for Education Evaluation and Regional Assistance · Cited by 125
Teaching the tacit knowledge of programming to noviceswith natural language tutoring
Lane, H. C., & VanLehn, K. A. (2005) · Computer Science Education · doi:10.1080/08993400500224286 · Cited by 101 · cited 2×
“…Berger, Healy, & Romero, 2003; Assessment and Learning in Knowledge Spaces [ALEKS]: Doignon & Falmagne, 1999), physics (e.g., Andes, Atlas, and Why/Atlas: VanLehn et al., 2002, 2007), and computer science (e.g., dialogue-based ITS: Lane & VanLehn, 2005; ACT Programming Tutor: Corbett, 2001).”
Evaluation of animalwatch: An intelligent tutoring system for arithmetic and fractions
Beal, C. R., Arroyo, I., Cohen, P. R., & Woolf, B. P. (2010) · Cited by 96
Johnson, B. G., Phillips, F., & Chase, L. G. (2009) · Journal of Accounting Education · doi:10.1016/j.jaccedu.2009.05.001 · Cited by 82
Meta-Cognitive Strategy Instruction in Intelligent Tutoring Systems: How, When, and Why
Chi, M., & VanLehn, K. A. (2010) · Educational Technology & Society · Cited by 71
Why/AutoTutor: A Test of Learning Gains from a Physics Tutor with Natural Language Dialog
Graesser, A., Jackson, G., Mathews, E., Mitchell, H., Olney, A. M., Ventura, M., Chipman, P., Franceschetti, D. R., Hu, X., Louwerse, M. M., et al. (2003) · Tilburg University Research Portal · Cited by 69
Evaluation of an Interactive Tutorial for Teaching the Central Limit Theorem
Aberson, C. L., Berger, D. E., Healy, M. R., Kyle, D. J., & Romero, V. L. (2000) · Teaching of Psychology · doi:10.1207/s15328023top2704_08 · Cited by 68
Research Synthesis: Effect Sizes
Thompson, B. (2006) · doi:10.4324/9780203874769.ch35 · Cited by 58 · cited 1×
“Instead, we adopted a meta-analytic thinking approach for interpreting the results (cf. Cumming & Finch, 2001; Thompson, 2002, 2006).”
Multiple Knowledge Sources in Intelligent Teaching Systems
Woolf, B. P., & Cunningham, P. A. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307063 · Cited by 52 · cited 1×
“For example, Woolf and Cunningham (1987) reported that each hour of ITS instruction costs 200 hours of work to build.”
Using the Web-Based Interactive Software ALEKS to Enhance College Algebra
Hagerty, G., & Smith, S. (2005) · Mathematics and computer education · Cited by 51
Evaluation of an Interactive Tutorial for Teaching Hypothesis Testing Concepts
Aberson, C. L., Berger, D. E., Healy, M. R., & Romero, V. L. (2003) · Teaching of Psychology · doi:10.1207/s15328023top3001_12 · Cited by 51
Research Methods Tutor: Evaluation of a dialogue-based tutoring system in the classroom
Arnott, E., Hastings, P., & Allbritton, D. W. (2008) · Behavior Research Methods · doi:10.3758/brm.40.3.694 · Cited by 43
An Interactive Tutorial for Teaching Statistical Power
Aberson, C. L., Berger, D. E., Healy, M. R., & Romero, V. L. (2002) · Journal of Statistics Education · doi:10.1080/10691898.2002.11910682 · Cited by 42
Name, N., & Johnson, B. G. (2011) · Issues in Accounting Education · doi:10.2308/iace.2011.26.1.87 · Cited by 40 · cited 1×
“They were Transaction Analysis and Recording Tutor (Johnson, Phillips, & Chase; 2009; Phillips & Johnson, 2011) and Design-Statistics Finder (Koch & Gobell, 1999, containing two studies).”
Web-Based Tutorials for Teaching Introductory Statistics
Bliwise, N. G. (2005) · Journal of Educational Computing Research · doi:10.2190/0d1j-1ce1-5uxy-3v34 · Cited by 37
The design and evaluation of Link: A computer‐based learning system for correlation
Morris, E. (2001) · British Journal of Educational Technology · doi:10.1111/1467-8535.00175 · Cited by 32
Further Results from the Evaluation of an Intelligent Computer Tutor to Coach Self-Explanation
Conati, C., & VanLehn, K. A. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45108-0_34 · Cited by 31 · cited 2×
“There are also ITS for specific skills, such as metacognitive skills (Aleven, McLaren, Roll, & Koedinger, 2006; Conati & VanLehn, 2000).”
A web-based tutoring system with styles-matching strategy for spatial geometric transformation
Wang, H., Li, T., & Chang, C. (2005) · Interacting with Computers · doi:10.1016/j.intcom.2005.11.002 · Cited by 29
Tamim, R. M., Lowerison, G., Schmid, R. F., Bernard, R. M., & Abrami, P. C. (2011) · Journal of Computing in Higher Education · doi:10.1007/s12528-010-9041-4 · Cited by 26
A hypertext-based tutorial with links to the Web for teaching statistics and research methods
Koch, C., & Gobell, J. L. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207686 · Cited by 23 · cited 2×
“They were Transaction Analysis and Recording Tutor (Johnson, Phillips, & Chase; 2009; Phillips & Johnson, 2011) and Design-Statistics Finder (Koch & Gobell, 1999, containing two studies).”
The Andes Physics Tutoring System: An Experiment in Freedom
VanLehn, K. A., van de Sande, B., Shelby, R., & Gershman, S. (2010) · Studies in computational intelligence · doi:10.1007/978-3-642-14363-2_21 · Cited by 22
Xu, Y. J., Meyer, K. A., & Morgan, D. D. (2009) · Journal of Statistics Education · doi:10.1080/10691898.2009.11889524 · Cited by 21
Controlled experiment replication in evaluation of e-learning system’s educational influence
Grubišić, A., Stankov, S., Rosić, M., & Žitko, B. (2009) · Computers & Education · doi:10.1016/j.compedu.2009.03.014 · Cited by 21
A Cognitive Technology to Teach Composition Skills: Four Studies with the R-Wise Writing Tutor
Rowley, K., Carlson, P. A., & Miller, T. M. (1998) · Journal of Educational Computing Research · doi:10.2190/kw4v-fjkd-l7j1-efk0 · Cited by 20
Web_soc: a socratic-dialectic-based collaborative tutoring system on the world wide web
Chang, K.-E., Sung, Y., Wang, K.-Y., & Dai, C.-Y. (2003) · IEEE Transactions on Education · doi:10.1109/te.2002.808276 · Cited by 13
Conati, C., Müldner, K., & Carenini, G. (2006) · Cited by 12
Stankov, S., Glavinić, V., & Grubišić, A. (2017) · doi:10.13140/rg.2.2.29526.96322 · Cited by 10
Mackh, S. J. (2003) · Cited by 9
A Study of the Effectiveness of a Multimedia Intelligent Tutoring System
Livergood, N. D. (1994) · Journal of Educational Technology Systems · doi:10.2190/dthe-9hmv-qgr9-d38m · Cited by 7
Callahan, J., Guarino, J., Chyung, S. Y., Gardner, J., Moll, A., Pyke, P., & Schrader, C. B. (2007) · ASEE Annual Conference & Exposition · Cited by 5
Revitalizing Algebra: the effect of the use of a cognitive tutor in a remedial course
Stylianou, D. A., & Shapiro, L. J. (2002) · Journal of Educational Television · doi:10.1080/1358165020270306 · Cited by 4
A DSM-III-R Computer Tutorial for Abnormal Psychology
Hayden, D. C. (1990) · Teaching of Psychology · doi:10.1207/s15328023top1703_21 · Cited by 4
Cited by
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
“• RQ3: What are the potential issues associated with ChatGPT, and how can they be addressed?”
BackgroundVision, challenges, roles and research issues of Artificial Intelligence in Education
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2020.100001 · Cited by 1,435
Variables associated with achievement in higher education: A systematic review of meta-analyses.
Schneider, M., & Preckel, F. (2017) · Psychological Bulletin · doi:10.1037/bul0000098 · Cited by 1,083
Artificial Intelligence trends in education: a narrative overview
Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.08.233 · Cited by 956
Metadata
- Authors
- Steenbergen-Hu, S., & Cooper, H.
- Venue
- Journal of Educational Psychology
- Published
- 2014-05
- Volume
- 106
- Issue
- 2
- Pages
- 331-347
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2316441559
- Cited by
- 355
- References
- 68
- Citations by year
- 2026: 48 · 2025: 55 · 2024: 50 · 2023: 28 · 2022: 24 · 2021: 31 · 2020: 31 · 2019: 22 · 2018: 22 · 2017: 13 · 2016: 15 · 2015: 12 · 2014: 4
- FWCI
- 1.50
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Journal of Educational Psychology (publishedVersion)
- DukeSpace (Duke University) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- intelligent tutoring systems, university students, academic learning, higher education, computer-assisted instruction, AutoTutor, ALKIS, effect size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.