Intelligent tutoring systems for programming education: a systematic review
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
A variety of intelligent tutoring systems have been created for the purpose of teaching computer programming. Most published literature focuses on systems that have been developed to teach programming within tertiary courses. A majority of systems have been developed to teach introductory programming concepts; other systems tutor more specific aspects of programming like scope or recursion. Literature reports that these systems address many of the difficulties associated with teaching programming to novices; however, individual systems vary greatly, and there is a large range of supplementary features developed in these systems. Most intelligent programming tutors involve some form of interactive programming exercises, but the use of supplementary features like plans, quizzes and worked solutions vary greatly between different systems. This systematic review reports key information about existing systems and the prevalence of different features within them. An overview of how supplementary features are integrated into these systems is given, along with implications for how intelligent programming tutors could be improved by supporting a wider range of supplementary features.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Intelligent feedback and hints can be embedded into programming tasks, helping with specific semantic or syntactic issues (Crow et al., 2018)”
From A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour (2024) · cites this paper as Crow et al., 2018
Citations
Cites
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“Backward snowballing as outlined in ’Guidelines for snowballing in systematic literature studies and a replication in software engineer-ing’ [28] was carried out for papers cited within the literature that met the inclusion criteria; this was done to ensure that research and IPTs that were…”
MethodResnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
BLOOM, B. S. (1984) · Educational Researcher · doi:10.3102/0013189x013006004 · Cited by 2,734 · cited 1×
“3160492 has been well established with experiments involving human tutors [3].”
BackgroundThe Scratch Programming Language and Environment
Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,410
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Kulik, J. A., & Fletcher, J. D. (2016) · Review of Educational Research · doi:10.3102/0034654315581420 · Cited by 930 · cited 1×
“Experiments within the field of intelligent tutoring have shown similar effect size benefits to human tutoring [11].”
BackgroundELM-ART: An Adaptive Versatile System for Web-Based Instruction
Weber, G., & Brusilovsky, P. L. (2001) · Cited by 542
PROUST: Knowledge-Based Program Understanding
Johnson, W. L., & Soloway, E. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232210 · Cited by 284
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258
The automated tutoring of introductory computer programming
Anderson, J. R., & Skwarecki, E. (1986) · Communications of the ACM · doi:10.1145/6592.6593 · Cited by 225
Hooshyar, D., Ahmad, R. B., Yousefi, M., Yusop, F. D., & Horng, S. ‐. (2015) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12099 · Cited by 96
Research works printed but not linked: 1
- Omar AlZoubi , Davide Fossati , Barbara Di Eugenio , and Nick Green , [n. d.]. ChiQat-Tutor : An integrated environment for learning recursion . In Proc. of the Second Workshop on AI-supported Education for Computer Science (AIEDCS)(at ITS 2014 ). Honolulu, HI ( 2014 ). Omar AlZoubi, Davide Fossati, Barbara Di Eugenio, and Nick Green, [n. d.]. ChiQat-Tutor: An integrated environment for learning recursion. In Proc. of the Second Workshop on AI-supported Education for Computer Science (AIEDCS)(at ITS 2014). Honolulu, HI (2014).
Cited by
Items 101–185 of 185
Design and Evaluation of a Programming Tutor Based on an Instant Messaging Interface
Álvarez, C., Rojas, L. A., & Valenzuela, J. D. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05064-0_1 · Cited by 4
Skalka, J. (2022) · Springer eBooks · doi:10.1007/978-3-031-13359-6_1 · Cited by 4
When is Automated Feedback a Barrier to Timely Feedback?
DeOrio, A., & Keefer, C. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--40405 · Cited by 6
Liang, Y., Wu, W., Wu, L., & Wang, M. (2019) · Communications in computer and information science · doi:10.1007/978-981-15-1899-7_4 · Cited by 4
RustViz: Interactively Visualizing Ownership and BorrowingPreprint
Gongming, -, L., Reddy, V., Almeida, M., Zhu, Y., Du, K., & Omar, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09012 · Cited by 4
Topical Analysis of Programming Education Research Literature
Malmi, L., López‐Pernas, S., Nguyen, H., Apiola, M., & Saqr, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3767734 · Cited by 5
An automatic feedback model for learning programming via block‐based programming platforms
Çakıroğlu, Ü., & Mumcu, S. (2023) · Computer Applications in Engineering Education · doi:10.1002/cae.22652 · Cited by 3
SCRIPT: Implementing an Intelligent Tutoring System for Programming in a German University Context
Deriyeva, A., Dannath, J., & Paaßen, B. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99261-2_16 · Cited by 3
Augmenting the teacher’s perspective on programming student's performance via permanent monitoring
Fonseca, N. G., Macedo, L., Marcelino, M. J., & Mendes, A. J. (2018) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8658924 · Cited by 3
Garcia, M. B., & Revano, T. F. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.4060572 · Cited by 3
Lohr, D., Kiesler, N., Keuning, H., & Jeuring, J. T. (2025) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3736181.3747150 · Cited by 2
Reorienting OOP Curriculum: From Knowledge-Centric Projects to Agentic AI-Driven Problem Solving
Wu, H., Yuan, J., & Deng, H. (2025) · International Conference on AI-enabled Education · doi:10.1145/3768421.3768503 · Cited by 3
Design Implications for Student and Educator Needs in AI-Supported Programming Learning Tools
Ma, B., Xie, Y., Li, H., Li, G., Chen, L. Y. E., Shimada, A., & Konomi, S. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2730083 · Cited by 3
Board 144: PRIME: Engaging STEM Undergraduates in Computer Science with Intelligent Tutoring Systems
Lester, J. C., Boyer, K. E., Wiebe, E., Mott, B., & Smith, A. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--32259 · Cited by 2
Koike, K., Mogi, T., Tomoto, T., Horiguchi, T., & Hirashima, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60152-2_12 · Cited by 2
Process-Oriented Understanding Estimation Using Code Puzzles
Ito, H., Shimakawa, H., & Harada, F. (2022) · Creative Education · doi:10.4236/ce.2022.133048 · Cited by 2
Estimation of Programming Understanding by Time Series Analysis of Code Puzzles
Ito, H., Shimakawa, H., & Harada, F. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-2394-4_44 · Cited by 2
Huu, P. N., Tangworakitthaworn, P., & Gilbert, L. H. (2022) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse54890.2022.9836290 · Cited by 2
Matsumoto, S., Ohno, A., & Shigematsu, H. (2022) · Information Engineering Express · doi:10.52731/iee.v8.i2.623 · Cited by 2
Agile Teaching: Automated Student Support and Feedback Generation
Bahrehvar, M., & Moshirpour, M. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343212 · Cited by 3
Students’ perceptions on different sources of self-feedback
Grion, V., Raffaghelli, J. E., Doria, B., & Serbati, A. (2024) · Educational Research and Evaluation · doi:10.1080/13803611.2024.2347349 · Cited by 5
Automatic Generation of Matching Rules for Programming Exercise Assessment
Wong, T.-L., Poon, C. K., Tang, C. M., Yu, Y. T., & Lee, V. C. S. (2020) · Communications in computer and information science · doi:10.1007/978-981-33-4594-2_11 · Cited by 2
Advanced Comprehension Analysis Using Code Puzzle: Considering the Programming Thinking Ability
Ito, H., Shimakawa, H., & Harada, F. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-71846-6_3 · Cited by 3
Tecson, C. A., & Rodrigo, M. M. T. (2021) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet51873.2021.9419633 · Cited by 2
Asano, Y., Dutta, M., Thakur, T., Solyst, J., Cristea, S., Jovic, H., Petersen, A., & Williams, J. J. (2021) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3430895.3460145 · Cited by 2
HProgramming: a Helping Tool for Teach and Learn Programming
Figueiredo, J. A. Q., & García‐Peñalvo, F. J. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-07018-1_6 · Cited by 2
Programming courses Teaching methods Before, During, and After COVID-19 Pandemic
Moosa, J., & Bahaaudeen, A. (2023) · International Conference on IT Innovation and Knowledge Discovery (ITIKD) · doi:10.1109/itikd56332.2023.10099736 · Cited by 2
Analysis of the Effectiveness of Feedback Provision in Intelligent Tutoring Systems
Jamaludin, N. H., & Romli, R. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9592-9_13 · Cited by 3
Alkhuwaylidee, A. R., Gatea, A. K., & Alomari, M. F. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-75091-5_5 · Cited by 1
Singla, A. K., Hobert, S., Joachim, S., Hennecke, M., Voelker, A., Thaler, A. M., Summerer, M. T., Schmid, U., Bayerkuhnlein, M., Wolter, D., et al. (2025) · Schriften aus der Fakultät Wirtschaftsinformatik und Angewandte Informatik der Otto-Friedrich-Universität Bamberg. · doi:10.20378/irb-107661 · Cited by 1
Park, S., & Kim, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98284-2_11 · Cited by 1
Personalized Learning of Programming Fundamentals Through Robotic Simulations
Leis, A., Mallo, A., Guerreiro-Santalla, S., Paz-López, A., & Bellas, F. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-93409-4_13 · Cited by 1
Learning Feedback Policy from Historical Data: An Offline Approach Within Pyrates
Kirouchenassamy, B., Yessad, A., Jolivet, S., Branthôme, M., Lallé, S., & Luengo, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98414-3_12 · Cited by 1
Feedback Systems on Computer Programming Courses: A Systematic Literature Review
Castellanos, H., Vieira, C., & Magana, A. J. (2026) · IEEE Access · doi:10.1109/access.2026.3665047 · Cited by 1
Grume, J. C., Miranda, J. P. P., Leon, A. P. D., Salenga, J. L., Hernandez, H. E., Castro, M. A., Maniago, V. G. M., Canlas, J. D., & Quiambao, J. B. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6546-6.ch010 · Cited by 1
Malhotra, V. M., & Anand, A. (2019) · Australasian Computing Education Conference · doi:10.1145/3286960.3286961 · Cited by 1
Intelligent Technologies in E-learning and Intelligent Tutoring Systems
Ivanova, T. (2021) · International Conference on Information Technologies (InfoTech) · doi:10.1109/infotech52438.2021.9548438 · Cited by 1
Estimating Learner’s Perspective in Programming: Analysis of Operation Time Series in Code Puzzles
Ito, H., Shimakawa, H., & Harada, F. (2021) · International Conference on Intelligent Systems and Knowledge Engineering (ISKE) · doi:10.1109/iske54062.2021.9755407 · Cited by 1
Teaching Programming Amid the Digital Churn: Guidelines and Curriculum for a Laboratory
Tezza, D., & Abbott, B. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05657-4_35 · Cited by 1
A Tool to Teach Expressions with Feedback About Broken Laws
Александрович, С. О., Penskoy, N., & Terekhov, G. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478432.3499082 · Cited by 1
Ontology Mapping for Personalization in Adaptive E-learning
Ivanova, T. (2022) · International Conference on Information Technologies (InfoTech) · doi:10.1109/infotech55606.2022.9897096 · Cited by 1
Python OCTS: Design, Implementation, and Evaluation of an Online Coding Tutorial System Prototype
Alasmari, O. A., Singer, J., & Ada, M. B. (2024) · IEEE World Engineering Education Conference (EDUNINE) · doi:10.1109/edunine60625.2024.10500548 · Cited by 3
Auglets: Intelligent Tutors for Learning Good Coding Practices by Solving Refactoring Problems
Kumar, A. N. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690119 · Cited by 1
Day, M., & Gonzalez-Sanchez, J. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3293851 · Cited by 1
Esteves, A. M. D. S., Filho, A. H., & Raabe, A. L. A. (2019) · Anais ... Workshops do Congresso Brasileiro de Informática na Educação · doi:10.5753/cbie.wcbie.2019.1442 · Cited by 1
Competitive Examinations in India: e-Assessment Based Case Study
Kaur, P., Kumar, H., & Kaushal, S. (2019) · IEEE International Conference on Technology for Education (T4E) · doi:10.1109/t4e.2019.00-30 · Cited by 1
Developing Eye Tracking Methods for Detecting Source Code Comprehension Strategies
Deitelhoff, F. (2020) · DuEPublico (University of Duisburg-Essen) · doi:10.17185/duepublico/73588 · Cited by 3
Improving Learning Outcomes Using an Intelligent Tutoring SystemPreprint
Vijayaraghavan, K., Ebrahimi, S., & Ferdoues, M. S. (2018) · viXra
Intelligent Tutoring System Design for Computer Network Learning
Budianto, A., Yuana, R. A., & Puspitaningrum, A. A. (2019) · International Conference on Vocational Education and Training (ICOVET) · doi:10.2991/icovet-18.2019.20
Conceptual Framework of an Intelligent Tutor for Teaching English Grammar to High School Students
Kaur, P., Kumar, H., & Kaushal, S. (2019) · International Conference on Education Technology and Computer · doi:10.1145/3369255.3369309
Huang, L., Poitras, E., Siegel, A. A., Sampangi, R. V., & Paulovich, F. V. (2021) · Advances in analytics for learning and teaching · doi:10.1007/978-3-030-81222-5_4
General Characteristics of the Assisted E-Learning System in Computer Sciences
Pană, M., & Zamfiroiu, A. (2023) · Smart innovation, systems and technologies · doi:10.1007/978-981-19-6755-9_10
Improving Effectiveness of Programming Assignments with Real-time Formative Feedback
Baimetov, I. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594145
Shigematsu, H., & Matsumoto, S. (2023) · IEEJ Transactions on Electronics Information and Systems · doi:10.1541/ieejeiss.143.701
A Systematic Mapping of Affective Intelligent Tutor Systems in Engineering Education
Figueira, T., Vieira, A. V. M., & Netto, J. F. D. M. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342953
Samonte, M. J. C., Evangelista, R. C. B., & Marcelo, D. N. B. (2024) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1145/3675812.3675854
Alasmari, O. A., Singer, J., & Ada, M. B. (2024) · International Conference on Education Technology and Computer · doi:10.1145/3702163.3702418
Empowering Students with Varying Self-Efficacy by Intelligent Hints to Enhance Programming Education
Lin, W.-C., Lin, C.-Z., Chang, C., & Chen, G. (2024) · International Conference on Education and E-Learning (ICEEL) · doi:10.1145/3719487.3719503
Voelker, A., Thaler, A. M., Summerer, M. T., & Schmid, U. (2025) · Schriften aus der Fakultät Wirtschaftsinformatik und Angewandte Informatik der Otto-Friedrich-Universität Bamberg. · doi:10.20378/irb-108886
AI-Driven Coding: Transforming Education and Industry
Azeroual, O. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5097-4.ch015
Pratiche di assessment e feedback supportate dalla tecnologia
Picasso, F., Serbati, A., & Venuti, P. (2024) · Research Trends in Humanities Education & Philosophy · doi:10.6093/2284-0184/10667
Teaching and Assessing Programming Language Tracing
Nelson, G. L. (2021) · ResearchWorks at the University of Washington (University of Washington) · Cited by 3
Lundberg, L. (2024) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Rantanen, V.-M. (2023) · Aaltodoc (Aalto University)
Li, P., Zhu, W., & Ou, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6026015
Lei, P. I. S., Mendes, A. J., & Lam, C. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346747
A Cognitive AI-Based Personalized Learning Environment for Data Structures and Algorithms
Chinnasamy, P., Gopal, M. V., Sai, M. V. N. A., Sharma, M., & GopiKrishna, K. (2026) · International Conference on Mobile Computing and Sustainable Informatics (ICMCSI) · doi:10.1109/icmcsi67283.2026.11412591
AI PROGRAMMING TUTOR FOR ANALYSIS AND CODE CORRECTIONPreprint
Brindha, N., Sangeetha, K., Suguna, M., Bala Selva Kumar, S., Brishco, K., & Dhanalakshmi, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6579750
Hidayatullah, L. A., Wisudiawan, G. A. A., & Nurtantyana, R. (2025) · International Conference on Artificial Intelligence, Blockchain, Cloud Computing, and Data Analytics (ICoABCD) · doi:10.1109/icoabcd67551.2025.11470870
Exploring the Holistic Impact of Adaptive Formative Assessment for Novice Programmers
Thangaraj, J., Monica, W. A. R. D., & O’Riordan, F. (2025) · International Conference on Computers in Education · doi:10.58459/icce.2025.6102 · Cited by 1
Decoding the Use of Artificial Intelligence in Programming Education: A Topic Modeling Analysis
Arslantaş, T. K., & Yürüm, O. R. (2026) · Computer Applications in Engineering Education · doi:10.1002/cae.70213
Pigozzo, A. B., & Pinheiro, D. J. C. (2026) · doi:10.5753/educomp.2026.18719
AI-Powered Intelligent Programming Tutor: Adaptive Learning and Auto-Evaluation
Rajendra, G., Reddy, A. V. N., Kumar, Y. N. V. N., Reddy, S. K. K., & Sai, K. J. (2026) · International Conference on Connected Intelligence for Industrial Applications (CI2A) · doi:10.1109/ci2a69097.2026.11577045
Utilizing Adaptive Gamification in an Intelligent Tutoring System for Software Engineering Education
Meißner, N., Trautwein, L., Speth, S., Koch, N. N., & Becker, S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786580.3786945
Münstermann, M., & Chounta, I. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3831394
Comparison of Human and GenAI Feedback Generation for Programming Assignments
Benko, Ľ., Pecuchova, J., & Benková, L. (2026) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-032-32104-6_4
Yepes, J. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60810
Exploring an Endless Learning Paradigm Supported by Large Model Services and Educational Agents
Ouyang, W., Mei, G., Pan, L., & Khan, S. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00046
Zhan, Y., Ji, H., Wu, W., Wu, Y., Arimilli, I., Tian, X., Jin, Q., & Yuan, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7396734
Designing Intelligent Tutoring Systems for Personalized Programming Education
Mavlyanovna, A. I. (2026) · European International Journal of Pedagogics
SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception RepairPreprint
Kopytskyi, M., Yang, J., & Seyedebrahimi, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15204
GenAI-Powered Postsecondary Computer Science Education Tools: A Systematic Literature Review
Sizemore, O., Bernstein, S., Ostrom, A., Rexin, T., & Stephens, K. (2026) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3856208.3856224
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
Self-citation
Metadata
- Authors
- Crow, T., Luxton-Reilly, A., & Wuensche, B.
- Affiliations
- University of Auckland (NZ)
- Venue
- Proceedings of the 20th Australasian Computing Education Conference
- Publisher
- ACM
- Published
- 2018-01-30
- Pages
- 53-62
- OpenAlex
- W2783082347
- Cited by
- 166
- References
- 27
- Citations by year
- 2026: 15 · 2025: 25 · 2024: 27 · 2023: 20 · 2022: 24 · 2021: 23 · 2020: 10 · 2019: 17 · 2018: 5
- FWCI
- 15.57
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Australasian Computing Education Conference (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- intelligent tutoring systems, programming education, novice programmers, worked solutions, quizzes, plans, scope, recursion, introductory programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.