Assistance and Feedback Mechanism in an Intelligent Tutoring System for Teaching Conversion of Natural Language into Logic
Open accessThe DOI matches its Crossref record; 71 of the 73 references it cites were found in a registry or an index.
Abstract
Logic as a knowledge representation and reasoning language is a fundamental topic of an Artificial Intelligence (AI) course and includes a number of sub-topics. One of them, which brings difficulties to students to deal with, is converting natural language (NL) sentences into first-order logic (FOL) formulas. To assist students to overcome those difficulties, we developed the NLtoFOL system and equipped it with a strong assistance and feedback mechanism. In this work, first, we present that feedback mechanism. The mechanism can provide assistance before an answer is submitted, if requested, but mainly it provides assistance after an answer is submitted. To that end, it characterizes the answer in terms of completeness and accuracy to determine the level of incorrectness, based on an answer categorization scheme, introduced in this paper. The automatically generated natural language feedback sequences grow from general to specific and can include statements on a student’s metacognitive state. Feedback is provided as natural language sentences automatically generated through a template-based natural language generation mechanism. Second, we present an extensive evaluation of the effectiveness of the assistance and feedback mechanism on students’ learning. The evaluation of feedback with students showed that full feedback sequences lead to greater learning gains than sequences consisting of only flag feedback and bottom-out hints ( n = 226), and that generic, template-based feedback sequences are comparable to the utility of problem-specific hints generated by human tutors ( n = 120).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As already mentioned, the main uses of AI applied to education are related to tutoring and assessment [40], although we also find other examples of use such as personalization [22,41,46] or quality assessment [20,37,39].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [40]
In the citing paper
“one of the works determines that error detection is essential to achieve a better understanding and for pedagogical decision-making [40]”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [40]
In the citing paper
“As already mentioned, the main uses of AI applied to education are related to tutoring and assessment [40], although we also find other examples of use such as personalization [22,41,46] or quality assessment [20,37,39].”
From Artificial Intelligence for Student Assessment: A Systematic Review (2021) · cites this paper as [40]
In the citing paper
“Selected studies suggest that survey data-based LA can foster predictive feedback (Broos et al., 2020), recommendation (Perikos et al., 2017), reflective feedback (Smith, 2020; Tsai et al., 2021), and assessment (Dalla Valle et al., 2018).”
From A systematic review of the role of learning analytics in enhancing feedback practices in higher education (2022) · cites this paper as Perikos et al., 2017
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,583
Assessment and Classroom Learning
Black, P., & Wiliam, D. (1998) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969595980050102 · Cited by 8,152
Shute, V. J. (2008) · Review of Educational Research · doi:10.3102/0034654307313795 · Cited by 4,631
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008
VanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323
Generating Typed Dependency Parses from Phrase Structure Parses
de Marneffe, M.-C., MacCartney, B., & Manning, C. D. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3wtkaidk6ea8 · Cited by 2,258
Cognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982
Moreno, R., & Mayer, R. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9047-2 · Cited by 1,666
Logical Foundations of Artificial Intelligence
(1987) · Elsevier eBooks · doi:10.1016/c2009-0-27551-9 · Cited by 1,567
Evidence in support of written corrective feedback
Bitchener, J. (2008) · Journal of Second Language Writing · doi:10.1016/j.jslw.2007.11.004 · Cited by 871
Knowledge Representation and Reasoning
(2024) · Encyclopedia of Computer Graphics and Games · doi:10.1007/978-3-031-23161-2_300700 · Cited by 819
The Behavior of Tutoring Systems
VanLehn, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 749
Exploring the Assistance Dilemma in Experiments with Cognitive Tutors
Koedinger, K. R., & Aleven, V. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9049-0 · Cited by 632
Help Seeking and Help Design in Interactive Learning Environments
Aleven, V., Stahl, E., Schworm, S., Fischer, F., & Wallace, R. (2003) · Review of Educational Research · doi:10.3102/00346543073003277 · Cited by 609
Orivel, F. (1987) · Economics of Education · doi:10.1016/b978-0-08-033379-3.50014-0 · Cited by 578
Feedback in written instruction: The place of response certitude
Kulhavy, R. W., & Stock, W. A. (1989) · Educational Psychology Review · doi:10.1007/bf01320096 · Cited by 531
The Andes Physics Tutoring System: Lessons Learned
VanLehn, K. A., Lynch, C. F., Schulze, K. G., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D. J., Weinstein, A., & Wintersgill, M. C. (2005) · doi:10.5555/1434930.1434932 · Cited by 518
Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor
Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(2)02 · Cited by 406
Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011) · Learning and Instruction · doi:10.1016/j.learninstruc.2010.07.004 · Cited by 382
Feedback Strategies for Interactive Learning Tasks
Narciss, S. (2007) · doi:10.4324/9780203880869.ch11 · Cited by 350
The role of feedback and self-efficacy on web-based learning: The social cognitive perspective
Wang, S.-L., & Wu, P.-Y. (2008) · Computers & Education · doi:10.1016/j.compedu.2008.03.004 · Cited by 320
Corbett, A. T., & Anderson, J. R. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365111 · Cited by 280
A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors
Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009) · doi:10.5555/1734243.1734245 · Cited by 277
Formative assessment and feedback: Making learning visible
Havnes, A., Smith, K., Dysthe, O., & Ludvigsen, K. (2012) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2012.04.001 · Cited by 276
Exploring feedback and student characteristics relevant for personalizing feedback strategies
Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G., & Melis, E. (2014) · Computers & Education · doi:10.1016/j.compedu.2013.09.011 · Cited by 260
Help Helps, But Only So Much: Research on Help Seeking with Intelligent Tutoring Systems
Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0089-1 · Cited by 255
Effects of feedback in a computer-based assessment for learning
van der Kleij, F. M., Eggen, T. J. H. M., Timmers, C. F., & Veldkamp, B. P. (2012) · Computers & Education · doi:10.1016/j.compedu.2011.07.020 · Cited by 239
Limitations of Student Control: Do Students Know when They Need Help?
Aleven, V., & Koedinger, K. R. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45108-0_33 · Cited by 236
The Cognitive Tutor Authoring Tools (CTAT): Preliminary Evaluation of Efficiency Gains
Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2006) · Lecture Notes in Computer Science · doi:10.1007/11774303_7 · Cited by 215
Fostering the Intelligent Novice: Learning From Errors With Metacognitive Tutoring
Mathan, S. A., & Koedinger, K. R. (2005) · Educational Psychologist · doi:10.1207/s15326985ep4004_7 · Cited by 197
Effective Feedback Content for Tutoring Complex Skills
McKendree, J. (1990) · Human-Computer Interaction · doi:10.1207/s15327051hci0504_2 · Cited by 149
Intelligent Tutors for All: The Constraint-Based Approach
Mitrovic, A., Martin, B., & Suraweera, P. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.74 · Cited by 134
On the Benefits of Seeking (and Avoiding) Help in Online Problem-Solving Environments
Roll, I., Baker, R. S. J. D., Aleven, V., & Koedinger, K. R. (2014) · Journal of the Learning Sciences · doi:10.1080/10508406.2014.883977 · Cited by 109
The effect of positive feedback in a constraint-based intelligent tutoring system
Mitrović, A., Ohlsson, S., & Barrow, D. K. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.07.002 · Cited by 101
Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback
Gerdes, A., Heeren, B., Jeuring, J., & van Binsbergen, L. T. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0080-x · Cited by 96
ASPIRE: An Authoring System and Deployment Environment for Constraint-Based Tutors
Mitrovic, A., Martin, B., Suraweera, P., Zakharov, K., Milik, N., Holland, J., & McGuigan, N. (2009) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2009-19(2)03 · Cited by 71
Presenting questions, processing responses, and providing feedback in CAI
Wager, W., & Wager, S. (1985) · Journal of Instructional Development · doi:10.1007/bf02906047 · Cited by 67
The AProS Project: Strategic Thinking & Computational Logic
Sieg, W. (2007) · Logic Journal of IGPL · doi:10.1093/jigpal/jzm026 · Cited by 65
Recasting the feedback debate: benefits of tutoring error detection and correction skills
Mathan, S. A., & Koedinger, K. R. (2003) · Cited by 60
Delivering hints in a dialogue-based intelligent tutoring system
Zhou, Y., Freedman, R., Glass, M., MICHAEL, J. A., Rovick, A. A., & Evens, M. W. (1999) · National Conference on Artificial Intelligence · Cited by 58
The Logic-ITA in the Classroom: A Medium Scale Experiment
Yacef, K. (2005) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2005-15(1)03 · Cited by 47
A Taxonomy for the Timing of Feedback in Computer-Based Instruction.
Dempsey, J. V., & Wager, S. U. (1988) · Educational Technology archive · Cited by 44
Error-Visualization by Error-Based Simulation and Its Management
Hirashima, T. (1998) · Cited by 41
Error-Based Simulation for Error-Visualization and Its Management
Hirashima, T., Horiguchi, T., & Kashihara, A. (1998) · Cited by 39
Intelligent teaching assistant systems
Yacef, K. (2003) · International Conference on Computers in Education, 2002. Proceedings. · doi:10.1109/cie.2002.1185885 · Cited by 39
Berlanga, A. J., van Rosmalen, P., Boshuizen, H. P. A., & Sloep, P. B. (2012) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2011.00425.x · Cited by 34
Adaptation of Elaborated Feedback in e-Learning
Vasilyeva, E., Pechenizkiy, M., & Bra, P. M. D. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70987-9_26 · Cited by 33
Relative Effectiveness of Computer-Based and Human Feedback for Enhancing Student Learning
Mandernach, J. (2005) · The Journal of Educators Online · doi:10.9743/jeo.2005.1.5 · Cited by 32
An Adaptive Feedback Framework to Support Reflection, Guiding and Tutoring
Gouli, E., Γόγουλου, Α., Papanikolaou, K. A., & Grigoriadou, M. (2006) · IGI Global eBooks · doi:10.4018/978-1-59140-690-7.ch008 · Cited by 30
Muñoz‐Merino, P. J., Kloos, C. D., & Muñoz-Organero, M. (2010) · IEEE Transactions on Education · doi:10.1109/te.2010.2045001 · Cited by 25
An Empirical Study of Errors in Translating Natural Language into Logic
Barker-Plummer, D., Cox, R., Dale, R., & Etchemendy, J. (2008) · eScholarship (California Digital Library) · Cited by 21
KRRT: Knowledge Representation and Reasoning Tutor System
Alonso, J.-A., Aranda, G., & Martn–Mateos, F. J. (2007) · Springer eBooks · doi:10.1007/978-3-540-75867-9_51 · Cited by 16
Perikos, I., Grivokostopoulou, F., Kovas, K., & Hatzilygeroudis, I. (2016) · Expert Systems · doi:10.1111/exsy.12182 · Cited by 12
Self-citationUsing edit distance to analyse errors in a natural language to logic translation corpus
Barker-Plummer, D., Dale, R., & Cox, R. (2012) · Figshare · Cited by 11
Student Translations of Natural Language into Logic: The Grade Grinder Corpus Release 1.0.
Barker-Plummer, D., Cox, R., & Dale, R. (2011) · Cited by 10
A Web-Based Interactive System for Learning NL to FOL Conversion
Hatzilygeroudis, I., & Perikos, I. (2009) · Studies in computational intelligence · doi:10.1007/978-3-642-02937-0_27 · Cited by 9
Self-citationDifficulty Estimator for Converting Natural Language into First Order Logic
Perikos, I., Grivokostopoulou, F., Hatzilygeroudis, I., & Kovas, K. (2011) · Smart innovation, systems and technologies · doi:10.1007/978-3-642-22194-1_14 · Cited by 9
Self-citationHints: You Can't Have Just One
Goldin, I. M., Koedinger, K. R., & Aleven, V. (2013) · Cited by 8
Dimensions of difficulty in translating natural language into first order logic
Barker-Plummer, D., Cox, R., & Dale, R. (2009) · Cited by 7
Adapting Tutoring Feedback Strategies to Motivation
Narciss, S., Александрович, С. С., & Andrés, E. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11200-8_22 · Cited by 6
Automatic Marking of NL to FOL Conversions
Perikos, I., Grivokostopoulou, F., & Hatzilygeroudis, I. (2012) · doi:10.2316/p.2012.774-070 · Cited by 6
Self-citationXTutor: An Intelligent Tutor System for Science and Math Based on Excel
Gheorghiu, R., & VanLehn, K. A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69132-7_98 · Cited by 5
Effects of delayed and immediate feedback in the computer-based testing environment
Powers, S., & López, L. M. G. (2009) · Cited by 5
Teaching NL to FOL and FOL to CF Conversions.
Hatzilygeroudis, I. (2007) · Cited by 4
Self-citationMaestro, P., & Simón-Hurtado, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_96 · Cited by 4
Research works printed but not linked: 1
- Moreno, A., & Budesca, N. (2000). Mathematical logic tutor-propositional calculus. In Proceedings of the First International Congress on Tools for Teaching Logic (pp. 99–106).
Cited by
Lin, C.-C., Huang, A. Y. Q., & Lu, O. H. T. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00260-y · Cited by 669
“…Yang et al., 2021a, b) from Asia, 9 papers (Afzaal et al., 2021; Corrigan et al., 2015a, b; Guerrero-Higueras et al., 2018; Llurba et al., 2022; Perikos et al., 2017; Ruipé-rez-Valiente et al., 2021; Serrano-Laguna et al., 2018; Tobarra et al., 2021) from Europe, 5 papers (Montpetit &…”
Artificial Intelligence for Student Assessment: A Systematic Review
González-Calatayud, V., Prendes-Espinosa, P., & Roig-Vila, R. (2021) · Applied Sciences · doi:10.3390/app11125467 · Cited by 503
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & De Weerdt, J. (2021) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 289
Banihashem, S. K., Noroozi, O., van Ginkel, S., Macfadyen, L. P., & Biemans, H. J. A. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100489 · Cited by 246
Ferreira‐Mello, R., André, M., Pinheiro, A., Costa, E., & Romero, C. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1332 · Cited by 193
“Moreover, there are a few works that aid the instructor to monitor students within this tools (Scheuer and McLaren, 2008; Perikos et al., 2017).”
Background
Metadata
- Authors
- Perikos, I., Grivokostopoulou, F., & Hatzilygeroudis, I.
- Affiliations
- University of Patras (GR)
- Corresponding author
- Perikos, Isidoros
- Venue
- International Journal of Artificial Intelligence in Education
- Publisher
- Elsevier BV
- Published
- 2017-09
- Volume
- 27
- Issue
- 3
- Pages
- 475-514
- ISSN
- 1560-4292 (ISSN-L)
- OpenAlex
- W2590947568
- Cited by
- 44
- References
- 73
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 9 · 2023: 3 · 2022: 3 · 2021: 2 · 2020: 4 · 2019: 5 · 2018: 2 · 2017: 3
- FWCI
- 2.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor, 3162 d)
- Locations
- International Journal of Artificial Intelligence in Education (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Edcuational Technology Systems
- Keywords
- intelligent tutoring systems, automated feedback, hint generation, artificial intelligence in education, knowledge representation, learning gains
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.