A Systematic Review of the Effects of Automatic Scoring and Automatic Feedback in Educational Settings

“This study follows the steps described in [30] and involves three stages: 1) planning, 2) conducting, and 3) reporting.”

“This phenomenon is not exclusive to automatic feedback, as studies have shown that the number of past tests studied is a strong indicator of future tests [67].”

“Other inputs include a short free form (e.g., a short sentence [12]), long free form (e.g., an essay [13]).”

“However, automatic feedback has some drawbacks, including the complexity of measuring the feedback quality compared to a manual grader [28].”

“Unit testing evaluation of programming assessments was also an early entry into the area, given its simplicity of usage and implementation [8]–[10].”

“This set is completed by applications in medicine where virtual reality and other technologies are being used to support immersive practical experiences such as virtual artificial intelligent assistants [40], surgical skill assessments [41], [42], physiotherapy training [43], and clinical skills [44].”

“The institutions and instructors acquire the ability to increase students per instructor and provide fast and consistent results [6].”

“For the discipline-specific ones and using the International Standard Classification of Education (ISCED) [33], most of the work fell into the categories of the sciences [37], including areas like geology [38], mathematics [20], computer science [24], computer networking [39] (47% of papers).”

“We performed an iterative search using Web of Science as a platform for the search given the quality of its database and the iterative filtering capabilities [31].”

“With the rapid growth of technology and internet access, the use of automatic scoring and feedback has accelerated [4], [5].”

“Automatic scoring and feedback consist of calculating grades on students’ work and providing personalized feedback using technological tools that do not require human participation [1].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“We also expect to see an increase in student engagement driven by the ability to solve problems in a more interactive way given the feedback [108]–[110].”

“The main positive effects of automatic feedback include the students using the feedback for improvement [23], increased student engagement [24], [25], and reduction of instructor bias [26].”

“…set is completed by applications in medicine where virtual reality and other technologies are being used to support immersive practical experiences such as virtual artificial intelligent assistants [40], surgical skill assessments [41], [42], physiotherapy training [43], and clinical skills [44].”

“Programming problems are the easiest to use as input for this technology, as they appear in a structured language that computers can understand [95].”

“From a student perspective, work can be developed to improve the design and delivery of the automatic feedback to improve the experience, including finding a way to personalize the feedback [79].”

“Static ones include comparing the answer to a key or set of keys [16] or running a fixed set of unit test cases [48].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“Other potential adverse effects included loss of human interaction and lack of interpersonal skills while solving problems [64], and lack of personalized feedback that could help outlier students [68], especially struggling students [69].”

“…set is completed by applications in medicine where virtual reality and other technologies are being used to support immersive practical experiences such as virtual artificial intelligent assistants [40], surgical skill assessments [41], [42], physiotherapy training [43], and clinical skills [44].”

“Automated scoring and feedback are not only present both in MOOCs [80] and other systems where the scale requires it but also in smaller settings as a tool to support learning, including introductory programming classes [21], [81]–[83].”

“The tools used to produce grades and feedback dynamically include ontologies [7], neural networks to identify possible solutions [42], [47], machine learning used to identify learning paths [49], [50], and machine learning used to determine student’s risk of falling [51].”

“Like massive open online courses (MOOCs), many new learning environments would not be possible without them [2].”

“Similarly, a more personalized student experience would be expected to lead to better matching between the learning experience and the student’s learning style [111]–[115].”

“Other potential adverse effects included loss of human interaction and lack of interpersonal skills while solving problems [64], and lack of personalized feedback that could help outlier students [68], especially struggling students [69].”

“Some of the effects cannot be easily mitigated, i.e., bringing back the student-professor interaction [29].”

“These questions include multiple-choice [18], fill-in-the-blank [19], or those with a solution presented in a structured language, i.e., a mathematical formula [20] or a program [21], [22].”

“As mentioned by [32], content analysis allows one to find the research trends by analyzing the articles’ content and grouping them according to the shared characteristics.”

“The main positive effects of automatic feedback include the students using the feedback for improvement [23], increased student engagement [24], [25], and reduction of instructor bias [26].”

“We also expect to see an increase in student engagement driven by the ability to solve problems in a more interactive way given the feedback [108]–[110].”

“For the discipline-specific ones and using the International Standard Classification of Education (ISCED) [33], most of the work fell into the categories of the sciences [37], including areas like geology [38], mathematics [20], computer science [24], computer networking [39] (47% of papers).”

“A couple of papers target areas such as music where immediate feedback is also used to improve the student experience in general musical learning [45] and instrument learning [46].”

“Given these potential issues, studies on automatic feedback, problem set up, and its effects on student’s education and experience are still being produced [17].”

“With this in mind, automatic scoring and feedback are being employed broadly in computer science, mathematics [98]–[101], [47], and similarly analytical courses, together with language-learning areas [102]–[105], [36], [97].”

“This was replaced by human-computer interactions and students to work the system (e.g., creating multiple accounts in a MOOC to gain access to the answers [65].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“…and feed-back are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) extended essays [94].”

“With the rapid growth of technology and internet access, the use of automatic scoring and feedback has accelerated [4], [5].”

“With this in mind, automatic scoring and feedback are being employed broadly in computer science, mathematics [98]–[101], [47], and similarly analytical courses, together with language-learning areas [102]–[105], [36], [97].”

“Automated scoring and feedback are not only present both in MOOCs [80] and other systems where the scale requires it but also in smaller settings as a tool to support learning, including introductory programming classes [21], [81]–[83].”

“These include mathematics [47], code [22], and controlled environments such as simulations [40].”

“Using the feedback as a trial-and-error exercise or accessing answers using the internet can cause a very detrimental effect on learning [116].”

“They primarily highlighted students’ positive reception of features such as immediate grading with the allowance of multiple submissions [54].”

“Short essays can also be looked at, as their complexity tends to be low [96], while long essays prove the most considerable challenge for this technology [97].”

“Automated scoring and feedback are not only present both in MOOCs [80] and other systems where the scale requires it but also in smaller settings as a tool to support learning, including introductory programming classes [21], [81]–[83].”

“With the expansion of these technologies, we expect to see the benefits presented in the works surveyed in this paper materialized beyond the furthering of access, particularly an improvement in grading consistency [106] and the freeing of instructor time to dedicate to other activities [107].”

“Similarly, a more personalized student experience would be expected to lead to better matching between the learning experience and the student’s learning style [111]–[115].”

“With the expansion of these technologies, we expect to see the benefits presented in the works surveyed in this paper materialized beyond the furthering of access, particularly an improvement in grading consistency [106] and the freeing of instructor time to dedicate to other activities [107].”

“These issues included students learning to solve the assessment questions without understanding the underlying concepts [66].”

“Other inputs include a short free form (e.g., a short sentence [12]), long free form (e.g., an essay [13]).”

“…or equivalent level (92% of papers), with small numbers at the early education (2% of papers), such as Saha’s study of automatic grading of explanatory answers in middle school [34] and secondary education (6% of papers) such as Anohah’s analysis of high school computing science courses [35].”

“Unit testing evaluation of programming assessments was also an early entry into the area, given its simplicity of usage and implementation [8]–[10].”

“The most common uses of automatic scoring and feedback are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) exten…”

“Chatbot technologies could eventually help this area provide a more personalized experience [75], [76].”

“With this in mind, automatic scoring and feedback are being employed broadly in computer science, mathematics [98]–[101], [47], and similarly analytical courses, together with language-learning areas [102]–[105], [36], [97].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“With this in mind, automatic scoring and feedback are being employed broadly in computer science, mathematics [98]–[101], [47], and similarly analytical courses, together with language-learning areas [102]–[105], [36], [97].”

“The most commonly mentioned advantages included bias reduction [52] and grading consistency [26], the ability for the instructor to shift focus away from grading into other activities [49], and allowing more students to participate in the learning experiences [53].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“…and feed-back are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) extended essays [94].”

“This tool continues to be used today [3].”

“Despite this, some of the works dealt with teaching topics in early education, including handwriting [36] and basic math [20].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“Chatbot technologies could eventually help this area provide a more personalized experience [75], [76].”

“…and feed-back are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) extended essays [94].”

“For the discipline-specific ones and using the International Standard Classification of Education (ISCED) [33], most of the work fell into the categories of the sciences [37], including areas like geology [38], mathematics [20], computer science [24], computer networking [39] (47% of papers).”

“Examples include [70]–[74] (36%).”

“These questions include multiple-choice [18], fill-in-the-blank [19], or those with a solution presented in a structured language, i.e., a mathematical formula [20] or a program [21], [22].”

“There are fundamental questions about the quality of content and student privacy [117], for example, which are not considered in the study.”

“Examples include [70]–[74] (36%).”

“Specific learning can be mitigated by generating dynamic problems unique for each student [77], [78].”

“The tools used to produce grades and feedback dynamically include ontologies [7], neural networks to identify possible solutions [42], [47], machine learning used to identify learning paths [49], [50], and machine learning used to determine student’s risk of falling [51].”

“Generating the questions in multiple-choice format simplifies any issues related to interpreting the answers [7].”

“Despite this, some of the works dealt with teaching topics in early education, including handwriting [36] and basic math [20].”

“…set is completed by applications in medicine where virtual reality and other technologies are being used to support immersive practical experiences such as virtual artificial intelligent assistants [40], surgical skill assessments [41], [42], physiotherapy training [43], and clinical skills [44].”

“…of automatic scoring and feed-back are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3)…”

“This set is completed by applications in medicine where virtual reality and other technologies are being used to support immersive practical experiences such as virtual artificial intelligent assistants [40], surgical skill assessments [41], [42], physiotherapy training [43], and clinical skills [44].”

“The most common uses of automatic scoring and feedback are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) exten…”

“The main positive effects of automatic feedback include the students using the feedback for improvement [23], increased student engagement [24], [25], and reduction of instructor bias [26].”

“Specific learning can be mitigated by generating dynamic problems unique for each student [77], [78].”

“…or equivalent level (92% of papers), with small numbers at the early education (2% of papers), such as Saha’s study of automatic grading of explanatory answers in middle school [34] and secondary education (6% of papers) such as Anohah’s analysis of high school computing science courses [35].”

“Automated scoring and feedback are not only present both in MOOCs [80] and other systems where the scale requires it but also in smaller settings as a tool to support learning, including introductory programming classes [21], [81]–[83].”

“Together with this, several works focused on the ability to create custom learnings paths based on student performance [55], [26], [53]–[58], and the ability to flag students at high risk of not succeeding [51], [59]–[63], [61].”

“Other potential adverse effects included loss of human interaction and lack of interpersonal skills while solving problems [64], and lack of personalized feedback that could help outlier students [68], especially struggling students [69].”

“The tools used to produce grades and feedback dynamically include ontologies [7], neural networks to identify possible solutions [42], [47], machine learning used to identify learning paths [49], [50], and machine learning used to determine student’s risk of falling [51].”

“A couple of papers target areas such as music where immediate feedback is also used to improve the student experience in general musical learning [45] and instrument learning [46].”

“…and feed-back are in three areas: 1) programming problems through mechanisms such as assisting the student with the coding [84], [85], analyzing coding patterns [86], [87], automatic grading [88]–[92], and customized feedback [84], [86], [85]; 2) short essays [93]; and 3) extended essays [94].”

“This strategy is especially useful in large-scale classrooms when manual scoring is impractical (Susanti et al., 2023; Hahn et al., 2021).”

“For example, Hahn et al. (2021) mentioned that an automatic feedback system was applied in music education since AI could provide immediate feedback of students, so the instructors could use the feedback to improve students’ experience in general musical and instrument learning.”

“Prior work found automated feedback contributed to improving learning performance (Keuning et al., 2018) and reducing bias (Hahn et al., 2021).”

“…(Liu et al., 2022), socio-emotional support (Li & Xing, 2021), equity and inclusion support (Nixon et al., 2024), automatically generating educational content (Sarsa et al., 2022), especially questions (Wang et al., 2021), and automatic contextual feedback (see the review of Hahn et al., 2021).”

“However, providing detailed and effective feedback is often a labor-intensive task for educators, with grading frequently cited as a major contributor to teacher workload and stress (Hahn et al., 2021).”

“Meanwhile, marking and feedback is frequently cited by teachers as a significant contributor towards workload and work-related stress (Hahn et al., 2021).”

“Alternative approaches explored the use of learner log data to enhance ITS components such as skill models and hints (e.g., [4, 8, 5]) as well as machine learning-based techniques for automated questions and feedback generation (e.g., [33, 22]).”

“Although these systems can process vast amounts of data and generate insights, they may occasionally yield inaccurate or contextually inappropriate feedback [24, 46, 56].”

“The two main advantages revolve around time saving for educators and helping students understand their mistakes and learn more effectively (Hooda et al., 2022; Hahn et al., 2021; Calatayud et al., 2021).”

“While the nuances between human and AI-feedback in crowdsourcing have yet to be comprehensively studied, researchers in education have assessed the usage of automatic feedback as a learning tool [34, 38, 55, 92].”

“In assessment and grading, algorithms automatically grade multiple-choice tests (Hahn et al., 2021) and analyse student writing to provide feedback on grammar, spelling, and style (Link et al., 2022).”

“Automated scoring and feedback involve using AI technologies to assess student performance and provide immediate, constructive feedback (Hahn et al., 2021).”

“On the other hand, AES helps students improve their writing quality through rapid real-time scoring and feedback (Hahn et al., 2021).”

“Hahn et al.[25] found prior methods have typically attempted automatic at the undergraduate level, but typically reduced bias and assisted teachers by decreasing the time they were required to spend grading.”

“Scholars have acknowledged the potential benefits associated with the integration of AI in educational settings, with particular attention given to the advantages of automated scoring and immediate feedback, especially in the context of the COVID-19 pandemic (Hahn et al., 2021).”

“As AI technology has advanced, these systems have evolved to handle more complex tasks, such as evaluating constructed responses and essays (Hahn et al., 2021; Caval-canti et al., 2021).”

“…scoring students’ open-ended responses have been proposed as the solution to alleviate educators’ marking workload while improving scoring objectivity and consistency, particularly in courses with high student enrol - ments and limited resources (Attali & Burstein, 2006; Hahn et al., 2021).”

“From the learner’s perspective this opens the door to iteratively answering the same question multiple times until the correct solution is found, each time receiving immediate evaluation [5].”

“However, language teachers may have trouble providing feedback due to their workload despite their awareness of its critical role in the progression of students (Golzar et al., 2022; Hahn et al., 2021).”

“For teachers, AI can facilitate access to more valuable information, enabling them to enhance their lessons (Jihyun Kim and Sellnow, 2020), obtain detailed feedback (Hahn et al., 2021), and optimise their teaching e ffi cacy (Teo et al., 2021).”

“This can lead to students focusing on finding the right answer rather than thinking creatively (Hahn et al., 2021).”

“Feedback from such tools benefits students in many ways [6, 7].”

“NLP techniques have also been adopted to evaluate free-text coursework, such as essays, allowing for automated grading and providing constructive feedback on content and writing style (Altoe & Joyner, 2019; Hahn et al., 2021).”

“There are also disadvantages; students are given much less creative freedom when being assessed automatically [16], and this may impact their feelings about programming.”

“Beyond the fundamental task of how to evaluate the correctness of student answers, a key challenge is determining the format, frequency, and precision of the feedback [18, 33] provided to students after each submission.”

“Although educational frameworks such as the Bologna process [2] emphasize the importance of student-centered and feedback-oriented practices, their implementation is often constrained by limited time and instructional resources [3], [4].”

“One important psychometric consideration related to mixed-format assessments is that scoring of CR items mainly relies on human judgment, even when automated scoring processes are used (Hahn et al., 2021; Hussein et al., 2019).”

“The increasing use of Automated Essay Scoring (AES) tools, driven by AI advancements, further highlights the potential of these technologies to streamline and enhance the assessment of complex chemistry concepts (Hahn et al., 2021; Zhai, 2021).”

“As highlighted in a systematic review on the benefits of automated grading by Hahn et al. (2021), automated grading and feedback offer several learning advantages, including consistency, reduced grading bias, and increased student participation.”

“This can be limited in a learning environment, which we want to avoid [13].”

“Artificial intelligence (AI) has become increasingly integrated into the core functions of higher education, influencing how universities teach, assess, and support learners (Hahn et al., 2021; Hockly, 2019; Lee et al., 2023).”

“…based on automated writing evaluation, leveraging nat-ural language processing and machine learning (Flecken-stein, Liebenow, and Meyer 2023; Hahn et al. 2021); however, despite its demonstrated effectiveness (Fleckenstein, Liebenow, and Meyer 2023), automated writing evaluation has…”

“Configuring these tools to support new exercises is often difficult or even impossible [20] and many are known disincentivize creative or innovative solutions [21].”

“…consistency in grading [15], improve student learning through providing immediate feedback and enabling iteration over problems [15, 16], increase student autonomy through providing feedback flexibly [15], and encourage students to start assignments early through automated feedback schedules [9].”

“The methodology given by M.G. Hahn et al[3] investigates the pivotal role of automatic scoring and feedback tools in the landscape of online learning, highlighting their significance in enabling the widespread adoption of platforms like massive open online courses (MOOCs).”

“However, providing individualized, high-quality essay feedback is time-consuming [29, 99, 108], thus limiting teachers’ capacity to provide detailed input to all students and thereby reducing opportunities for effective practice [26].”

“Instead of identifying a student’s specific error and offering targeted guidance, the model may return a complete solution without addressing the underlying confusion (Hahn, Navarro, De La Fuente Valentin, & Burgos, 2021).”

“Ultimately, the findings are expected to provide actionable insights for educators, policymakers, and developers toward creating hybrid assessment models that combine the efficiency of AI with the depth of human judgment (Hahn et al., 2021).”

“These limitations can disincentivize students from building creative or innovative solutions [4].”

“Conventional handwritten scoring systems, although comprehensive, are time-consuming and susceptible to human bias and subjectivity [7].”

“Students and teachers may run into issues with precise specifications being enforced [10], and may feel unfairly penalised for missing elements that they believe are unimportant, such as capitalisation and punctuation [20] (although this does give them to chance to program to specific criteria).”

“…questions to provide predefined feedback messages, while other approaches attempted to train classifiers on past student responses to select predefined feedback messages, but lacked the ability to provide open-Email ended, text-based feedback Cavalcanti et al. (2021); Hahn et al. (2021).”

“While automatic scoring enables educational scaling [10], human feedback produces superior conceptual understanding [11], and vague feedback risks encouraging students to game the system [6].”

“However, feedback is challenging and time-consuming to provide at scale, especially for complex performances such as essay writing, often resulting in students receiving insufficient or no feedback [15, 16, 53, 59].”

“For example, recent work [ 1 , 2 ] highlights advances in learning analytics for generating personalized feedback at scale and ongoing debates about feedback quality and ethics in automated systems.”

“In essence, such methods were static standard answers or scoring rubrics, rendering them unable to handle complex problems [12], [13].”

“…systems can enhance student performance and improve the speed and consistency of feedback delivery, although challenges remain in demonstrating benefits for instructors, ensuring pedagogical usefulness, and generating feedback that is sufficiently informative rather than merely corrective [13].”

“The advantages of using AES for scoring are reducing the teacher’s workload by automating the grading process of essays [2] as well as minimizing bias and inconsistency in scoring [3].”

“This design embodies a “human-in-the-loop” approach increasingly recognized as best practice in educational technology (Hahn et al., 2021).”

“[15].”

“Beyond these issues, asking teachers to assess comments manually undermines a key benefit of automated assessment, which is providing rapid feedback to students [5, 9, 16, 17, 25].”

“It evaluated trends, benefits, and drawbacks of automatic scoring tools and feedback methods [12].”

“At the same time, they can offer further educational advantages by reducing biases and increasing grading consistency [15].”

“Therefore, there has been growing interest since 2016 in developing an efficient and accurate automated scoring system based on machine learning [4].”

“AI quizzes are more efficient in tracking a student’s performance over time and they offer a clear picture of the strengths as well as weaknesses of the students’ abilities (Hahn et al., 2021).”

“Additionally, automatic scoring and feedback can scale the number of students without adding a proportional number of instructors, improve student experience by reducing time between submission and feedback, and remove bias in scoring (Hahn et al., 2021).”

“…one or have not been considered when preparing the problem, (b) such restrictions may train the student to answer the question instead of learning the concepts [21]; and (c) in the case of LLMs, this may result in a downward trend in final grades with an increased reliance on the use of LLMs [17].”

“Additionally, this system accelerates the grading process, providing timely feedback crucial for students’ academic progress and comprehension [2, 3].”