L2 student engagement with automated feedback on writing: Potential for learning and issues of trust

“Some researchers, such as Ranalli [30], went further and strongly criticized the use of AWE tools in the language classroom and pointed out that these tools “ did not live up to expectations ” (p. 2).”

“In addition, the instructor ’ s willingness to apply AWF (e.g. Jiang et al ., 2020), perceived trust in AWF (e.g. Ranalli, 2021; Zhai & Ma, 2021), perceived ease of use (R. Li, 2021), and self-efficacy (e.g. Li et al ., 2019) are all found to play a role in users ’ interaction with AWF.”

“Researchers also note challenges such as the varied level of technological readiness among teachers and students as well as concerns regarding accuracy, biases, accountability, transparency, and ethics (e.g., Kohnke et al., 2023; Memarian & Doleck, 2023; Ranalli, 2021).”

“Nevertheless, we reviewed the publications in these journals because they have been widely recognized as the most rigorous among research journals (Authors, 2020, 2021).”

“In previous studies, researchers identified that students used similar cognitive strategies, such as noticing and understanding (e.g. Zhang 2017; Koltovskaia 2020; Ranalli 2021).”

“Previous studies have generally attributed the individual differences of learners to language pro fi ciency (Zhang and Hyland 2018; Ranalli 2021).”

“This finding underlines the great potential for integrating Grammarly feedback into writing instruction, supplementing teacher feedback, as reported in previous studies by O’Neill and Russell (2019b), Ventayen and Orlanda-Ventayen (2018), and Ranalli (2021).”

“For example, Ranalli (2021) revealed that students’ distrust of Grammarly-generated feedback led to a rushed and superficial approach when addressing the feedback.”

“…’ Neill & Russell, 2019b; Parra & Calero, 2019; John & Woll, 2020; Koltovskaia, 2020; Tabrizi & Etemad, 2021; Guo et al., 2021; Thi & Nikolov, 2021; Ranalli, 2021; Barrot, 2021) Spelling (Blázquez & Fan, 2019; Chacón-Beltrán, 2016; Lawley, 2016; Rimbar, 2017; Tight, 2017; Lin et al., 2017)…”

“We adopted a multiple-case study design as an established means of gathering rich data to analyse students’ experiences of language learning (Duff, 2008) and responses to feedback (e.g. Han & Hyland, 2015; Ranalli, 2021).”

“However, since other studies (e.g., Guo et al., 2021; Jiang & Yu, 2020; Ranalli, 2021) found that AWE systems might mislabel a correct usage as erroneous, the second author of the study and an independent researcher checked the accuracy of errors identified by Criterion following the procedure…”

“…a good understanding of internal factors of students (e.g., backgrounds, attitudes, confidence, and motivations of students) (Ferris et al., 2013; Ranalli, 2021); (3) student engagement is subject to change over time, which indicates that student engagement can be either fostered or Frontiers…”

“Lastly, the synthesis revealed that certain variables, such as social and personal factors influencing AWE engagement (e.g., Bru-dermann et al., 2021) and the effect of writing task or text genre on AWE engagement (e.g., Ranalli, 2021), were not systematically considered in the reviewed studies.”

“…Phakiti 2014; Dikli and Bleyle 2014), a few recent studies have viewed AWE systems as a supplement to teacher-student or student-student interaction rather than a complete replacement, focusing on student engagement with AWE feedback in their writing process (Ranalli 2021; Zhang and Hyland 2018).”

“For instance, mixed results were frequent regarding AWE’s impact on student engagement because of the individual, contextual, affective and cognitive factors that come into play (Koltovskaia, 2020; Ranalli, 2021; Shi, 2021).”

“Therefore, we examined the accuracy of Grammarly feedback following the work of Ranalli (2021) and classified it into three categories: accurate, inaccurate, and indeterminate feedback.”

“As educational technologies have become central to students ’ learning experiences, understanding the development of trust between students and these systems has become important for their effective use and adoption [6 – 9].”

“See Ranalli (2021)‘s findings that suggest that language learners’ experience with technology may influence their trust in technology.”

“Finally, only a handful of studies examined what we called participants’ (g) sensemaking with AI, or how various individual and contextual factors shaped the ways they engaged with the tool, feedback, or writing process (e.g., Ranalli, 2021).”

“Nevertheless, some researchers argued that in real-time practice, the retention of AI-feedback-assisted writing is not satisfying (Stevenson & Phakiti, 2014), since some students only used AI feedback tool for proofreading “with no or little cognitive processing” (Ranalli, 2021).”

“…education, researchers have suggested that teacher feedback and AWE feedback should be integrated to enhance the effectiveness of feedback for L2 writing, considering the affor-dances and limitations of the two feedback sources (Link et al., 2022; Ranalli, 2021; Thi & Nikolov, 2022).”

“This includes validating the claims made by software companies and holding them accountable (Chapelle et al., 2015; Ranalli, 2021).”

“Studies have found that users’ perceptions of privacy risks and expectations of privacy protection are directly related to the continuity of courses, learning activities, and learning performance (May and George, 2011; Jim, 2021).”

“…under the research topic of Learner Perceptions include Almusharraf and Alotaib (2022), Barrot (2023), Ebadi et al. (2022), Ebyary (2022), Fan (2023), Miranty and Widiati (2021), Parra and Calero (2019), Ranalli (2021), Thi and Nikolov (2022), Ventayen & Orlanda-Ventayen (2018), and Zaini (2018).”

“To assist teachers in providing frequent and timely feedback to large numbers of students, various automated writing evaluation (AWE) tools (Ranalli, 2021) have been developed to provide automated written corrective feedback (Ranalli, 2019) on students ’ writing.”

“Because the role of trust in feedback sources has been claimed to in fl uence learners ’ writing performance, there has been a clarion call for experimental studies to be conducted to verify the effect of trust in feedback on students ’ writing performance (Ranalli, 2021).”

“In previous studies on automated writing evaluation and feedback quality, trust in feedback refers to the degree to which learners believe automated feedback tools provide valuable comments and suggestions (e.g., Ranalli 2021).”

“Students have been shown to heavily rely on AI for superficial functions such as grammar checking, spelling correction, and paraphrasing, and not for more substantial and challenging learning exercises (Le, 2023; Ranalli, 2021).”

“Furthermore, the role of teacher feedback has been well documented (e.g., Sermsook et al., 2017), but little is known about the role of AI-operated feedback in noticing errors and uptake (Barrot, 2023a; Ranalli, 2021).”

“There has been increasing interest in L2 student engagement as the key to unlocking how and why learners do or do not gain utility from written feedback (Han & Hyland, 2015; Han & Xu, 2021; Ranalli, 2021; Uscinski, 2017; Yu & Jiang, 2020; Zhang, 2020).”

“Many recent studies also notice students’ unaction to some AWE feedback, as in Lavolette et al. (2015), Zhang (2016), Bai and Hu (2017), Ranalli (2018, 2021), and Koltovskaia (2020); and to peer feedback as in Yu et al. (2018), Fan and Xu (2020), and Aben et al. (2022) as well, to name a few.”

“Thisimmediacysignificantlyenhancesinteractivity andreducesfeelingsofisolationinvirtuallearningenvironments [85].”

“Ranalli (2021) noted that students’ willingness to accept feedback from AI tools depends on their trust in the tool.”

“First, the correlation between automated and human grading of student writing (Bridgeman et al., 2012); and second, the effects of automated analysis on student writing performance (Jiang et al., 2020; Ranalli, 2021).”

“…students with different language proficiency were employed to analyze students’ engagement with AWCF qualitatively in the use of stimulated recall, semi-structured interviews and retrospective verbal reports, reflective journals, etc. (Han, 2017; Koltovskaia, 2020; Zhang, 2020; Ranalli, 2021).”

“First, as comments are provided by the instructor, human-delivered SWCF may be perceived as trustable, which is said to increase revision engagement (Ranalli, 2021), by learners.”

“Recent studies have highlighted the importance of student engagement with feedback in L2 writing (e.g. Zhang and Hyland 2018; Fan and Xu 2020; Ranalli 2021).”

“The findings underscore both the importance of accurate and appropriately aligned AF and the need for learners to develop metacognitive skills in AI-assisted learning environments (Ranalli 2021).”

“The impact of GenAI tools on improving linguistic accuracy is well documented, with platforms such as Grammarly and Criterion providing precise, immediate feedback to address common writing challenges (Liu et al., 2023; Ranalli, 2021).”

“Some scholars also express concerns over the potential misuse of AWE, such as relying solely on automated feedback without incorporating certain human evaluations (Ranalli, 2021).”

“Ranalli (2021) found that trust, rather than proficiency, shaped feedback engagement among Chinese EsL students (B2–C1), highlighting the complex, contextual nature of engagement (Hiver et al., 2024).”

“It was utilized largely for proofreading purposes rather than contributing to learning (Ranalli, 2021).”

“…& Wilson, 2020) students in the United States; undergraduate students in Taiwan (Liao, 2016), Japan (Suzuki et al., 2019), the United States (Ranalli, 2021), Saudi Arabia (Aluthman, 2016), Hong Kong (Cheng, 2017), China (Zhang & Hyland, 2018), and Bahraini (Wali & Huijser, 2018);…”

“These systems, driven by natural language processing algorithms, provide diagnostic scores and comprehensive feedback on written performances (e.g., McCarthy et al., 2022; Ragheb Al-Ghezi et al., 2023; Ranalli, 2021).”

“…on feedback sources, concentrating on teachers (e.g. Han & Hyland, 2015; Zheng & Yu, 2018), peers (e.g. Geng & Yu, 2023; Han & Yang, 2021), automated systems (e.g. Ranalli, 2021; Y. Xu & Han, 2020), and multiple-sources interaction environment (e.g. Z. Zhang & Hyland, 2022; Y. Zhang & Jiang, 2022).”

“These figures are higher than studies of teacher WFC in tertiary-level process writing environments (Christiansen & Bloch, 2016; Conrad & Goldstein, 1999; Ene & Upton, 2014; Hyland, 2003; Nurmukhamedov & Kim, 2009; Ranalli, 2021; Zhang & Hyland, 2018), where comments across multiple drafts serve as scaffolding to help…”

“Most studies on learner engagement have focused on university contexts (Zhang & Hyland, 2018; Zhang & Hyland, 2022; Man et al., 2021; Zhang, 2022; Koltovskaia & Mahapatra, 2022; Ranalli, 2021).”

“Students may allocate a hierarchy of trust and value to different feedback processes based on the feedback providers’ qualities, regardless of the content of the feedback message received (Ranalli 2021; Zhou et al. 2021).”