The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 1×
“The bene fi ts of feedback on improving writing have been widely reported in prior studies (Bakla, 2020; Hattie & Timperley, 2007; Huisman et al., 2019).”
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
The Role of the Linguistic Environment in Second Language Acquisition
Long, M. H. (1996) · Handbook of Second Language Acquisition · doi:10.1016/b978-012589042-7/50015-3 · Cited by 3,927 · cited 2×
“The use of AWE may also be informed by the interaction hypothesis, which emphasizes the modi fi cation of linguistic inputs through various interaction forms such as clarifying, repeating, and expanding (Long, 1996).”
BackgroundMethodFlipped classroom improves student learning in health professions education: a meta-analysis
HEW, K. F., & LO, C. K. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1144-z · Cited by 1,207
Complexity, accuracy and fluency in second language acquisition
Housen, A., & Kuiken, F. (2009) · Applied Linguistics · doi:10.1093/applin/amp048 · Cited by 863 · cited 3×
“Previous researchers held that constructs describing learners ’ language performance and pro fi ciency are multi-componential in nature, and the principal components of these constructs can be comprehensively and appropriately captured by the notions of complexity, accuracy, and fl uency (CAF) (Ellis & Ellis, 2008; Hous…”
BackgroundMethodPre-post effect sizes should be avoided in meta-analyses
Cuijpers, P., Weitz, E., Cristea, I. A., & Twisk, J. (2017) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796016000809 · Cited by 432
The Efficacy of Written Corrective Feedback in Improving L2 Written Accuracy: A Meta‐Analysis
Kang, E., & Han, Z. (2015) · The Modern Language Journal · doi:10.1111/modl.12189 · Cited by 385
Linguistic features in writing quality and development: An overview
Crossley, S. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.11.03.01 · Cited by 310
Huisman, B., Saab, N., van den Broek, P., & van Driel, J. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1545896 · Cited by 306
Wilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
The effects of computer-generated feedback on the quality of writing
Stevenson, M., & Phakiti, A. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.11.007 · Cited by 275 · cited 2×
“To our knowledge, the current study is the fi rst to follow up on multiple calls for quantitative synthesis of AWE ’ s overall effect (e.g., Fu et al., 2022; Nunes et al., 2021; Stevenson & Phakiti, 2014).”
BackgroundMethodZhu, M., Liu, O. L., & Lee, H.-S. (2020) · Computers & Education · doi:10.1016/j.compedu.2019.103668 · Cited by 229
Dikli, S., & Bleyle, S. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.03.006 · Cited by 193 · cited 1×
“Students were generally satis fi ed with the AWE corrective feedback on grammar and mechanics but argued that higher-level issues such as ideation, organization, and style still depended on teachers ’ assistance (Dikli & Bleyle, 2014; Wilson & Czik, 2016).”
BackgroundWang, Y.-J., Shang, H.-F., & Briody, P. (2013) · Computer Assisted Language Learning · doi:10.1080/09588221.2012.655300 · Cited by 185 · cited 1×
“The widespread use of AWE has led to an increasing number of empirical studies investigating its effectiveness in improving writing (e.g., Cheng, 2017; Sun & Fan, 2022; Wang, 2020; Wang et al., 2013; Wilson & Czik, 2016).”
BackgroundAutomated Writing Evaluation and Feedback: Multiple Metrics of Efficacy
Wilson, J., & Roscoe, R. D. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633119830764 · Cited by 176 · cited 3×
“A similar insigni fi cant result was reported in another study evaluating the effectiveness of Project Essay Grade (PEG) writing using multiple metrics (Wilson & Roscoe, 2020).”
BackgroundResultKarim, K., & Nassaji, H. (2020) · Language Teaching Research · doi:10.1177/1362168818802469 · Cited by 164
AcaWriter: A learning analytics tool for formative feedback on academic writing
Knight, S., Shibani, A., Abel, S., Gibson, A., & Ryan, P. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.12.01.06 · Cited by 151
Palermo, C., & Thomson, M. M. (2018) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2018.07.002 · Cited by 107 · cited 2×
“Writing is a complex cognitive skill that requires not only sustained, deliberate practice but frequent, appropriate, and effective feedback as well (Palermo & Thomson, 2018).”
BackgroundRoscoe, R. D., Wilson, J., Johnson, A. C., & Mayra, C. R. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2016.12.076 · Cited by 98 · cited 3×
“They rated AWE as easy to use, fair, and useful but critiqued the tool in terms of the quantity, speci fi city, and personalization of the feedback (Roscoe et al., 2017).”
BackgroundMethodInfluentialChang, T.-S., Li, Y., Huang, H.-W., & Whitfield, B. (2021) · 2021 2nd International Conference on Education Development and Studies · doi:10.1145/3459043.3459065 · Cited by 74
Computer-Mediated Corrective Feedback to Improve L2 Writing Skills: A Meta-Analysis
Mohsen, M. A. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331211064066 · Cited by 56 · cited 9×
“Mohsen (2022) con fi rmed a large effect size ( d = 1.21) of computer-mediated written feedback (CMWF), including both AWE and non-AWE technologies, on L2 writing skills.”
BackgroundMethodResultInfluential
Cited by
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311
A systematic review of AI-based automated written feedback research
Shi, H., & Aryadoust, V. (2024) · ReCALL · doi:10.1017/s0958344023000265 · Cited by 222
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145
Shi, H., Chai, C. S., Zhou, S., & Aubrey, S. (2026) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2454541 · Cited by 109
“The studies of AWE in the past two decades have shown that AWE has a positive overall effect on writing quality and is helpful for L2 students to improve their linguistic accuracy (Ding & Zou, 2023; Karatay & Karatay, 2024; Shi & Aryadoust, 2024; Zhai & Ma, 2023).”
ChatGPT for Automated Writing Evaluation in Scholarly Writing Instruction
Parker, J. L., Becker, K., & Carroca, C. (2023) · Journal of Nursing Education · doi:10.3928/01484834-20231006-02 · Cited by 77
Metadata
- Authors
- Zhai, N., & Ma, X.
- Venue
- Journal of Educational Computing Research
- Publisher
- SAGE Publications
- Published
- 2022-11-17
- Volume
- 61
- Issue
- 4
- Pages
- 875-900
- ISSN
- 0735-6331 (ISSN-L)
- 1541-4140
- OpenAlex
- W4309259993
- Cited by
- 115
- References
- 23
- FWCI
- 12.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.