The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 1×
“The bene fi ts of feedback on improving writing have been widely reported in prior studies (Bakla, 2020; Hattie & Timperley, 2007; Huisman et al., 2019).”
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553
The Role of the Linguistic Environment in Second Language Acquisition
Long, M. H. (1996) · Handbook of Second Language Acquisition · doi:10.1016/b978-012589042-7/50015-3 · Cited by 3,927 · cited 2×
“The use of AWE may also be informed by the interaction hypothesis, which emphasizes the modi fi cation of linguistic inputs through various interaction forms such as clarifying, repeating, and expanding (Long, 1996).”
BackgroundMethodComplexity, accuracy and fluency in second language acquisition
Housen, A., & Kuiken, F. (2009) · Applied Linguistics · doi:10.1093/applin/amp048 · Cited by 863 · cited 3×
“Previous researchers held that constructs describing learners ’ language performance and pro fi ciency are multi-componential in nature, and the principal components of these constructs can be comprehensively and appropriately captured by the notions of complexity, accuracy, and fl uency (CAF) (Ellis & Ellis, 2008; Hous…”
BackgroundMethodPre-post effect sizes should be avoided in meta-analyses
Cuijpers, P., Weitz, E., Cristea, I. A., & Twisk, J. (2017) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796016000809 · Cited by 432
The Efficacy of Written Corrective Feedback in Improving L2 Written Accuracy: A Meta‐Analysis
Kang, E., & Han, Z. (2015) · The Modern Language Journal · doi:10.1111/modl.12189 · Cited by 385
Linguistic features in writing quality and development: An overview
Crossley, S. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.11.03.01 · Cited by 310
Huisman, B., Saab, N., van den Broek, P., & van Driel, J. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1545896 · Cited by 306
Cited by
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311
A systematic review of AI-based automated written feedback research
Shi, H., & Aryadoust, V. (2024) · ReCALL · doi:10.1017/s0958344023000265 · Cited by 222
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145
Shi, H., Chai, C. S., Zhou, S., & Aubrey, S. (2026) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2454541 · Cited by 109
ChatGPT for Automated Writing Evaluation in Scholarly Writing Instruction
Parker, J. L., Becker, K., & Carroca, C. (2023) · Journal of Nursing Education · doi:10.3928/01484834-20231006-02 · Cited by 77
Metadata
- Authors
- Zhai, N., & Ma, X.
- Venue
- Journal of Educational Computing Research
- Publisher
- SAGE Publications
- Published
- 2022-11-17
- Volume
- 61
- Issue
- 4
- Pages
- 875-900
- ISSN
- 0735-6331 (ISSN-L)
- 1541-4140
- OpenAlex
- W4309259993
- Cited by
- 115
- References
- 22
- FWCI
- 12.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis. https://app.cochat.ai/papers/10.1177/07356331221127300
- The paper
- Zhai, N., & Ma, X. (2023). The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis. Journal of Educational Computing Research. https://doi.org/10.1177/07356331221127300
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.