The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy
Abstract
The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing ( n = 56) or a word-processing condition that used Google Docs ( n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several studies have demonstrated that automated essay scoring tools like MI Write can help students to improve their writing motivation (Wilson & Czik, 2016), writing self-efficacy (Wilson & Roscoe, 2020) and writing skills (Palermo & Thomson, 2018)”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Wilson & Roscoe, 2020
Citations
Cites
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 79,882
Comparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266
Structural Equations with Latent Variables
Bollen, K. A. (1989) · Wiley series in probability and statistics · doi:10.1002/9781118619179 · Cited by 17,944
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,621
Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge
Mishra, P., & Koehler, M. J. (2006) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1111/j.1467-9620.2006.00684.x · Cited by 12,604
A Reliability Coefficient for Maximum Likelihood Factor Analysis
Tucker, L. R., & Lewis, C. (1973) · Psychometrika · doi:10.1007/bf02291170 · Cited by 7,330 · cited 1×
“To assess the fit of the path models, we examined the chi-square test, the standardized root mean residual (SRMR), comparative fit index (CFI; Bentler, 1990), the Tucker–Lewis index (TLI; Tucker & Lewis, 1973), and the root mean square error of approximation (RMSEA; Steiger & Lind, 1980).”
MethodWolf, M. M. (1978) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1978.11-203 · Cited by 3,176
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Statistically based tests for the number of common factors
Steiger, J. H. (1980) · Medical Entomology and Zoology · Cited by 2,413
Hew, K. F., & Brush, T. A. (2006) · Educational Technology Research and Development · doi:10.1007/s11423-006-9022-5 · Cited by 2,373
Research works printed but not linked: 12
- Archer A. L., 2011, Explicit instruction: Effective and efficient teaching
- Balfour S. P., 2013, Research & Practice in Assessment, 8, 40
- Bennett R. E., 2004, Moving the field forward some thoughts on validity and automated scoring. Research Memorandum (RM-04-01)
- Graham S., 2016, Teaching secondary students to write effectively
- Grimes D., 2010, Journal of Technology, Learning, and Assessment, 8, 1
- Keith T. Z., 2003, Automated essay scoring: A cross-disciplinary perspective, 147
- Muthén L. K., Mplus user's guide
- National Center for Education Statistics, 2012, The Nation's Report Card: Writing 2011
- Page E. B., 2003, Automated essay scoring: A cross-disciplinary perspective, 43
- Shermis M. D., 2016, Handbook of writing research, 395
- Skalicky S., 2016, Advances in Language and Literacy Studies, 7, 248
- Wilson J., 2014, Learning Disabilities: A Contemporary Journal, 12, 93
Cited by
Sorted by citations
Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509
Nazari, N., Shabbir, M. S., & Setiawan, R. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07014 · Cited by 405
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311
A systematic review of AI-based automated written feedback research
Shi, H., & Aryadoust, V. (2024) · ReCALL · doi:10.1017/s0958344023000265 · Cited by 222
Metadata
- Authors
- Wilson, J., & Roscoe, R. D.
- Affiliations
- University of Delaware (US)
- Arizona State University (US)
- Corresponding author
- Wilson, Joshua
- Venue
- Journal of Educational Computing Research
- Publisher
- SAGE Publications
- Published
- 2019-03-26
- Volume
- 58
- Issue
- 1
- Pages
- 87-125
- ISSN
- 0735-6331 (ISSN-L)
- 1541-4140
- OpenAlex
- W2923646821
- Cited by
- 176
- References
- 75
- Citations by year
- 2026: 34 · 2025: 30 · 2024: 30 · 2023: 27 · 2022: 26 · 2021: 17 · 2020: 10 · 2019: 2
- FWCI
- 37.10
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Computing Research (publishedVersion)
- Topics
- Social Sciences › Education › Writing and Handwriting Education
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- automated writing evaluation, writing self-efficacy, writing quality, Google Docs, writing instruction, teacher perceptions, social validity, junior high school students, writing feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
