Cited by 176Writing and Handwriting EducationStudent Assessment and FeedbackEducational Technology and Assessment

The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.

Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy

Wilson, Joshua, Roscoe, Rod D.·2020·Journal of Educational Computing Research

DOI 10.1177/0735633119830764

Open this paper in CoChat — free account

Abstract

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing ( n = 56) or a word-processing condition that used Google Docs ( n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.

Integrity signals

Registry

Crossref · 10.1177/0735633119830764

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Using multivariate statistics
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 79,882
Comparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266
Structural Equations with Latent Variables
Bollen, K. A. (1989) · Wiley series in probability and statistics · doi:10.1002/9781118619179 · Cited by 17,944
Confirmatory factor analysis for applied research
Brown, T. A. (2007) · Choice Reviews Online · doi:10.5860/choice.44-2769 · Cited by 15,621
Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge
Mishra, P., & Koehler, M. J. (2006) · Teachers College Record: The Voice of Scholarship in Education · doi:10.1111/j.1467-9620.2006.00684.x · Cited by 12,604
A Reliability Coefficient for Maximum Likelihood Factor Analysis
Tucker, L. R., & Lewis, C. (1973) · Psychometrika · doi:10.1007/bf02291170 · Cited by 7,330 · cited 1×
“To assess the ﬁt of the path models, we examined the chi-square test, the standardized root mean residual (SRMR), comparative ﬁt index (CFI; Bentler, 1990), the Tucker–Lewis index (TLI; Tucker & Lewis, 1973), and the root mean square error of approximation (RMSEA; Steiger & Lind, 1980).”
Method
SOCIAL VALIDITY: THE CASE FOR SUBJECTIVE MEASUREMENT or HOW APPLIED BEHAVIOR ANALYSIS IS FINDING ITS HEART 1
Wolf, M. M. (1978) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1978.11-203 · Cited by 3,176
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Statistically based tests for the number of common factors
Steiger, J. H. (1980) · Medical Entomology and Zoology · Cited by 2,413
Integrating technology into K-12 teaching and learning: current knowledge gaps and recommendations for future research
Hew, K. F., & Brush, T. A. (2006) · Educational Technology Research and Development · doi:10.1007/s11423-006-9022-5 · Cited by 2,373
Handbook of Writing Research
MacArthur, C. A., Graham, S. W., & Fitzgerald, J. (2005) · Cited by 2,327
Addressing first- and second-order barriers to change: Strategies for technology integration
Ertmer, P. A. (1999) · Educational Technology Research and Development · doi:10.1007/bf02299597 · Cited by 2,189
SELF-EFFICACY BELIEFS, MOTIVATION, AND ACHIEVEMENT IN WRITING: A REVIEW OF THE LITERATURE
Pajares, F. (2003) · Reading & Writing Quarterly · doi:10.1080/10573560308222 · Cited by 1,398
The Journal of Teacher Education
(1956) · Journal of Teacher Education · doi:10.1177/002248715600700102 · Cited by 1,201
Understanding the relationship between teachers’ pedagogical beliefs and technology use in education: a systematic review of qualitative evidence
Tondeur, J., van Braak, J., Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2016) · Educational Technology Research and Development · doi:10.1007/s11423-016-9481-2 · Cited by 1,146
Missing Data in Educational Research: A Review of Reporting Practices and Suggestions for Improvement
Peugh, J., & Enders, C. K. (2004) · Review of Educational Research · doi:10.3102/00346543074004525 · Cited by 1,107
Modeling and remodeling writing
Hayes, J. R. (2012) · Written Communication · doi:10.1177/0741088312451260 · Cited by 893
Training writing skills: a cognitive developmental perspective
Kellogg, R. T. (2008) · Journal of Writing Research · doi:10.17239/jowr-2008.01.01.1 · Cited by 868
A meta-analysis of writing instruction for students in the elementary grades.
Graham, S., McKeown, D., Kiuhara, S. A., & Harris, K. R. (2012) · Journal of Educational Psychology · doi:10.1037/a0029185 · Cited by 828
Teachers' perceptions of the barriers to technology integration and practices with technology under situated professional development
Kopcha, T. J. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.05.014 · Cited by 672
Untitled work
(2007) · Cited by 563
Handbook of Automated Essay Evaluation
(2013) · Routledge eBooks · doi:10.4324/9780203122761 · Cited by 518
Automated Essay Scoring
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860 · Cited by 512
Examining dimensions of self-efficacy for writing
Bruning, R., Dempsey, M., Kauffman, D. F., McKim, C., & Zumbrunn, S. (2013) · Journal of Educational Psychology · doi:10.1037/a0029692 · Cited by 470
Untitled work
(2003) · Cited by 449
Self-efficacy beliefs and the writing performance of entering high school students
Pajares, F., & Johnson, M. J. (1998) · Psychology in the Schools · doi:10.1002/(sici)1520-6807(199604)33:2 · Cited by 394
Teaching Writing to Elementary Students in Grades 4–6: A National Survey
Gilbert, J. K., & Graham, S. (2010) · The Elementary School Journal · doi:10.1086/651193 · Cited by 386
Sources of Writing Self-Efficacy Beliefs of Elementary, Middle, and High School Students
Pajares, F., Johnson, M. J., & Usher, E. L. (2007) · Research in the Teaching of English · doi:10.58680/rte20076485 · Cited by 373
Formative Assessment and Writing: A Meta-Analysis
Graham, S., Hebert, M., & Harris, K. R. (2015) · The Elementary School Journal · doi:10.1086/681947 · Cited by 340
Automated writing evaluation: Defining the classroom research agenda
Warschauer, M., & Ware, P. (2006) · Language Teaching Research · doi:10.1191/1362168806lr190oa · Cited by 337
The structural relationship between writing attitude and writing achievement in first and third grade students
Graham, S., Berninger, V. W., & Fan, W. (2007) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2007.01.002 · Cited by 326
Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality
Wilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
Self-citation
On the relation between automated essay scoring and modern views of the writing construct
Deane, P. (2012) · Assessing Writing · doi:10.1016/j.asw.2012.10.002 · Cited by 288
The effects of computer-generated feedback on the quality of writing
Stevenson, M., & Phakiti, A. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.11.007 · Cited by 275
Improving the writing skills of college students
Kellogg, R. T., & Raulerson, B. A. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194058 · Cited by 270
Training Advanced Writing Skills: The Case for Deliberate Practice
Kellogg, R. T., & Whiteford, A. P. (2009) · Educational Psychologist · doi:10.1080/00461520903213600 · Cited by 262
Automated Writing Assessment in the Classroom
Warschauer, M., & Grimes, D. (2008) · Pedagogies: An International Journal · doi:10.1080/15544800701771580 · Cited by 243
State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration
Shermis, M. D. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.04.001 · Cited by 227
Relationships between writing motivation, writing activity, and writing performance: effects of grade, sex, and ability
Troia, G. A., Harbaugh, A. G., Shankland, R. K., Wolbers, K. A., & Lawrence, A. M. (2012) · Reading and Writing · doi:10.1007/s11145-012-9379-2 · Cited by 215
Word processing programs and weaker writers/readers: a meta-analysis of research findings
Morphy, P., & Graham, S. (2011) · Reading and Writing · doi:10.1007/s11145-010-9292-5 · Cited by 214
Teaching writing to middle school students: a national survey
Graham, S., Capizzi, A. M., Harris, K. R., Hebert, M. A., & Morphy, P. (2013) · Reading and Writing · doi:10.1007/s11145-013-9495-7 · Cited by 211
Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback?
Dikli, S., & Bleyle, S. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.03.006 · Cited by 193
Do students' beliefs about writing relate to their writing self-efficacy, apprehension, and performance?
Sanders‐Reio, J., Alexander, P. A., Reio, T. G., & Newman, I. (2014) · Learning and Instruction · doi:10.1016/j.learninstruc.2014.02.001 · Cited by 183
Motivation Research in Writing: Theoretical and Empirical Considerations
Troia, G. A., Shankland, R. K., & Wolbers, K. A. (2012) · Reading & Writing Quarterly · doi:10.1080/10573569.2012.632729 · Cited by 174
Validity arguments for diagnostic assessment using automated writing evaluation
Chapelle, C. A., Cotos, E., & Lee, J. (2015) · Language Testing · doi:10.1177/0265532214565386 · Cited by 158
Teacher Feedback, Writing Assignment Quality, and Third-Grade Students' Revision in Lower-And Higher-Achieving Urban Schools
Matsumura, L. C., Patthey‐Chavez, G. G., Valdés, R., & Garnier, H. (2002) · The Elementary School Journal · doi:10.1086/499713 · Cited by 147
Automated scoring and feedback systems: Where are we and where are we heading?
Xi, X. (2010) · Language Testing · doi:10.1177/0265532210364643 · Cited by 144
Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom.
Roscoe, R. D., & McNamara, D. S. (2013) · Journal of Educational Psychology · doi:10.1037/a0032340 · Cited by 144
Self-citation
When “the state of the art” is counting words
Perelman, L. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.05.001 · Cited by 139
Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings?
Condon, W. (2012) · Assessing Writing · doi:10.1016/j.asw.2012.11.001 · Cited by 113
Teacher implementation of Self-Regulated Strategy Development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students
Palermo, C., & Thomson, M. M. (2018) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2018.07.002 · Cited by 107
Presentation, expectations, and experience: Sources of student perceptions of automated writing evaluation
Roscoe, R. D., Wilson, J., Johnson, A. C., & Mayra, C. R. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2016.12.076 · Cited by 98
Self-citation
Does Automated Feedback Help Students Learn to Write?
Kellogg, R. T., Whiteford, A. P., & Quinlan, T. (2010) · Journal of Educational Computing Research · doi:10.2190/ec.42.2.c · Cited by 88
A Critical Interpretative Synthesis: The Integration of Automated Writing Evaluation into Classroom Writing Instruction
Stevenson, M. (2016) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2016.05.001 · Cited by 85
Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance
Duijnhouwer, H., Prins, F. J., & Stokking, K. M. (2010) · Educational Research and Evaluation · doi:10.1080/13803611003711393 · Cited by 83
Research in Higher Education
Horowitz, M. (2013) · Research in Higher Education · doi:10.1007/11162.1573-188x · Cited by 76
Automated Scoring Technologies and the Rising Influence of Error
Cheville, J. (2004) · The English Journal · doi:10.2307/4128980 · Cited by 72
Trait Ratings for Automated Essay Grading
Shermis, M. D., Koch, C. M., Page, E. B., Keith, T. Z., & Harrington, S. (2002) · Educational and Psychological Measurement · doi:10.1177/0013164402062001001 · Cited by 67
IEEE Transactions on Learning Technologies
(2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3224333 · Cited by 57
Awaiting a new wave: The status of state writing assessment in the United States
Behizadeh, N., & Pang, M. E. (2016) · Assessing Writing · doi:10.1016/j.asw.2016.05.003 · Cited by 53
Student Use of Automated Essay Evaluation Technology During Revision
Moore, N. S., & MacArthur, C. A. (2016) · Journal of Writing Research · doi:10.17239/jowr-2016.08.01.05 · Cited by 51
CCCC Position Statement on Teaching, Learning, and Assessing Writing in Digital Environments
(2004) · College Composition and Communication · doi:10.58680/ccc20042788 · Cited by 46
Associated effects of automated essay evaluation software on growth in writing quality for students with and without disabilities
Wilson, J. (2016) · Reading and Writing · doi:10.1007/s11145-016-9695-z · Cited by 41
Self-citation
Utility and usability as factors influencing teacher decisions about software integration
Okumuş, S., Lewis, L., Wiebe, E., & Hollebrands, K. F. (2016) · Educational Technology Research and Development · doi:10.1007/s11423-016-9455-4 · Cited by 38
Automated Writing Analysis for Writing Pedagogy
Cotos, E. (2015) · Writing & Pedagogy · doi:10.1558/wap.v7i2-3.26381 · Cited by 37
Does Automated Feedback Improve Writing Quality
Wilson, J., Olinghouse, N. G., & Andrada, G. N. (2014) · Learning disabilities · Cited by 35
Self-citation
Social validity: Perceptions of check and connect with early literacy support
Lyst, A. M., Gabriel, S., O'Shaughnessy, T. E., Meyers, J. P., & Meyers, B. (2005) · Journal of School Psychology · doi:10.1016/j.jsp.2005.04.004 · Cited by 34
Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4
Wilson, J. (2018) · Journal of School Psychology · doi:10.1016/j.jsp.2017.12.005 · Cited by 34
Self-citation
Text-based writing of low-skilled postsecondary students: relation to comprehension, self-efficacy and teacher judgments
Perin, D., Lauterbach, M. D., Raufman, J., & Kalamkarian, H. S. (2016) · Reading and Writing · doi:10.1007/s11145-016-9706-0 · Cited by 33
Using Automated Feedback to Improve Writing Quality
Wilson, J., & Andrada, G. N. (2015) · Advances in higher education and professional development book series · doi:10.4018/978-1-4666-9441-5.ch026 · Cited by 31
Self-citation
On the vulnerability of automated scoring to construct-irrelevant response strategies (CIRS): An illustration
Bejar, I. I., Flor, M., Futagi, Y., & Ramineni, C. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.06.001 · Cited by 28
The Impact of Automated Essay Scoring on Writing Outcomes
Shermis, M. D., GARVAN, C. W., & Diao, Y. (2008) · Cited by 23
An Examination of How a Teacher's Use of Digital Tools Empowers and Constrains Language Arts Instruction
Hutchison, A. C., & Woodward, L. (2014) · Computers in the Schools · doi:10.1080/07380569.2014.967629 · Cited by 21
Automated Scoring in Assessment Systems
Bunch, M. B., Vaughn, D., & Miel, S. (2015) · Advances in higher education and professional development book series · doi:10.4018/978-1-4666-9441-5.ch023 · Cited by 17
Trait Ratings for Automated Essay Grading
Shermis, M. D., Koch, C. M., Page, E. B., et al. (2002) · Educational and Psychological Measurement · doi:10.1177/001316440206200101 · Cited by 14
Revealing the iceberg: Creative writing, process & deliberate practice
Monk, J. (2015) · English in Education · doi:10.1111/eie.12091 · Cited by 12

Research works printed but not linked: 12

Archer A. L., 2011, Explicit instruction: Effective and efficient teaching
Balfour S. P., 2013, Research & Practice in Assessment, 8, 40
Bennett R. E., 2004, Moving the field forward some thoughts on validity and automated scoring. Research Memorandum (RM-04-01)
Graham S., 2016, Teaching secondary students to write effectively
Grimes D., 2010, Journal of Technology, Learning, and Assessment, 8, 1
Keith T. Z., 2003, Automated essay scoring: A cross-disciplinary perspective, 147
Muthén L. K., Mplus user's guide
National Center for Education Statistics, 2012, The Nation's Report Card: Writing 2011
Page E. B., 2003, Automated essay scoring: A cross-disciplinary perspective, 43
Shermis M. D., 2016, Handbook of writing research, 395
Skalicky S., 2016, Advances in Language and Literacy Studies, 7, 248
Wilson J., 2014, Learning Disabilities: A Contemporary Journal, 12, 93

Sorted by citations

Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509
Application of Artificial Intelligence powered digital writing assistant in higher education: randomized controlled trial
Nazari, N., Shabbir, M. S., & Setiawan, R. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07014 · Cited by 405
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409
Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311
A systematic review of AI-based automated written feedback research
Shi, H., & Aryadoust, V. (2024) · ReCALL · doi:10.1017/s0958344023000265 · Cited by 222

See all 176

Authors: Wilson, J., & Roscoe, R. D.
Affiliations: University of Delaware (US)
Arizona State University (US)
Corresponding author: Wilson, Joshua
ORCID: Wilson, Joshua
Roscoe, Rod D.
Venue: Journal of Educational Computing Research
Publisher: SAGE Publications
Published: 2019-03-26
Volume: 58
Issue: 1
Pages: 87-125
ISSN: 0735-6331 (ISSN-L)
1541-4140
DOI: 10.1177/0735633119830764
OpenAlex: W2923646821
Cited by: 176
References: 75
Citations by year: 2026: 34 · 2025: 30 · 2024: 30 · 2023: 27 · 2022: 26 · 2021: 17 · 2020: 10 · 2019: 2
FWCI: 37.10
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Journal of Educational Computing Research (publishedVersion)
Topics: Social Sciences › Education › Writing and Handwriting Education
Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
Keywords: automated writing evaluation, writing self-efficacy, writing quality, Google Docs, writing instruction, teacher perceptions, social validity, junior high school students, writing feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks