The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Rater types in writing performance assessments: A classification approach to rater variability
Abstract
Research on rater effects in language performance assessments has provided ample evidence for a considerable degree of variability among raters. Building on this research, I advance the hypothesis that experienced raters fall into types or classes that are clearly distinguishable from one another with respect to the importance they attach to scoring criteria. To examine the rater type hypothesis, I asked 64 raters actively involved in scoring examinee writing performance on a large-scale assessment instrument to indicate on a four-point scale how much importance they would attach to each of nine routinely used criteria. The criteria covered various performance aspects, such as fluency, completeness, and grammatical correctness. In a preliminary step, many-facet Rasch analysis revealed that raters differed significantly in their views on the importance of the various criteria. A two-mode clustering technique yielded a joint classification of raters and criteria, with six rater types emerging from the analysis. Each of these types was characterized by a distinct scoring profile, indicating that raters were far from dividing their attention evenly among the set of criteria. Moreover, rater background variables were shown to partially account for the scoring profile differences. The findings have implications for assessing the quality of large-scale rater-mediated language testing, rater monitoring, and rater training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,120
Journal of the Royal Statistical Society.
(1923) · The Economic Journal · doi:10.1093/ej/33.130.269 · Cited by 3,219
Language Testing in Practice: Designing and Developing Useful Language Tests
Bachman, L. F., & Palmer, A. S. (1996) · Cited by 1,521
Optimizing rating scale category effectiveness.
Linacre, J. M. (2002) · PubMed · Cited by 1,319
Measuring Second Language Performance
Kleinsasser, R. C., & McNamara, T. (1998) · The Modern Language Journal · doi:10.2307/330236 · Cited by 1,253
Weigle, S. C. (2002) · doi:10.1017/cbo9780511732997 · Cited by 779
Language Testing and Validation
Weir, C. J. (2005) · Palgrave Macmillan UK eBooks · doi:10.1057/9780230514577 · Cited by 652
Detecting and measuring rater effects using many-facet Rasch measurement: Part II.
Myford, C. M., & Wolfe, E. W. (2003) · PubMed · Cited by 587
Language Testing and Validation: An Evidence-Based Approach
Weir, C. J. (2004) · Cited by 463
Using FACETS to model rater training effects
Weigle, S. C. (1998) · Language Testing · doi:10.1177/026553229801500205 · Cited by 367
Rater characteristics and rater bias: implications for training
Lumley, T., & McNamara, T. F. (1995) · Language Testing · doi:10.1177/026553229501200104 · Cited by 343
Assessment criteria in a large-scale writing test: what do they really mean to the raters?
Lumley, T. (2002) · Language Testing · doi:10.1191/0265532202lt230oa · Cited by 321
Verbal Protocol Analysis in Language Testing Research: A Handbook
Green, A. J. K. (1998) · Cited by 296
Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model
Engelhard, G. (1994) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1994.tb00436.x · Cited by 288
Expertise in evaluating second language compositions
Cumming, A. (1990) · Language Testing · doi:10.1177/026553229000700104 · Cited by 269
Decision Making while Rating ESL/EFL Writing Tasks: A Descriptive Framework
Cumming, A. H., Kantor, R. N., & Powers, D. E. (2002) · Modern Language Journal · doi:10.1111/1540-4781.00137 · Cited by 243
Fit analysis in latent trait measurement models.
Smith, R. M. (2000) · PubMed · Cited by 242
Effects of training on raters of ESL compositions
Weigle, S. C. (1994) · Language Testing · doi:10.1177/026553229401100206 · Cited by 208
Magnitude and moderators of bias in observer ratings: A meta-analysis.
Hoyt, W. T., & Kerns, M.-D. (1999) · Psychological Methods · doi:10.1037//1082-989x.4.4.403 · Cited by 196
Two-mode clustering methods: astructuredoverview
Van Mechelen, I., Bock, H.-H., & De Boeck, P. (2004) · Statistical Methods in Medical Research · doi:10.1191/0962280204sm373ra · Cited by 194
Generalizability of writing scores: an application of structural equation modeling
Schoonen, R. (2005) · Language Testing · doi:10.1191/0265532205lt295oa · Cited by 192
Bachman, L. F., Lynch, B. K., & Mason, M. (1995) · Language Testing · doi:10.1177/026553229501200206 · Cited by 190
Eckes, T. (2005) · Language Assessment Quarterly · doi:10.1207/s15434311laq0203_2 · Cited by 188
Self-citationLievens, F. (2001) · Journal of Applied Psychology · doi:10.1037/0021-9010.86.2.255 · Cited by 168
Item Response Theory and the validation of an ESP test for health professionals
McNamara, T. F. (1990) · Language Testing · doi:10.1177/026553229000700105 · Cited by 165
Assessing Second Language Writing: The Rater’s Perspective
Lumley, T. (2005) · Cited by 161
Weigle, S. C. (1999) · Assessing Writing · doi:10.1016/s1075-2935(00)00010-6 · Cited by 157
A Structural Theory For Intergroup Beliefs and Action
Guttman, L. (1959) · American Sociological Review · doi:10.2307/2089380 · Cited by 155
How characteristics of student essays influence teachers' evaluations.
Freedman, S. W. (1979) · Journal of Educational Psychology · doi:10.1037/0022-0663.71.3.328 · Cited by 138
Performance Assessment in Language Testing
Shohamy, E. (1995) · Annual Review of Applied Linguistics · doi:10.1017/s0267190500002683 · Cited by 115
Wolfe, E. W. (1997) · Assessing Writing · doi:10.1016/s1075-2935(97)80006-2 · Cited by 108
The Stability of Rater Severity in Large‐Scale Assessment Programs
Congdon, P. J., & MeQueen, J. (2000) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2000.tb01081.x · Cited by 106
Cognitive Differences in Proficient and Nonproficient Essay Scorers
Wolfe, E. W., KAO, C.-W., & Ranney, M. A. (1998) · Written Communication · doi:10.1177/0741088398015004002 · Cited by 94
Individual Feedback to Enhance Rater Training: Does It Work?
Elder, C., Knoch, U., Barkhuizen, G., & von Randow, J. (2005) · Language Assessment Quarterly · doi:10.1207/s15434311laq0203_1 · Cited by 88
Do English and ESL Faculties Rate Writing Samples Differently?
Brown, J. D. (1991) · TESOL Quarterly · doi:10.2307/3587078 · Cited by 82
Writing assessment: Raters' elaboration of the rating task
DeRemer, M. (1998) · Assessing Writing · doi:10.1016/s1075-2935(99)80003-8 · Cited by 76
A Model of Rater Behavior in Essay Grading Based on Signal Detection Theory
DeCarlo, L. T. (2005) · Journal of Educational Measurement · doi:10.1111/j.0022-0655.2005.00004.x · Cited by 68
Native and Nonnative Speakers' Pragmatic Interpretations of English Texts
Hinkel, E. (1994) · TESOL Quarterly · doi:10.2307/3587437 · Cited by 62
The assessment of writing ability: expert readers versus lay readers
Schoonen, R., Vergeer, M., & Eiting, M. H. (1997) · Language Testing · doi:10.1177/026553229701400203 · Cited by 55
Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions
Sakyi, A. (2000) · Dialnet (Universidad de la Rioja) · Cited by 52
The impact of training on rater variability
Barrett, S. (2001) · Cited by 41
Eckes, T., Ellis, M., Kalnbērziņa, V., Pižorn, K., Springer, C., Szollás, K., & Tsagari, C. (2005) · Language Testing · doi:10.1191/0265532205lt313oa · Cited by 28
Self-citationBeurteilerübereinstimmung und Beurteilerstrenge
Eckes, T. (2004) · Diagnostica · doi:10.1026/0012-1924.50.2.65 · Cited by 16
Self-citation
Cited by
Eurasian Journal of Educational Research
(2021) · Eurasian Journal of Educational Research · doi:10.14689/ejer · Cited by 638
Pack, A., Barrett, A., & Escalante, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100234 · Cited by 166
Can AI Provide Useful Holistic Essay Scoring?
Tate, T. P., Steiss, J., Bailey, D., Graham, S., Moon, Y., Ritchie, D., Tseng, W., & Warschauer, M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100255 · Cited by 60
Crossley, S. (2019) · Journal of Writing Research · doi:10.17239/jowr-2019.11.02.01 · Cited by 58
“The importance of grammatical and structural errors can be seen clearly in writing assessment literature which highlights the importance of these errors in explaining writing quality (Eckes, 2008; Santos, 1988; Zhu, 2003), teaching these rules to students (Graham, 1983; Morris, Blanton, L.”
BackgroundLi, J., & Huang, J. (2022) · Assessing Writing · doi:10.1016/j.asw.2021.100604 · Cited by 31
Metadata
- Authors
- Eckes, T.
- Venue
- Language Testing
- Publisher
- SAGE Publications
- Published
- 2008-04
- Volume
- 25
- Issue
- 2
- Pages
- 155-185
- ISSN
- 0265-5322 (ISSN-L)
- 1477-0946
- OpenAlex
- W1984484958
- Cited by
- 257
- References
- 43
- FWCI
- 15.89
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.