The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Rater types in writing performance assessments: A classification approach to rater variability
Abstract
Research on rater effects in language performance assessments has provided ample evidence for a considerable degree of variability among raters. Building on this research, I advance the hypothesis that experienced raters fall into types or classes that are clearly distinguishable from one another with respect to the importance they attach to scoring criteria. To examine the rater type hypothesis, I asked 64 raters actively involved in scoring examinee writing performance on a large-scale assessment instrument to indicate on a four-point scale how much importance they would attach to each of nine routinely used criteria. The criteria covered various performance aspects, such as fluency, completeness, and grammatical correctness. In a preliminary step, many-facet Rasch analysis revealed that raters differed significantly in their views on the importance of the various criteria. A two-mode clustering technique yielded a joint classification of raters and criteria, with six rater types emerging from the analysis. Each of these types was characterized by a distinct scoring profile, indicating that raters were far from dividing their attention evenly among the set of criteria. Moreover, rater background variables were shown to partially account for the scoring profile differences. The findings have implications for assessing the quality of large-scale rater-mediated language testing, rater monitoring, and rater training.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Common European Framework of Reference for Languages: learning, teaching, assessment
Verhelst, N. D., Avermaet, P. V., Takala, S., Figueras, N., & North, B. (2009) · Ghent University Academic Bibliography (Ghent University) · Cited by 6,085
Journal of the Royal Statistical Society.
(1923) · The Economic Journal · doi:10.1093/ej/33.130.269 · Cited by 3,219
Language Testing in Practice: Designing and Developing Useful Language Tests
Bachman, L. F., & Palmer, A. S. (1996) · Cited by 1,521
Optimizing rating scale category effectiveness.
Linacre, J. M. (2002) · PubMed · Cited by 1,319
Measuring Second Language Performance
Kleinsasser, R. C., & McNamara, T. (1998) · The Modern Language Journal · doi:10.2307/330236 · Cited by 1,253
Weigle, S. C. (2002) · doi:10.1017/cbo9780511732997 · Cited by 779
Language Testing and Validation
Weir, C. J. (2005) · Palgrave Macmillan UK eBooks · doi:10.1057/9780230514577 · Cited by 652
Language Testing and Validation: An Evidence-Based Approach
Weir, C. J. (2004) · Cited by 463
Detecting and measuring rater effects using many-facet Rasch measurement: Part II.
Myford, C. M., & Wolfe, E. W. (2003) · PubMed · Cited by 385
Using FACETS to model rater training effects
Weigle, S. C. (1998) · Language Testing · doi:10.1177/026553229801500205 · Cited by 367
Cited by
Eurasian Journal of Educational Research
(2021) · Eurasian Journal of Educational Research · doi:10.14689/ejer · Cited by 638
Pack, A., Barrett, A., & Escalante, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100234 · Cited by 166
Crossley, S. (2019) · Journal of Writing Research · doi:10.17239/jowr-2019.11.02.01 · Cited by 58
“The importance of grammatical and structural errors can be seen clearly in writing assessment literature which highlights the importance of these errors in explaining writing quality (Eckes, 2008; Santos, 1988; Zhu, 2003), teaching these rules to students (Graham, 1983; Morris, Blanton, L.”
BackgroundCan AI Provide Useful Holistic Essay Scoring?
Tate, T. P., Steiss, J., Bailey, D., Graham, S., Moon, Y., Ritchie, D., Tseng, W., & Warschauer, M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100255 · Cited by 60
Li, J., & Huang, J. (2022) · Assessing Writing · doi:10.1016/j.asw.2021.100604 · Cited by 31
Metadata
- Authors
- Eckes, T.
- Venue
- Language Testing
- Publisher
- SAGE Publications
- Published
- 2008-04
- Volume
- 25
- Issue
- 2
- Pages
- 155-185
- ISSN
- 0265-5322 (ISSN-L)
- 1477-0946
- OpenAlex
- W1984484958
- Cited by
- 257
- References
- 43
- FWCI
- 15.89
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Rater types in writing performance assessments: A classification approach to rater variability. https://app.cochat.ai/papers/10.1177/0265532207086780
- The paper
- Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing. https://doi.org/10.1177/0265532207086780
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.