Methods for ordinal peer grading
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Massive Online Open Courses have the potential to revolutionize higher education with their wide outreach and accessibility, but they require instructors to come up with scalable alternates to traditional student evaluation. Peer grading -- having students assess each other -- is a promising approach to tackling the problem of evaluation at scale, since the number of "graders" naturally scales with the number of students. However, students are not trained in grading, which means that one cannot expect the same level of grading skills as in traditional settings. Drawing on broad evidence that ordinal feedback is easier to provide and more reliable than cardinal feedback [5, 38, 29, 9], it is therefore desirable to allow peer graders to make ordinal statements (e.g. "project X is better than project Y") and not require them to make cardinal statements (e.g. "project X is a B-"). Thus, in this paper we study the problem of automatically inferring student grades from ordinal peer feedback, as opposed to existing methods that require cardinal peer feedback. We formulate the ordinal peer grading problem as a type of rank aggregation problem, and explore several probabilistic models under which to estimate student grades and grader reliability. We study the applicability of these methods using peer grading data collected from a real class --- with instructor and TA grades as a baseline --- and demonstrate the efficacy of ordinal feedback techniques in comparison to existing cardinal peer grading methods. Finally, we compare these peer-grading techniques to traditional evaluation techniques.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434
Plackett, R. L. (1975) · Applied Statistics · doi:10.2307/2346567 · Cited by 731 · cited 4×
“An extension to the Bradley-Terry model (the Plackett-Luce model [33]) allows us to use distributions over rankings, while still retaining convexity and simplicity of gradient computation.”
MethodInfluentialThe method of paired comparisons for social values.
Thurstone, L. L. (1927) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0065439 · Cited by 467 · cited 6×
“Many popular methods used today [15, 25, 11] build on classical models and techniques such as the seminal work by Thurstone [39], Mallows [28], Bradley & Terry [8], Luce [27] and Plackett [33].”
MethodInfluentialPeer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · cited 1×
“In peer grading, students — not instructors or TAs — provide feedback on the work of other students in their class [14, 22], meaning that the number of “graders” naturally grows with the number of students.”
BackgroundTuned Models of Peer Assessment in MOOCs
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013) · arXiv · doi:10.48550/arxiv.1307.2579 · Cited by 262 · cited 5×
“While most previous uses of peer grading have relied on simple estimation techniques like averaging cardinal feedback scores, recently a probabilistic learning algorithms has been proposed for peer grade estimation [32].”
BackgroundMethodInfluentialFreeman, S., & Parks, J. W. (2010) · CBE—Life Sciences Education · doi:10.1187/cbe.10-03-0017 · Cited by 44 · cited 1×
“In peer grading, students — not instructors or TAs — provide feedback on the work of other students in their class [14, 22], meaning that the number of “graders” naturally grows with the number of students.”
Background
Cited by
Lan, A. S., Vats, D., Waters, A. E., & Baraniuk, R. G. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724664 · Cited by 60
“…multiple choice input and binary grading schemes (correct/incorrect), which are known to convey less information about the learners’ knowledge than open response questions [17], or peer-grading schemes [25, 26], which shift the burden of grading from the course instructor to the learners.”
BackgroundProbabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs
Mi, F., & Yeung, D.-Y. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9210 · Cited by 34
(2019) · Educational Data Mining · Cited by 7
Labutov, I., & Studer, C. (2016) · ETH Zurich · doi:10.3929/ethz-b-000460330
Metadata
- Authors
- Raman, K., & Joachims, T.
- Venue
- Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2014-08-24
- Pages
- 1037-1046
- OpenAlex
- W2166164009
- Cited by
- 109
- References
- 7
- FWCI
- 8.70
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Methods for ordinal peer grading. https://app.cochat.ai/papers/10.1145/2623330.2623654
- The paper
- Raman, K., & Joachims, T. (2014). Methods for ordinal peer grading. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. https://doi.org/10.1145/2623330.2623654
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.