The PeerRank Method for Peer Assessment
Open accessAbstract
We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show considerable promise, reducing the error in grade predictions by a factor of 2 or more in many cases over the natural baseline of averaging peer grades.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The first methods for aggregating peer grades and many more recent methods produce point estimates of each assignment's grade and each grader's quality.
“The first methods for aggregating peer grades-and many others introduced more recently-produce point estimates of each assignment's grade and each grader's quality (Walsh 2014;Chakraborty, Jindal, and Nath 2018;Prajapati et al. 2020;de Alfaro and Shavlovsky 2014;Hamer, Ma, and Kwong 2005).”
From Better Peer Grading through Bayesian Inference (2023) · cites this paper as Walsh 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- [1][1] N. Alon, F. Fischer, A. Procaccia, and M. Tennenholtz, 'Sum of us: Strategyproof selection from the selectors', in Proceedings of the 13th Conferene on Theoretical Aspects of Rationality and Knowledge , pp. 101-110, (2011). -
- [2][2] L. de Alfaro and M. Shavlovsky, 'Crowdgrader: Crowdsourcing the evaluation of homework assignments', Technical Report 1308.5273, arXiv.org, (August 2013). -
- [3][3] G. DeClippel, H. Moulin, and N. Tideman, 'Impartial division of a dollar', Journal of Economic Theory , 139 , 176-191, (2007). -
- [4][4] R. Holzman and H. Moulin, 'Impartial nominations for a prize', Econometrica , 81 (1), 173-196, (2013). -
- [5][5] P.M. Kerr, K.H. Park, and B.R. Domazlicky, 'Peer grading of essays in a principles of microeconomics course', Journal of Education for Business , 70 (6), 357-361, (1995). -
- [6][6] M.R. Merrifield and D.G. Saari, 'Telescope time without tears: a distributed approach to peer review', Astronomy & Geophysics , 50 (4), 4.16-4.20, (2009). -
- [7][7] P. Naghizadeh and M. Liu, 'Incentives, quality, and risks: A look into the NSF proposal review pilot', Technical Report 1307.6528, arXiv.org, (July 2013). -
- [9][9] Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, and Daphne Koller, 'Tuned models of peer assessment in MOOCs', in Proceedings of The 6th International Conference on Educational Data Mining (EDM 2013) , (2013). -
Cited by
Sorted by citations
Better Peer Grading through Bayesian Inference
Zarkoob, H., D'Eon, G., Podina, L., & Leyton-Brown, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25757 · Cited by 6 · see citing works
- Authors
- Walsh, T.
- Affiliations
- UNSW Sydney (AU)
- Corresponding author
- Walsh, Toby
- ORCID
- Walsh, Toby
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1405.7192
- OpenAlex
- W2950051380
- Cited by
- 36
- References
- 0
- Citations by year
- 2024: 3 · 2023: 1 · 2022: 1 · 2021: 3 · 2020: 3 · 2019: 3 · 2018: 3 · 2017: 4 · 2016: 9 · 2015: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Sociology and Political Science › Access Control and Trust
- Keywords
- peer assessment, PageRank, grade prediction, incentive compatibility, fixed point equations, axiomatic properties, multi-agent systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
