Tuned Models of Peer Assessment in MOOCs
Open accessAbstract
In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Piech et al. (2013) introduced the first system using graphical models to simultaneously determine distributions over both grades for student submission and accuracy assessments for each grader.
“A seminal paper due to Piech et al. (2013) introduced the first such system, using graphical models to simultaneously determine distributions over both grades for student submission and accuracy assessments for each grader.”
From Better Peer Grading through Bayesian Inference (2023) · cites this paper as Piech et al. (2013)
Section 3.2 (p. 4) says the models infer a joint posterior distribution over submission scores and each grader’s bias and reliability, supporting the substance of the claim. But the paper does not claim to be the first such system; its Related Work section (p. 7) describes earlier Bayesian models for peer grading. The priority claim is therefore unsupported. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe PG1 model assumes that each assignment has a latent true grade and each peer grader has a latent bias and reliability.
“Their core”
From Better Peer Grading through Bayesian Inference (2023) · cites this paper as Piech et al. (2013)
These parameters can be estimated from grading data through Bayesian inference, producing posterior distributions on each assignment's grade and each student's grading abilities.
“Their core”
From Better Peer Grading through Bayesian Inference (2023) · cites this paper as Piech et al. (2013)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Impact of Self- and Peer-Grading on Student Learning
Sadler, P., & Good, E. (2006) · Educational Assessment · doi:10.1207/s15326977ea1101_1 · Cited by 535 · cited 1× · see citing works
“Peer assessment — which has been historically used for logistical, pedagogical, metacognitive, and affective benefits [18] — offers a promising solution that can scale the grading of complex assignments in courses with tens or even hundreds of thousands of students.”
Background
Flagged first, then by citations
Research works printed but not linked: 14
- [1][1] K. Ashley and I. Goldin. Toward ai-enhanced computer-supported peer review in legal education. In 24th International Conference on Legal Knowledge and Information Systems (JURIX) , volume 235, 2011. -
- [2][2] Y. Bachrach, T. Minka, J. Guiver, and T. Graepel. How To Grade a Test Without Knowing the Answers A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing. In The 29th Annual International Conference on Machine Learning , ICML '12, 2012. -
- [5][5] L. Charlin, R. Zemel, and C. Boutilier. A framework for optimizing paper matching. In Proceedings of Uncertainty in Artificial Intelligence (UAI'11) , 2011. -
- [6][6] S. Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE PAMI , (6):721-741, 1984. -
- [7][7] I. Goldin. Accounting for peer reviewer bias with bayesian models. In Proceedings of the Workshop on Intelligent Support for Learning Groups at the 11th International Conference on Intelligent Tutoring Systems , 2012. -
- [8][8] I. M. Goldin and K. D. Ashley. Peering inside peer review with bayesian models. In Proceedings of the 15th international conference on Artificial intelligence in education , AIED'11, pages 90-97, Berlin, Heidelberg, 2011. Springer-Verlag. -
- [9][9] V. E. Johnson. On bayesian analysis of multi-rater ordinal data: An application to automated essay grading. Journal of the American Statistical Association , 91:42-51, 1996. -
- [11][11] G. Karabatsos and W. Batchelder. Markov chain estimation for test theory without an answer key. Psychometrika , 68(3):373-389, 2003. -
- [12][12] C. Kulkarni, K. Pang-Wei, H. Le, D. Chia, K. Papadopoulos, D. Koller, and S. R. Klemmer. Scaling self and peer assessment to the global design classroom. In Proceedings of CHI'13 (to appear) , 2013. -
- [13][13] R. J. Mislevy, R. G. Almond, D. Yan, and L. S. Steinberg. Bayes nets in educational assessment: Where the numbers come from. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence , pages 437-446. Morgan Kaufmann Publishers Inc., 1999. -
- [14][14] F. ˚ A. Nielsen. A new anew: Evaluation of a word list for sentiment analysis in microblogs. arXiv preprint arXiv:1103.2903 , 2011. - · arXiv
- [15][15] S. Rogers, M. Girolami, and T. Polajnar. Semi-parametric analysis of multi-rater data. Statistics and Computing , 20(3):317-334, July 2010. -
- [16][16] A. A. Russell. Calibrated peer review - a writing and critical-thinking instructional tool. Teaching Tips: Innovations in Undergraduate Science Instruction , page 54, 2004. -
- [18][18] J. Whitehill, P. Ruvolo, T. fan Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 , pages 2035-2043. MIT Press, 2009.
Cited by
Sorted by citations
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
Why do Institutions Offer MOOCs?
Hollands, F. M., & Tirthali, D. (2014) · Online Learning · doi:10.24059/olj.v18i3.464 · Cited by 101 · see citing works
Using learning analytics to understand student perceptions of peer feedback
Misiejuk, K., Wasson, B., & Egelandsdal, K. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106658 · Cited by 98 · see citing works
MOOCs Meet Measurement Theory: A Topic-Modelling Approach
He, J., Rubinstein, B., Bailey, J., Zhang, R., Milligan, S., & Chan, J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10151 · Cited by 10 · see citing works
Better Peer Grading through Bayesian Inference
Zarkoob, H., D'Eon, G., Podina, L., & Leyton-Brown, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25757 · Cited by 6 · see citing works
- Authors
- Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2013
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1307.2579
- OpenAlex
- W2122588206
- Cited by
- 262
- References
- 1
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 16 · 2023: 10 · 2022: 10 · 2021: 12 · 2020: 29 · 2019: 16 · 2018: 21 · 2017: 26 · 2016: 42 · 2015: 40 · 2014: 34 · 2013: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- peer assessment, peer grading, MOOCs, grading accuracy, student engagement, open-ended tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
