The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Practical Peer Prediction for Peer Assessment
Open accessAbstract
We provide an empirical analysis of peer prediction mechanisms, which reward participants for information in settings when there is no ground truth against which to score reports. We simulate the mechanisms on a dataset of three million peer assessments from the edX MOOC platform. We evaluate different mechanisms on score variability, which is connected to fairness, risk aversion, and participant learning. We also assess the magnitude of the incentives to invest effort, and study the effect of participant coordination on low-information signals. We find that the correlated agreement mechanism has lower variation in reward than other mechanisms. A concern is that the gain from exerting effort is relatively low across all mechanisms, due to frequent disagreement between peers. Our conclusions are relevant for crowdsourcing in education as well as other domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Peer assessment scales well to large class sizes.
“Not only does it scale well to large class sizes, such as those in massive open online classes (MOOCs) (Shnayder & Parkes, 2016)”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Shnayder & Parkes, 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · cited 4×
“Figure 2 shows the full distribution of the number of prompts per course.”
BackgroundInfluential
Flagged first, then by citations
Research works printed but not linked: 24
- Busemeyer, J. R., and Townsend, J. T. 1993. Decision Field Theory: A Dynamic-Cognitive Approach to Decision Making in an Uncertain Environment. Psychological review 100(3):432-459.
- Cho, K.; Schunn, C. D.; and Wilson, R. W. 2006. Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology 98(4):891-901.
- Dasgupta, A., and Ghosh, A. 2013. Crowdsourced Judge- ment Elicitation with Endogenous Proficiency. In Proceed- ings of the 22nd International Conference on World Wide Web (WWW'13), 1-17.
- Erev, I., and Barron, G. 2005. On adaptation, maximiza- tion, and reinforcement learning among cognitive strategies. Psychological review 112(4):912-931.
- Erev, I., and Roth, A. E. 2014. Maximization, learning, and economic behavior. Proceedings of the National Academy of Science (PNAS) 111:10818-10825.
- Falchikov, N. 1995. Peer feedback marking: Developing peer assessment. Innovations in Education & Training In- ternational 32(2):175-187.
- Faltings, B.; Pu, P.; and Tran, B. D. 2014. Incentives to Counter Bias in Human Computation. In The Second AAAI Conference on Human Computation & Crowdsourc- ing (HCOMP 2014), 59-66.
- Gao, X. A.; Mao, A.; Chen, Y.; and Adams, R. P. 2014. Trick or Treat : Putting Peer Prediction to the Test. In The Fifteenth ACM Conference on Economics and Computation (EC'14).
- Gao, X. A.; Wright, R. J.; and Leyton-Brown, K. 2016. In- centivizing Evaluation via Limited Access to Ground Truth: Peer Prediction Makes Things Worse. 2nd Workshop on Al- gorithmic Game Theory and Data Science at EC 2016.
- Goldfinch, J., and Raeside, R. 1990. Development of a peer assessment technique for obtaining individual marks on a group project. Assessment & Evaluation in Higher Educa- tion 15(3):210-231.
- Jurca, R., and Faltings, B. 2009. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research 34(1):209-253.
- Kamble, V.; Shah, N.; Marn, D.; Parekh, A.; and Ramachan- dran, K. 2015. Truth Serums for Massively Crowdsourced Evaluation Tasks. arXiv:1507.07045. · arXiv
- Kong, Y., and Schoenebeck, G. 2016. A Framework For De- signing Information Elicitation Mechanism That Rewards Truth-telling. arXiv:1605.01021. · arXiv
- Kulkarni, C.; Bernstein, M.; and Klemmer, S. 2015. Peer- Studio: Rapid Peer Feedback Emphasizes Revision and Im- proves Performance. The Proceedings of the Second ACM Conference on Learning @ Scale.
- Piech, C.; Huang, J.; Chen, Z.; Do, C.; Ng, A.; and Koller, D. 2013. Tuned Models of Peer Assessment in MOOCs. Proceedings of the 6th International Conference on Educa- tional Data Mining, Memphis, Tennessee.
- Prelec, D. 2004. A Bayesian Truth Serum For Subjective Data. Science 306(5695):462.
- Radanovic, G., and Faltings, B. 2013. A Robust Bayesian Truth Serum for Non-Binary Signals. In The Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI-13), 833- 839.
- Radanovic, G.; Faltings, B.; and Jurca, R. 2016. Incentives for Effort in Crowdsourcing using the Peer Truth Serum. ACM Transactions on Intelligent Systems and Technology (January).
- Russell, A. A. 2004. Calibrated peer review-a writing and critical-thinking instructional tool. Teaching Tips: Innova- tions in Undergraduate Science Instruction 54.
- Shnayder, V.; Agarwal, A.; Frongillo, R.; and Parkes, D. C. 2016. Informed truthfulness in multi-task peer prediction. The Seventeenth ACM Conference on Economics and Com- putation (EC'16).
- Shnayder, V.; Frongillo, R.; and Parkes, D. C. 2016. Measur- ing performance of peer prediction mechanisms using repli- cator dynamics. The 25th International Joint Conference on Artificial Intelligence (IJCAI-16).
- von Ahn, L., and Dabbish, L. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '04, 319- 326. New York, NY, USA: ACM.
- Witkowski, J., and Parkes, D. C. 2012a. A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012).
- Witkowski, J., and Parkes, D. C. 2012b. Peer Prediction Without a Common Prior. In Proceedings of the 13th ACM Conference on Electronic Commerce (EC2012).
Cited by
Sorted by citations
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409
Evaluating the Quality of Learning Resources: A Learnersourcing Approach
Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2021.3058644 · Cited by 40
Metadata
- Authors
- Shnayder, V., & Parkes, D.
- Affiliations
- Harvard University (US)
- ORCID
- Parkes, David
- Venue
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Published
- 2016-09-21
- Volume
- 4
- Pages
- 199-208
- ISSN
- 2769-1330 (ISSN-L)
- 2769-1349
- OpenAlex
- W2601131632
- Cited by
- 17
- References
- 1
- Citations by year
- 2025: 3 · 2023: 4 · 2022: 4 · 2021: 2 · 2020: 2 · 2018: 1 · 2017: 1
- FWCI
- 0.53
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Science Foundation (1301976)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Keywords
- peer prediction, peer assessment, incentive mechanisms, score variability, fairness, risk aversion, effort incentives, MOOCs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
