Better Peer Grading through Bayesian Inference
Open accessAbstract
Peer grading systems aggregate noisy reports from multiple students to approximate a "true" grade as closely as possible. Most current systems either take the mean or median of reported grades; others aim to estimate students’ grading accuracy under a probabilistic model. This paper extends the state of the art in the latter approach in three key ways: (1) recognizing that students can behave strategically (e.g., reporting grades close to the class average without doing the work); (2) appropriately handling censored data that arises from discrete-valued grading rubrics; and (3) using mixed integer programming to improve the interpretability of the grades assigned to students. We demonstrate how to make Bayesian inference practical in this model and evaluate our approach on both synthetic and real-world data obtained by using our implemented system in four large classes. These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper extends Bayesian peer-grading models by adding latent grader effort, treating discrete rubric reports as censored observations, and using mixed-integer programming to produce interpretable discrete final grades. Experiments on four large-class datasets and synthetic data show improved held-out likelihood, accurate grade/reliability recovery with enough data, and MIP grades that rarely differ from MAP estimates.
Key findings
- A model combining effort and censoring had the highest held-out likelihood; ablations showed removing censoring always caused significant drops, while removing effort had significant effects in one class. Model changes altered final grades by an average of 0.05–0.15 points on a 5-point scale and very rarely exceeded a single point.
- Replacing MAP grades with MIP output changed only 6.5% of grades across the four offerings, using defaults S = 0.09 and minimum non-zero weight T = 0.1.
- On synthetic data, reliability estimates reached a Spearman correlation of 0.6 after one week and 0.9 after eight weeks; effort estimates were harder, rising from 0.3 to 0.7. True-grade MAE fell from 0.52 with one grader to 0.41 with four graders—below a TA with reliability 2 (MAE 0.48)—and reached below 0.2 with 32 graders.
- Robustness tests found many hyperparameter misspecifications had small, statistically insignificant effects, but incorrect true-grade prior mean or standard deviation increased true-grade MAE from 0.44 to as high as 0.53; incorrect low-effort precision hurt effort estimates; and a bias prior standard deviation far below its true value hurt bias estimates.
Methods
- The paper extends the PG1 graphical model with latent effort decisions, effort probabilities, censored discrete grade reports, and a MIP formulation for explainable weighted-average grades. Inference uses Gibbs sampling, with marginalization and discrete grid approximations for censored observations.
- Real data came from four offerings of an undergraduate computer science course between September 2018 and December 2021: approximately 120 students, 11 weekly essays, four rubric components graded 0–5, 6,088–7,068 peer grades per dataset, 3–5 TAs who spot-checked 474–644 essays, and 60–84 gold-standard calibration submissions. Evaluation used 10-fold stratified cross-validation, held-out likelihood, and paired t-tests.
- Synthetic experiments simulated 10 weeks, 120 students, four peer grades per submission, three TAs grading 25% of submissions, and TAs with mean reliability 2; results report means and 95% confidence intervals across 15 simulated datasets, using MAE and Spearman rank-order correlation.
Limitations
- The paper states that the model has 9 hyperparameters, making tuning daunting for instructors, though it reports robustness to misspecifying reliability, bias, and effort priors. It also states that effort probabilities are difficult to estimate, converge more slowly than reliabilities, and are worsened by coarse rubrics where high-effort grades can match the class average.
- The paper states that real-world datasets provide no ground-truth values for true grades or reliabilities, so real-data evaluation relies on held-out likelihood rather than parameter recovery. Synthetic experiments assume data generated from the best-fitting models and hyperparameters.
- Evident from the text: validation is limited to four offerings of one undergraduate course and one grading rubric, and the MIP evaluation uses default constants rather than a broad sensitivity analysis over those constants.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
8 of 18 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured
- Premisecitation
The first methods for aggregating peer grades and many more recent methods produce point estimates of each assignment's grade and each grader's quality.
“The first methods for aggregating peer grades-and many others introduced more recently-produce point estimates of each assignment's grade and each grader's quality (Walsh 2014;Chakraborty, Jindal, and Nath 2018;Prajapati et al. 2020;de Alfaro and Shavlovsky 2014;Hamer, Ma, and Kwong 2005).”
Quote in the paperSubstantial claim1 of 3 sources checked - Premisecitation
These parameters can be estimated from grading data through Bayesian inference, producing posterior distributions on each assignment's grade and each student's grading abilities.
“Their core”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Piech et al. (2013) source
- Premisecitation
Piech et al. (2013) introduced the first system using graphical models to simultaneously determine distributions over both grades for student submission and accuracy assessments for each grader.
“A seminal paper due to Piech et al. (2013) introduced the first such system, using graphical models to simultaneously determine distributions over both grades for student submission and accuracy assessments for each grader.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Piech et al. (2013) source
Section 3.2 (p. 4) says the models infer a joint posterior distribution over submission scores and each grader’s bias and reliability, supporting the substance of the claim. But the paper does not claim to be the first such system; its Related Work section (p. 7) describes earlier Bayesian models for peer grading. The priority claim is therefore unsupported. (LLM, conf=0.87)
Experimental — accuracy not yet measured
- Premisecitation
Hartline et al. (2020) showed that the most robust 'low effort' strategy is to report the class average, because it minimizes the expected distance to an effortful report.
“Hartline et al. (2020)showed that the most robust”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Hartline et al. (2020) source
Section 6, “Application to Peer Grading” (page unavailable), says that when a peer does not read a submission, her optimal strategy is to report her prior belief—the class average mark for each criterion. It does not say this is the “most robust” strategy or explain its optimality as minimizing expected distance to an effortful report. (LLM, conf=0.88)
Experimental — accuracy not yet measured
- Premisecitation
Burrell and Schoenebeck (2021) found that rewards from out-of-the-box peer prediction mechanisms do not accurately reflect grader effort levels on realistic simulated data.
“Further, Burrell and Schoenebeck (2021) found that rewards from out-of-the-box peer prediction mechanisms do not accurately reflect grader effort levels on realistic simulated data.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Burrell and Schoenebeck (2021) source
- Findingown data
The effort and censoring model extensions improved likelihood on held-out data, using real data from four offerings of a large class.
“First, using real data from four offerings of a large class, we show that each of our effort and censoring model extensions improved likelihood on held-out data”
Quote in the paperSubstantial claim - Findingown data
Grade aggregation using the paper's model accurately estimates students' likelihood of submitting uninformative grades.
“These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.”
Quote in the paperSubstantial claim- Rests on an assumption: Students can behave strategically (e.g., reporting grades close to the class average without doing the work, or entering insincere peer grades unrelated to a submission's quality if they can get away with it).
- Findingreasoning
True grade estimates improved very little as the number of weeks grew.
“Perhaps surprisingly, true grade estimates improved very little as the number of weeks grew.”
Quote in the paperSubstantial claim - Findingown data
Grade aggregation using the paper's model accurately estimates the variation in students' inherent grading error.
“These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.”
Quote in the paperSubstantial claim- Rests on an assumption: Individual peer graders are biased (consistently give generous or harsh grades).
- Rests on an assumption: Individual peer graders are noisy (the same grader could grade an assignment differently on different days).
- Findingown data
Grade aggregation using the paper's model accurately estimates true grades.
“These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.”
Quote in the paperSubstantial claim- Rests on an assumption: Students can behave strategically (e.g., reporting grades close to the class average without doing the work, or entering insincere peer grades unrelated to a submission's quality if they can get away with it).
- Rests on an assumption: Censored data arises from discrete-valued grading rubrics.
- Findingown data
Some models produced inaccurate reliability and effort estimates.
“We found that some models produced inaccurate reliability and effort estimates”
Quote in the paperNot a substantial claim - Findingown data
Removing each feature degraded the model's held-out likelihood.
“Our results, shown in Figure 3, indicate that removing each feature degraded the model's held-out likelihood”
Quote in the paperSubstantial claim- Rests on an assumption: The TAs were diligent and responsible.
- Rests on an assumption: The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
- Findingown data
Adding more graders decreased true grade MAE far lower, reaching below 0.2 with 32 graders.
“Adding more graders decreased true grade MAEs far lower, reaching below 0.2 with 32 graders.”
Quote in the paperSubstantial claim - Findingown data
Across the four offerings, replacing MAP grades with MIP output would have caused only 6.5 percent of grades to change.
“Across the four offerings, replacing the MAP grades with the MIP's output would have caused only 6.5 percent of grades to change.”
Quote in the paperNot a substantial claim - Findingown data
The explainable grades output by our MIP rarely differed from the (non-explainable) MAP estimates.
“and that the explainable grades output by our MIP rarely differed from the (non-explainable) MAP estimates.”
Quote in the paperSubstantial claim - Findingown data
With additional graders, the MIP gained the flexibility to recover MAP grades, reaching nearly equal MAEs at 7 graders per submission.
“However, with additional graders, the MIP gained the flexibility to recover MAP grades, reaching nearly equal MAEs at 7 graders per submission.”
Quote in the paperNot a substantial claim - Findingown data
Effort had significant effects in one class.
“and efforts having significant effects in one class”
Quote in the paperNot a substantial claim- Rests on an assumption: The TAs were diligent and responsible.
- Rests on an assumption: The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
- Findingown data
The model's posterior beliefs were robust to misspecifying the reliability, bias, and effort priors.
“Luckily, our robustness experiments in Section 5.2 showed that the model's posterior beliefs were robust to misspecifying the reliability, bias, and effort priors.”
Quote in the paperSubstantial claim - Findingown data
Bias estimates improved with additional data.
“Bias estimates, given in the appendix, also improved with additional data.”
Quote in the paperNot a substantial claim - Findingown data
Using an incorrect mean or standard deviation for the true grade prior substantially increased true grade MAE from 0.44 to as high as 0.53.
“using an incorrect mean or standard deviation for the true grade prior substantially increased true grade MAE from 0.44 to as high as 0.53”
Quote in the paperSubstantial claim - Findingown data
Many changes to the hyperparameters had small and statistically insignificant effects on the inference results.
“Overall, we found that many of these changes to the hyperparameters had small and statistically insignificant effects on the inference results; these complete results are shown in Appendix G.”
Quote in the paperNot a substantial claim - Findingown data
The model using both effort and censoring features had the highest held-out likelihood compared to models with one or neither feature.
“We found that the model using both features had the highest held-out likelihood.”
Quote in the paperSubstantial claim- Rests on an assumption: The TAs were diligent and responsible.
- Rests on an assumption: The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
- Findingown data
Incorrectly specifying the low effort distribution τl was very detrimental to the effort probability estimates.
“incorrectly specifying the low effort distribution τ ℓ was very detrimental to the effort probability estimates”
Quote in the paperNot a substantial claim - Findingown data
The model's performance was quite robust to changes in the reliability and effort probability priors.
“Notably, we found that the model's performance was quite robust to changes in the reliability and effort probability priors.”
Quote in the paperNot a substantial claim - Findingown data
Our model extensions (grader effort and censored observations) improved fit.
“Overall, our experiments show that our model extensions (grader effort and censored observations) improved fit”
Quote in the paperSubstantial claim - Findingown data
Using a bias prior with a standard deviation far below its true value hurt bias estimates.
“using a bias prior with a standard deviation far below its true value hurt bias estimates”
Quote in the paperNot a substantial claim - Findingreasoning
In the paper's parameter recovery experiments, effort probability estimates were poorer than reliability estimates when little data was available.
“Our parameter recovery experiments in Section 5.1 found that graders' effort probabilities were difficult to estimate: compared to reliabilities, effort probability estimates were much poorer with little data, and converged more slowly as data became available.”
Quote in the paperSubstantial claim - Findingreasoning
In Fall21 and Winter21, modeling effort made the smallest difference in performance.
“in Fall21 and Winter21, the two offerings of the class where modeling effort made the smallest difference in performance”
Quote in the paperNot a substantial claim - Findingown data
Increasing the number of weeks (grades per grader) improved grader quality estimates.
“The results (Figure 4ab) show that increasing the size of the dataset in this way improved grader quality estimates.”
Quote in the paperNot a substantial claim - Findingown data
The 10-fold stratified cross-validation closely approximated leave-one-out cross-validation on one of the four datasets.
“An exploratory experiment on one of our four datasets confirmed that this approach closely approximated leave-one-out cross-validation.”
Quote in the paperSubstantial claim - Findingown data
Censoring always caused significant drops in performance.
“with censoring always causing significant drops in performance”
Quote in the paperSubstantial claim- Rests on an assumption: The TAs were diligent and responsible.
- Rests on an assumption: The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
- Findingreasoning
In the paper's parameter recovery experiments, effort probability estimates converged more slowly than reliability estimates as data became available.
“Our parameter recovery experiments in Section 5.1 found that graders' effort probabilities were difficult to estimate: compared to reliabilities, effort probability estimates were much poorer with little data, and converged more slowly as data became available.”
Quote in the paperSubstantial claim - Findingown data
Increasing the number of graders had little effect on the model's ability to recover students' reliabilities, biases, and effort probabilities.
“Increasing the amount of data in this way had little effect on the model's ability to recover students' reliabilities, biases, and effort probabilities.”
Quote in the paperSubstantial claim - Findingown data
The best-performing model included both censoring and biases for each dataset.
“Each bar represents the change relative to the best-performing model, which included both censoring and biases for each dataset.”
Quote in the paperSubstantial claim - Interpretationreasoning
These problems can be addressed by inferring distributions over grade and reliability estimates rather than point estimates.
“These problems can be addressed by inferring distributions over grade and reliability estimates rather than point estimates.”
Quote in the paperSubstantial claim - Interpretationreasoning
Error in estimates of reliabilities, biases, and effort probabilities was driven primarily by noise in the reported grades, not by noise in the underlying true grade estimates.
“This suggests that error in those estimates was driven primarily by noise in the reported grades, not by noise in the underlying true grade estimates.”
Quote in the paperSubstantial claim - Interpretationreasoning
There remain obstacles to deploying AI-based peer grading systems in practice.
“Despite the considerable intellectual progress just described, there remain obstacles to deploying AI-based peer grading systems in practice.”
Quote in the paperSubstantial claim - Interpretationreasoning
The posterior distribution over the PG1 model's parameters must be estimated numerically.
“Instead, the posterior must be estimated numerically.”
Quote in the paperSubstantial claim - Study design and conductown data
The models were validated on four years of real classroom data.
“We validated our models on four years of real classroom data and investigated both their ability to recover true parameters and their robustness on synthetic data.”
Quote in the paperSubstantial claim - Interpretationreasoning
The MAP approach can sometimes produce final grades larger or smaller than any peer grade (e.g., when the model assigns biases having the same sign to all graders).
“It can also sometimes produce final grades larger or smaller than any peer grade (e.g., when the model assigns biases having the same sign to all graders).”
Quote in the paperSubstantial claim - Interpretationreasoning
With only 4 grades per submission, most of the inaccuracy in the model's true grade estimates was driven by aggregating a small number of noisy signals, rather than by inaccurate estimates of graders' reliabilities, biases, and efforts.
“This suggests that, with only 4 grades per submission, most of the inaccuracy in the model's true grade estimates was driven by aggregating a small number of noisy signals, rather than because estimates of graders' reliabilities, biases, and efforts were inaccurate.”
Quote in the paperSubstantial claim - Interpretationnone given
The resulting weighted averages from the MIP can never produce a grade outside of the range of the peer grades.
“Notice that the resulting weighted averages can never produce a grade outside of the range of the peer grades.”
Quote in the paperNot a substantial claim - Interpretationown data
Probabilistic peer grading systems can be extended to correctly model discrete peer grades.
“We have shown how probabilistic peer grading systems can be extended to provide incentives for effortful grading; to correctly model discrete peer grades; and to output discrete, interpretable final grades that approximate MAP estimates.”
Quote in the paperSubstantial claim - Interpretationreasoning
Gibbs sampling is particularly attractive for PG1 because the true grade, bias, and reliability priors are conjugate priors for the normally-distributed peer grade likelihoods, giving each of the Gibbs updates a simple closed form.
“This approach is particularly attractive for PG1, as the true grade, bias, and reliability priors are conjugate priors for the normally-distributed peer grade likelihoods, giving each of the Gibbs updates a simple closed form.”
Quote in the paperSubstantial claim - Premisenone given
No system unifies graphical models and mechanism design by providing both Bayesian parameter estimates and meaningful incentives for students to invest effort in peer grading.
“Statistically rich methods based on graphical models and economically informed mechanism design approaches have been developed independently; we are not aware of any system that unifies the two by providing both Bayesian parameter estimates and meaningful incentives for students to invest effort in peer grading.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The censored observation model correctly leads us to consider grader disagreement to be more likely when an assignment's true grade is 3.51 than when it is 3.0.
“Observe that it correctly leads us to consider grader disagreement to be more likely when an assignment's true grade is 3.51 than when it is 3.0.”
Quote in the paperNot a substantial claim - Interpretationreasoning
In settings where the course staff use the same rubric as the students to grade submissions and where TA grades replace peer grades after an appeal, assigning real-valued final grades to students incentivizes half of them to ask for regrades.
“Furthermore, in settings where the course staff use the same rubric as the students to grade submissions and where TA grades replace peer grades after an appeal, assigning real-valued final grades to students incentivizes half of them to ask for regrades (e.g., if 3.6 is their true grade, the rounded TA grade would be 4).”
Quote in the paperSubstantial claim - Interpretationown data
Extensions to probabilistic peer grading systems output interpretable, discrete final grades that closely approximate maximum a posteriori estimates.
“output interpretable, discrete final grades that closely approximate maximum a posteriori estimates”
- Also stated in: § 3 Methods · plain
“Third, we introduce a mixed-integer programming method for identifying interpretable weighted averages of the peer grades that are faithful to the model's posterior beliefs.”
Quote in the paperSubstantial claim - Also stated in: § 3 Methods · plain
- Study design and conductown data
Real class data gave no way to check the accuracy of parameter estimates.
“While our experiments on real class data allowed us to test how well our models described real peer-grading behavior, they gave us no way to check the accuracy of the parameter estimates.”
Quote in the paperSubstantial claim- Rests on an assumption: Giving accurate grades (for submissions or students' grading abilities) is a primary focus of peer grading systems.
- Interpretationreasoning
The PG1 model does not give rise to a closed-form expression for the posterior distribution over its parameters.
“Such a complex model does not give rise to a closed-form expression for the posterior distribution over these parameters.”
Quote in the paperSubstantial claim - Study design and conductown data
The paper's approach was evaluated on both synthetic and real-world data obtained by using its implemented system in four large classes.
“We show how to make Bayesian inference practical in this model and evaluate our approach on both synthetic and real-world data obtained by using our implemented system in four large classes.”
Quote in the paperSubstantial claim - Premisereasoning
Even when point estimate methods maximize likelihood, such point estimates can be overly confident.
“Even when they do maximize likelihood, such point estimates can be overly confident.”
Quote in the paperSubstantial claim - Interpretationown data
Extensions to probabilistic peer grading systems improve inference quality in the presence of discrete grading rubrics.
“improve inference quality in the presence of discrete grading rubrics”
- Also stated in: § 3 Methods · boosted
“Second, we show how to better handle the common case where graders select discrete grades from a coarse rubric by modeling these reports as censored observations of an underlying real value.”
Quote in the paperSubstantial claim - Also stated in: § 3 Methods · boosted
- Interpretationreasoning
Extensions to probabilistic peer grading systems can detect low-effort, strategic behavior by students, enabling its disincentivization.
“introducing extensions to probabilistic peer grading systems that can detect (and hence enable the disincentivization of) low-effort, strategic behavior by students”
- Also stated in: § 3 Methods · plain
“First, we show how low-effort grading behavior can be disincentivized by augmenting the probabilistic model to include latent variables describing graders' effort.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Each time a grader grades a submission, they make a binary decision about whether to make an effort.
- The authors state this as an assumption: If a grader does not make an effort, they choose a random grade from a fixed low-effort distribution Dℓ.
- The authors state this as an assumption: Effort decisions are independent of the content of the submission.
- Also stated in: § 3 Methods · plain
- Interpretationnone given
Probabilistic peer grading systems can be extended to output discrete, interpretable final grades that approximate MAP estimates.
“We have shown how probabilistic peer grading systems can be extended to provide incentives for effortful grading; to correctly model discrete peer grades; and to output discrete, interpretable final grades that approximate MAP estimates.”
Quote in the paperSubstantial claim - Interpretationreasoning
The proposed MIP formulation maps posterior grade distributions to discrete final grades that can be explained as rounded weighted averages of reported peer grades.
“We propose a novel mixed integer programming (MIP) formulation that maps posterior grade distributions to discrete final grades that can be explained as rounded weighted averages of reported peer grades.”
Quote in the paperSubstantial claim- Rests on an assumption: Students tend to fixate on low-quality peer reviews and strongly prefer for such graders to receive zero weights rather than small positive weights.
- Interpretationreasoning
Inaccurate reliability and effort estimates are less important because rewarding good grading only requires students to be ranked in the correct order.
“but judged that this was less important because rewarding good grading only requires students to be ranked in the correct order.”
Quote in the paperSubstantial claim- Rests on an assumption: Rewarding good grading only requires students to be ranked in the correct order.
- Interpretationreasoning
We can still straightforwardly compute the likelihood of a reported grade for any setting of the submission's true grade and the grader's reliability and bias, enabling a discrete approximation of the Gibbs updates.
“While we are no longer able to apply Gibbs updates by evaluating a closed-form expression, we can still straightforwardly compute the likelihood of a reported grade for any setting of the submission's true grade and the grader's reliability and bias, enabling a discrete approximation of the Gibbs updates.”
Quote in the paperSubstantial claim - Interpretationreasoning
Probabilistic peer grading systems can be extended to provide incentives for effortful grading.
“We have shown how probabilistic peer grading systems can be extended to provide incentives for effortful grading; to correctly model discrete peer grades; and to output discrete, interpretable final grades that approximate MAP estimates.”
Quote in the paperSubstantial claim - Study design and conductown data
Real-world datasets give no way to reason about ground-truth values for parameters, so the models' posteriors cannot be evaluated for accuracy in recovering true parameter values.
“Of course, our real-world datasets give us no way to reason about ground-truth values for any parameters (like true grades, reliabilities, etc), so we cannot evaluate how accurately our models' posteriors recover these parameters' true values.”
Quote in the paperSubstantial claim - Interpretationnone given
Degraded reliability estimation implies degraded true grade estimation.
“Of course, degraded reliability estimation implies degraded true grade estimation.”
Quote in the paperNot a substantial claim - Interpretationnone given
The proposed approach for extending PG-style models to realistic discrete grade distributions maintains the tractability of Gibbs updates.
“We propose an approach for extending PG-style models to realistic discrete grade distributions that maintains the tractability of Gibbs updates.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 25
- May combine two claims. 19
- The quote only partly states this claim. 2
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationThe extensions can be added to the PG5 model of Mi and Yeung (2015), additionally modeling correlations between students' submission grades and reliabilities.
“In Appendix B, we show how these extensions could be added to the PG5 model of Mi and Yeung (2015), additionally modeling correlations between students' submission grades and reliabilities.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Mi and Yeung (2015) source
Classified as probably not a substantial claim. May combine two claims.
PremisecitationThe PG1 model assumes that each assignment has a latent true grade and each peer grader has a latent bias and reliability.
“Their core”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Piech et al. (2013) source
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationFollow-up work extended these models to allow for more complex reliability-grade correlations (PG4 and PG5) and to more explicitly account for differences between a single grader's reported grades (PG6 and PG7).
“Follow-up work by others further extended these models to allow for more complex reliability-grade correlations (the PG4 and PG5 models of Mi and Yeung 2015) and to more explicitly account for differences between a single grader's reported grades (the PG6 and PG7 models of Wang et al. 2019).”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Mi and Yeung 2015 source
- cites Wang et al. 2019
May combine two claims.
Findingown dataReliability estimates had a correlation of 0.6 after one week of data and improved to 0.9 after 8 weeks.
“Reliability estimates had an appreciable correlation of 0.6 after just one week of data, improving substantially to 0.9 after 8 weeks.”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown dataThe method for generating interpretable grades closely tracked MAP estimates, using real data from four offerings of a large class.
“and that our method for generating interpretable grades closely tracked MAP estimates (Section 4)”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAdding additional peer graders on each submission substantially reduced true grade MAE, from 0.52 with a single grader to 0.41 with four, which is below the MAE of a TA.
“These results indicate that adding additional peer graders on each submission substantially reduced true grade MAE, from 0.52 with a single grader to 0.41 with four-well below the MAE of a TA.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.
Findingnone givenIn Fall21 and Winter21, greater efforts were made to detect low-effort behavior.
“we made greater efforts to detect low-effort behavior”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe previous section showed that a model incorporating both efforts and censoring had the best performance on all four datasets, but its optimal hyperparameters varied.
“The previous section showed that a model incorporating both efforts and censoring had the best performance on all four datasets, but its optimal hyperparameters varied.”
Quote in the paperNot a substantial claim- Rests on an assumption: The TAs were diligent and responsible.
- Rests on an assumption: The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
May combine two claims.
FindingreasoningIn simulations, changing the number of students in the class had surprisingly little impact on the inference problem's difficulty, because each grader continued to grade 40 submissions and each submission continued to receive 4 grades.
“However, this change had surprisingly little impact on the inference problem's difficulty, because as the class size varies, each grader continued to grade a total of 40 submissions, and each submission continued to receive 4 grades.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
Findingnone givenIn our experience, students find such grades confusing and unfair; they instead expect to receive final grades that interpolate their received grades, such as averages weighted by each grader's perceived trustworthiness.
“In our experience, students find such grades confusing and unfair; they instead expect to receive final grades that interpolate their received grades, such as averages weighted by each grader's perceived trustworthiness.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAll model changes made small but meaningful changes to final grades, averaging between 0.05 and 0.15 points on a 5-point scale, and very rarely exceeding a single point change.
“All model changes made small but meaningful changes to the final grades, averaging between 0.05 and 0.15 points on our 5-point scale, and very rarely ever exceeding a single point change.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenA hypothetical TA with a reliability of 2 achieves a mean absolute error of 0.48.
“a hypothetical TA with a reliability of 2 (who achieves a mean absolute error of 0.48)”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWith only a single grader, the MIP output was equal to the mean of the Gibbs samples, which is much less accurate than the MAP.
“With only a single grader, the MIP output was equal to the mean of the Gibbs samples, which is much less accurate than the MAP.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEffort estimates had a correlation of 0.3 after one week of data and improved to 0.7 in later weeks.
“Effort estimates followed a similar trend, but were much more difficult to estimate: one week of data produced a much poorer correlation of 0.3, with later weeks improving to 0.7.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataUsing 4,000 Gibbs samples caused average error of 1%, 3%, and 1% on estimates of true grades, reliabilities, and effort compared to 10,000 samples, and had a small impact on results.
“comparing to runs with 10,000 Gibbs samples, our protocol of gathering 4,000 samples caused an average error of 1%, 3%, and 1% on our estimates of true grades, reliabilities, and effort, respectively, and therefore had a small impact on our results.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataAccuracy and RMSE gave qualitatively similar conclusions to MAE.
“We also computed accuracy (the fraction of true grade MAP estimates equal to the rounded true grade) and RMSE, finding qualitatively similar conclusions with these measures.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningWhen low-effort behavior is successfully disincentivized, detecting it will have less impact on model performance.
“of course, when such behavior is successfully disincentivized, detecting it will have less impact on model performance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataThe study collected 4 runs of 1,100 Gibbs samples, discarding the first 100 burnin samples from each run and concatenating the remainder, taking about 8 CPU hours.
“Each time we fit our model to a dataset, we collected 4 runs of 1,100 Gibbs samples, discarding the first 100 burnin samples from each run and concatenating the remainder; this took about 8 CPU hours.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenThe study used paired t-tests to compare held-out likelihoods of models on the same dataset.
“We use paired t-tests to make statistical comparisons between held-out likelihoods of several models on the same dataset.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationPiech et al. also introduced PG2 and PG3 models that respectively permit graders' biases to change over time and students' grading reliability to be correlated with their own assignments' grades.
“Piech et al. also introduced PG2 and PG3 models that respectively permit graders' biases to change over time and students' grading reliability to be correlated with their own assignments' grades.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Piech et al.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisereasoningWork in this area is mostly centered around the idea of peer prediction, developing mechanisms that incentivize graders to grade carefully by rewarding them based on comparing their peer grades to others'.
“Work in this area is mostly centered around the idea of peer prediction, developing mechanisms that incentivize graders to grade carefully by rewarding them based on comparing their peer grades to others'.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationAn extensive line of work in mechanism design focuses on incentivizing high quality reporting in peer grading and other crowdsourcing environments.
“In response to this issue, an extensive line of work in the mechanism design literature focuses on incentivizing high quality reporting in peer grading and other crowdsourcing environments (Prelec 2004;Jurca andFaltings 2009, 2005;Faltings, Li, and Jurca 2012;Witkowski and Parkes 2012;Witkowski et al. 2013;Radanovic andFaltings 2013, 2014;Riley 2014;Kamble et al. 2015;Kong, Ligett, and Schoenebeck”
Quote in the paperNot a substantial claim0 of 3 sources checkedClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningThe specification of low-effort behavior is important both for boosting model performance and for disincentivizing bad behavior.
“The second is the specification of low-effort behavior, which is important both for boosting model performance and for disincentivizing bad behavior”
Quote in the paperSubstantial claim- Rests on an assumption: If the model is good at identifying low effort behavior, students will exhibit this behavior less often.
Classified as probably not a substantial claim.
Study design and conductcitationThe study evaluated models based on held-out likelihood.
“Instead, we evaluated our models based on their ability to predict held out data: specifically, their held-out likelihood (Vehtari, Gelman, and Gabry 2016).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Vehtari, Gelman, and Gabry 2016
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenA key advantage of a Bayesian approach to reasoning about peer grades is that it yields distributional posterior beliefs about each quantity of interest rather than point estimates.
“A key advantage of a Bayesian approach to reasoning about peer grades is that it yields distributional posterior beliefs about each quantity of interest rather than point estimates.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenPerforming naive Gibbs sampling on the latent peer grade variables would lead to extremely sample-inefficient inference: the posterior distribution has multiple modes where the true grade nearly matches one latent peer grade, and Gibbs sampling rarely moves between these modes.
“Performing naive Gibbs sampling on the latent peer grade variables would lead to extremely sample-inefficient inference: the posterior distribution has multiple modes where the true grade nearly matches one latent peer grade, and Gibbs sampling rarely moves between these modes.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Premisenone givenMost current peer grading systems either take the mean or median of reported grades; others aim to estimate students' grading accuracy under a probabilistic model.
“Most current systems either take the mean or median of reported grades; others aim to estimate students' grading accuracy under a probabilistic model.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationcitationThe MAP approach leaves students with little insight about how their peer grades influenced the calculation; this can lead to reduced trust in the system and more appeals.
“While this approach is statistically sensible, it leaves students with little insight about how their peer grades influenced the calculation; this can lead to reduced trust in the system (Kizilcec 2016) and more appeals.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kizilcec 2016 source
Classified as probably not a substantial claim.
Study design and conductreasoningThe study evaluated reliability and effort estimates with the Spearman rank-order correlation coefficient.
“Accordingly, we evaluated our reliability and effort estimates with the Spearman rank-order correlation coefficient, measuring how similarly students were ranked by the estimates and true values.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenMapping the mean of the Gibbs samples to the nearest rubric element can lead to rounding errors.
“One might map the mean of the Gibbs samples to the nearest rubric element, but this can lead to rounding errors.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningExplicitly modeling low-effort behavior can reduce students' incentives for such behavior and help avoid assigning high reliabilities to low-effort students.
“We can reduce students' incentives for such low-effort behavior by explicitly modeling it, helping us to avoid assigning high reliabilities to low-effort students.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Each time a grader grades a submission, they make a binary decision about whether to make an effort.
- The authors state this as an assumption: If a grader does not make an effort, they choose a random grade from a fixed low-effort distribution Dℓ.
- The authors state this as an assumption: Effort decisions are independent of the content of the submission.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThese mechanisms typically rely on restrictive modeling assumptions, making them inapplicable to most practical peer grading systems.
“While our model of low-effort grading is inspired by work in this area, these mechanisms typically rely on restrictive modeling assumptions, making them inapplicable to most practical peer grading systems.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenRegardless of the choice of Dl, Gibbs sampling remains straightforward: both efforts and effort probabilities yield closed-form updates, and all other parameter updates simply exclude grades that a given sample calls low effort.
“Regardless of our choice of D ℓ , Gibbs sampling remains straightforward: both efforts and effort probabilities yield closed-form updates, and all other parameter updates simply exclude grades that a given sample calls low effort.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoningThe 10-fold stratified cross-validation ensured that no two peer grades on the same submission were in the same group.
“We first split the dataset into 10 groups of n/10 peer grades, ensuring that no two peer grades on the same submission were in the same group.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationEven if the mapping from real-valued grades to the course's rubric produces accurate grades, students must be able to understand it in order to trust the system.
“Even if this mapping produces accurate grades, students must be able to understand it in order to trust the system (Kizilcec 2016).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kizilcec 2016 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe extensions could be applied to any of the PG* models in the literature.
“but they could be applied to any of the PG* models in the literature”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenAt their best, methods that produce point estimates maximize the likelihood of the data given a model.
“At their best, methods that produce point estimates maximize the likelihood of the data given a model, e.g., by assigning each grader a”
Quote in the paperNot a substantial claim
The paper relies on this without support.
For peer grading systems to be both useful to instructors and acceptable to students, they must produce grades sufficiently similar to those an instructor would have given.
“In order for peer grading systems to be both useful to instructors and acceptable to students, they must produce grades that are sufficiently similar to those that an instructor would have given.”
The paper relies on this without support.
Individual peer graders are biased (consistently give generous or harsh grades).
“This is a challenging task because individual peer graders will be biased (consistently give generous or harsh grades); noisy (the same grader could grade an assignment differently on different days); and potentially strategic (some students will enter insincere peer grades unrelated to a submission's quality if they can get away with it).”
The paper relies on this without support.
Individual peer graders are noisy (the same grader could grade an assignment differently on different days).
“This is a challenging task because individual peer graders will be biased (consistently give generous or harsh grades); noisy (the same grader could grade an assignment differently on different days); and potentially strategic (some students will enter insincere peer grades unrelated to a submission's quality if they can get away with it).”
The paper relies on this without support.
Students can behave strategically (e.g., reporting grades close to the class average without doing the work, or entering insincere peer grades unrelated to a submission's quality if they can get away with it).
“This is a challenging task because individual peer graders will be biased (consistently give generous or harsh grades); noisy (the same grader could grade an assignment differently on different days); and potentially strategic (some students will enter insincere peer grades unrelated to a submission's quality if they can get away with it).”
“recognizing that students can behave strategically (e.g., reporting grades close to the class average without doing the work)”
The paper relies on this without support.
Students can subvert systems that assess grading quality by comparing individual grades to each other if they coordinate on all reporting the same grade.
“For example, students can subvert systems that assess grading quality by comparing individual grades to each other if they coordinate on all reporting the same grade.”
The paper relies on this without support.
The statistical literature allows both students and the peer grading system itself to assign real-valued grades, whereas real instructors tend to use coarse-valued rubrics, particularly when eliciting grades from students.
“Furthermore, the statistical literature allows both students and the peer grading system itself to assign real-valued grades, whereas real instructors tend to use coarse-valued rubrics, particularly when eliciting grades from students.”
The paper relies on this without support.
Censored data arises from discrete-valued grading rubrics.
“appropriately handling censored data that arises from discretevalued grading rubrics”
The paper relies on this without support.
Mixed integer programming improves the interpretability of the grades assigned to students.
“using mixed integer programming to improve the interpretability of the grades assigned to students”
The paper relies on this without support.
In order for a statistical model to be able to accurately recover true grades from peer reports, students must invest the effort required to grade as well as they can.
“In order for a statistical model to be able to accurately recover true grades from peer reports, students must invest the effort required to grade as well as they can.”
The paper relies on this without support.
It is not easy to determine whether a student has done a good job of peer grading when there are no instructor or TA grades to which their evaluation can be compared.
“However, it is not easy to determine whether a student has done a good job of peer grading when there are no instructor or TA grades to which their evaluation can be compared.”
The authors state this as an assumption.
Each time a grader grades a submission, they make a binary decision about whether to make an effort.
“Each time a grader v grades a submission u, we assume they make a binary decision about whether to make an effort z v u on the submission.”
The authors state this as an assumption.
If a grader does not make an effort, they choose a random grade from a fixed low-effort distribution Dℓ.
“otherwise, they choose a random grade from a fixed”
The authors state this as an assumption.
If a grader makes an effort, they produce a noisy grade as usual.
“If they choose to make an effort, they produce a noisy grade as usual”
The authors state this as an assumption.
Effort decisions are independent of the content of the submission.
“For simplicity, we model these effort decisions as being independent of the content of the submission.”
The authors state this as an assumption.
Each submission is graded on C components with possible grades of {0, 1, ..., M} for each.
“we assume that each submission is graded on C components C = {1, 2, . . ., C}, with possible grades of {0, 1, . . ., M } for each.”
The paper relies on this without support.
Some low-effort students may adopt other, idiosyncratic strategies such as assigning everything a low or high grade.
“Some low-effort students may adopt other, idiosyncratic strategies such as assigning everything a low or high grade.”
The paper relies on this without support.
In practice, submissions are usually graded on coarse, discrete rubrics such as five-point scales, and virtually no class allows graders arbitrary decimal precision.
“In practice, submissions are usually graded on coarse, discrete rubrics such as five-point scales (and virtually no class allows graders arbitrary decimal precision).”
The paper relies on this without support.
Students expect to receive discrete grades rather than probability distributions.
“However, students still expect to receive discrete grades rather than probability distributions.”
The paper relies on this without support.
Students tend to fixate on low-quality peer reviews and strongly prefer for such graders to receive zero weights rather than small positive weights.
“However, students also tend to fixate on low-quality peer reviews and strongly prefer for such graders to receive zero weights rather than small positive weights.”
The paper relies on this without support.
The TAs were diligent and responsible.
“Our TAs were diligent and responsible”
The paper relies on this without support.
The gold-standard calibration submissions' grades were painstakingly agreed upon by the whole course staff.
“these grades were painstakingly agreed upon by the whole course staff”
The paper relies on this without support.
Giving accurate grades (for submissions or students' grading abilities) is a primary focus of peer grading systems.
“Of course, giving accurate grades (either for submissions or students' grading abilities) is a primary focus of peer grading systems.”
The paper relies on this without support.
Rewarding good grading only requires students to be ranked in the correct order.
“rewarding good grading only requires students to be ranked in the correct order”
The paper relies on this without support.
If the model is good at identifying low effort behavior, students will exhibit this behavior less often.
“if the model is good at identifying low effort behavior, students will exhibit this behavior less often.”
The authors state this as an assumption.
Low-effort graders in the model choose grades that are as difficult as possible to distinguish from effortful graders.
“our low-effort graders choose grades that are as difficult as possible to distinguish from effortful graders.”
The paper relies on this without support.
Coarse rubrics cause many high-effort grades to match the class average exactly.
“The problem is exacerbated by coarse rubrics, which cause many high-effort grades to match the class average exactly.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Single-grader MIP output cannot generally equal the Gibbs-sample mean
MinorWith one peer grade, the MIP's weighted average must equal that reported grade. It cannot generally equal the mean of the Gibbs samples; the two could match only by coincidence.
“With only a single grader, the MIP output was equal to the mean of the Gibbs samples, which is much less accurate than the MAP.”
“Notice that the resulting weighted averages can never produce a grade outside of the range of the peer grades.”
Beta prior mean is misstated
MinorThe model defines the effort probability prior as Beta(α e, β e), whose mean is α e /(α e + β e), not α e /β e as stated in the hyperparameter discussion.
“the effort probability mean α e /β e ;”
“(Effort prob.) e v ∼ Beta(α e , β e );”
Cites
Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949 · see citing works
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611 · see citing works
Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images*
Geman, S., & Geman, D. (1993) · Journal of Applied Statistics · doi:10.1080/02664769300000058 · Cited by 984 · see citing works
Topping, K. J. (2009) · Theory Into Practice · doi:10.1080/00405840802577569 · Cited by 785 · see citing works
Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
Geman, S., & Geman, D. (1987) · Elsevier eBooks · doi:10.1016/b978-0-08-051581-6.50057-x · Cited by 750 · see citing works
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · see citing works
Online peer assessment: effects of cognitive and affective feedback
Lu, J., & Law, N. W. Y. (2011) · Instructional Science · doi:10.1007/s11251-011-9177-2 · Cited by 272 · see citing works
Tuned Models of Peer Assessment in MOOCs
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013) · arXiv · doi:10.48550/arxiv.1307.2579 · Cited by 262 · see citing works
A Robust Bayesian Truth Serum for Small Populations
Witkowski, J., & Parkes, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8261 · Cited by 143 · see citing works
Informed Truthfulness in Multi-Task Peer Prediction
Shnayder, V., Agarwal, A., Frongillo, R., et al. (2016) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2940716.2940790 · Cited by 114 · see citing works
Output Agreement Mechanisms and Common Knowledge
Waggoner, B., & Chen, Y. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13151 · Cited by 110 · see citing works
A Robust Bayesian Truth Serum for Non-Binary Signals
Radanović, G., & Faltings, B. V. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8677 · Cited by 108 · see citing works
Mechanisms for Making Crowds Truthful
Jurca, R., & Faltings, B. V. (2009) · Cited by 99 · see citing works
de Alfaro, L., & Shavlovsky, M. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538900 · Cited by 92 · see citing works
Dwelling on the Negative: Incentivizing Effort in Peer Prediction
Witkowski, J., Bachrach, Y., Key, P. B., et al. (2013) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v1i1.13089 · Cited by 62 · see citing works
A method of automatic grade calibration in peer assessment
Hamer, J., Kenneth, T. K., & Kwong, H. H. F. (2005) · Cited by 60 · see citing works
Incentives for Truthful Information Elicitation of Continuous Signals
Radanović, G., & Faltings, B. V. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8797 · Cited by 58 · see citing works
Enforcing Truthful Strategies in Incentive Compatible Reputation Mechanisms
Jurca, R., & Faltings, B. V. (2005) · Lecture notes in computer science · doi:10.1007/11600930_26 · Cited by 51 · see citing works
Gao, X. A., Mao, A., Chen, Y., & Adams, R. P. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602865 · Cited by 39 · see citing works
The PeerRank Method for Peer Assessment
Walsh, T. (2014) · arXiv · doi:10.48550/arxiv.1405.7192 · Cited by 36 · see citing works
Probabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs
Mi, F., & Yeung, D.-Y. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9210 · Cited by 34 · see citing works
Putting Peer Prediction Under the Micro(economic)scope and Making Truth-Telling Focal
Kong, Y., Ligett, K., & Schoenebeck, G. R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-662-54110-4_18 · Cited by 32 · see citing works
Minimum Truth Serums with Optional Predictions
Riley, B. (2014) · Cited by 24 · see citing works
Truth Serums for Massively Crowdsourced Evaluation TasksPreprint
Kamble, V., Shah, N. B., Marn, D., et al. (2015) · arXiv (Cornell University) · Cited by 19 · see citing works
Eliciting truthful measurements from a community of sensors
Faltings, B., Li, J. J., & Jurca, R. (2012) · 2012 3rd IEEE International Conference on the Internet of Things · doi:10.1109/iot.2012.6402303 · Cited by 18 · see citing works
Incentivizing evaluation with peer prediction and limited access to ground truth
Gao, X. A., Wright, J. R., & Leyton‐Brown, K. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2019.03.004 · Cited by 16 · see citing works
Self-citationHuman-Machine Hybrid Peer Grading in SPOCs
Han, Y., Wu, W., Yan, Y., & Zhang, L. (2020) · IEEE Access · doi:10.1109/access.2020.3043291 · Cited by 14 · see citing works
Deep Bayesian Trust: A Dominant and Fair Incentive Mechanism for Crowd
Goel, N., & Faltings, B. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011996 · Cited by 7 · see citing works
(2019) · Educational Data Mining · Cited by 7 · see citing works
Measurement Integrity in Peer Prediction: A Peer Assessment Case Study
Burrell, N., & Schoenebeck, G. (2021) · arXiv · doi:10.48550/arxiv.2108.05521 · Cited by 7
Report-Sensitive Spot-Checking in Peer-Grading Systems
Zarkoob, H., Fu, H. D., & Leyton‐Brown, K. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/ecst4444 · Cited by 6 · see citing works
Self-citationMeasurement Integrity in Peer Prediction: A Peer Assessment Case Study
Burrell, N., & Schoenebeck, G. R. (2023) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3580507.3597744 · Cited by 5 · see citing works
(2018) · arXiv (Cornell University) · Cited by 3 · see citing works
Surrogate Scoring Rules and a Dominant Truth SerumPreprint
Liu, Y., & Chen, Y. (2018) · arXiv (Cornell University) · Cited by 2 · see citing works
Hartline, J. D., Li, Y., Shan, L., & Wu, Y. (2020) · arXiv · doi:10.48550/arxiv.2007.02905 · Cited by 1 · see citing works
SwaGrader: A Honest Effort Extracting, Modular Peer-Grading Tool
Prajapati, S., Gupta, A., Nigam, S. K., & Nath, S. (2020) · Proceedings of the 7th ACM IKDD CoDS and 25th COMAD · doi:10.1145/3371158.3371205 · Cited by 1
Truthful Peer Grading with Limited Effort from Teaching StaffPreprint
Jindal, J., & Nath, S. (2018) · arXiv.org · doi:10.48550/arxiv.1807.11657
Surrogate Scoring RulesPreprint
Liu, Y., Wang, J., & Chen, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09158
Mechanisms for Making Crowds Truthful
Jurca, R., & Faltings, B. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2621
The PeerRank Method for Peer Assessment
Toby, W. (2014) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-61499-419-0-909
The Square Root Agreement Rule for Incentivizing Truthful Feedback on Online Platforms
Kamble, V., Shah, N., Marn, D., Parekh, A., & Ramachandran, K. (2015) · arXiv · doi:10.48550/arxiv.1507.07045
Flagged first, then by citations
Cited by
Sorted by citations
Manipulation and peer mechanisms: A survey
Olckers, M., & Walsh, T. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104196 · Cited by 12 · see citing works
On the Supervision of Peer Assessment Tasks: An Efficient Instructor Guidance Technique
Hernández-González, J., & Herrera, P. J. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3319733 · Cited by 5 · see citing works
Learning in Teams: Peer Evaluation for Fair Assessment of Individual Contributions
Duzhin, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230322 · Cited by 4 · see citing works
Removing Bias and Incentivizing Precision in Peer-grading
Chakraborty, A., Jindal, J., & Nath, S. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15329 · Cited by 5 · see citing works
Paul, U., Shah, S., Mylavarapu, S. V., et al. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15743-0_27 · Cited by 1 · see citing works
Paul, U., Mantravadi, A., Shah, J., Shah, S., Mylavarapu, S. V., Rashid, M. P., & Gehringer, E. F. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328205
- Authors
- Zarkoob, H., D'Eon, G., Podina, L., & Leyton-Brown, K.
- Affiliations
- University of British Columbia (CA)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2023-06-26
- Volume
- 37
- Issue
- 5
- Pages
- 6137-6144
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4382239954
- Cited by
- 6
- References
- 32
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 2
- FWCI
- 1.81
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Compute Canada
- Alberta Machine Intelligence Institute
- University of British Columbia
- Defense Advanced Research Projects Agency (FA8750, FA8750-19-2-0222)
- Simons Institute for the Theory of Computing, University of California Berkeley
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- peer grading, Bayesian inference, strategic reporting, censored data, rubrics, mixed integer programming, large classes, robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
