Cited by 390Student Assessment and FeedbackOnline Learning and AnalyticsOnline and Blended Learning

The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 52 sources its claims cite were checked for support.

Peer and self assessment in massive online classes

Open access

Kulkarni, Chinmay, Wei, Koh Pang, Le, Huy, Chia, Daniel, Papadopoulos, Kathryn, Cheng, Justin, Koller, Daphne, Klemmer, Scott R.·2013·ACM Transactions on Computer-Human Interaction

DOI 10.1145/2505057

Open this paper in CoChat — free account Open PDF

Abstract

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Integrity signals

Registry

Crossref · 10.1145/2505057

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

2 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

2 of 52 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Supported
  In “Delivers high quality information to students about their learning” (p. 9), the paper defines good-quality feedback as information that helps students identify problems and take action to improve their performance. It also recommends timely feedback and corrective advice (p. 10), supporting the claim’s emphasis on accuracy and actionability. (LLM, conf=0.93)
  cites Nicol and Macfarlane-Dick 2006 source
- Not checked — no full text of the sourcecites Boud 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Partly supported
  The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)
  cites Cheshire and Antin 2008 sourceExperimental — accuracy not yet measured
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students also reported that they felt their peers put less effort into peer assessment than they did (Table III).On a Mann-Whitney U-test, mean ratings were 4.57 for peer effort and 5.46 for their own effort (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that giving students feedback about their grading bias increased subsequent accuracy.”
- Also stated in: § 1.4. Contributions · plain
  “First, we found that giving students feedback about whether they scored peers high or low increased their subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, students assessed their work 7% higher than staff did.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second iteration had 42.9% of students' grades within 5% of the staff grade, and 65.5% within 10%.”
- Also stated in: § 3.2. Results: Grading Agreement · plain
  “The second iteration improved to 42.9% within 5% of the staff grade, and 65.5% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the first iteration, 34.0% of submissions had a median peer grade within 5% of the staff grade, and 56.9% within 10% (Figure 4).”
§ Peer and Self Assessment in Massive Online Classes > 3.2. Results: Grading Agreement
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After revising rubrics, median grading error decreased from 12.4% to 9.9%.”
- Also stated in: § 1.4. Contributions · plain
  “After making these changes, the mean error on grades decreased from 12.4% to 9.9%.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second iteration had 42.9% of students' grades within 5% of the staff grade, and 65.5% within 10%.”
- Also stated in: § 3.2. Results: Grading Agreement · plain
  “The second iteration improved to 42.9% within 5% of the staff grade, and 65.5% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, 20% of students voluntarily assessed more submissions than required.”
- Also stated in: § 3.5. Results: Student Reactions · plain
  “Student reactions to the peer assessment system were generally positive, and 20% of students completed more peer assessments than the class required them to (Figure 7).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, students reported learning more by assessing their peers than by assessing themselves: mean ratings were 4.97 and 4.51, respectively, for peer and self-assessment (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Load-bearingStrength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
- Also stated in: § Abstract · plain
  “In this class, peer grades correlated highly with staff-assigned grades.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, over the second iteration of the class, the average disagreement between peer raters was 25.0%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Forty-two percent of students cited seeing other students' work as the biggest benefit of peer assessment; 31% reported learning how to communicate their ideas as a benefit.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Forty-two percent of students cited seeing other students' work as the biggest benefit of peer assessment; 31% reported learning how to communicate their ideas as a benefit.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the first iteration, 34.0% of submissions had a median peer grade within 5% of the staff grade, and 56.9% within 10% (Figure 4).”
§ Peer and Self Assessment in Massive Online Classes > 3.2. Results: Grading Agreement
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students also rated peers' work from their own country 3.6% higher than those from elsewhere.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Strength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the repeated sampling scheme described in Section 3.1, five assignments had 26.4%, 36.2%,36.9%,43.9%, and 36.8% of submissions estimated within 5% of the staff grade.”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Agreement of peer grades with staff grades generally increases across the class.”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Using repeated sampling, we estimate more than 75% of students got a course grade within 5% of staff grade (assuming grades in different assignments are uncorrelated).”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:hedged· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- The authors state this as an assumption: Grades in different assignments are uncorrelated.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because currently about 40% of assignments are a full letter grade away”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Holding out 10% of ground truth grades, and testing on samples drawn from them, the regression model yields an accuracy of 35.8% of samples within 5%, and 58.8% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments to an (unreasonable) 101 per submission increases the number of submissions graded within 10% of the staff grade by 8.1%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The effect of the presence of feedback is significant: t(4, 998) = -3.38,p < 0.01.”
- Also stated in: § 7. CONCLUSIONS AND FUTURE WORK · boosted
  “We demonstrated that providing students feedback about their rating bias improves subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Within a 10% range, the assignments had, respectively, 49.1%, 53.6%, 60.9%, 68.5%, and 64.3% within 10% (Figure 10(a)).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we only consider raters who finished the class (and exclude those who dropped out), we see that staff agreement increases as well.”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the feedback condition, 4.4% more samples obtained a grade within 5% of the staff grade than those without feedback.”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We estimate that fewer than 45 students (approximatel 6%) were affected by grading errors in each iteration of the class.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:hedged· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
In the paper
“Furthermore, viewing and critiquing others' work plays a key pedagogical role in these domains [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They also suggest that aggregating peer grades leads to a remarkable increase in agreement with staff grades (Section 3.2).”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
None of the sources cited for this claim have been checked yet.
In the paper
“In-class peers can assess each other well [Falchikov and Goldfinch 2000;Carlson and Berry 2003;Gerdeman et al. 2007].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Falchikov and Goldfinch 2000 source
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Carlson and Berry 2003
- Not checked — no full text of the sourcecites Gerdeman et al. 2007
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results suggest that correlation among staff grades is many times higher than agreement among peer raters.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:hedged· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
In the paper
“Fields like design have also traditionally relied on intimate colocation to enable these activities and to confer values and norms [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
None of the sources cited for this claim have been checked yet.
In the paper
“Many such classes use automated assessment (e.g., Widom [2012]), which precludes the open-ended work that is a hallmark of education in creative fields like design [Buxton 2007].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Widom [2012]
- Not checked — no full text of the sourcecites Buxton 2007
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
In the paper
“Actionable feedback is most useful if it is personalized and targets the student's recent work [Gallien and Oomen-Early 2008].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Not checked — no full text of the sourcecites Gallien and Oomen-Early 2008
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each student graded at least one ground-truth submission per assignment; a groundtruth assignment had a median of 160 assessments.”
§ Peer and Self Assessment in Massive Online Classes > 3. HOW ACCURATE WAS PEER ASSESSMENT? 3.1. Methods To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).Across both iterations, staff graded 99 ground-truth submissions.Each student graded at least one ground-truth submission per assignment; a groundtruth assignment had a median of 160 assessments.(Some students graded more than one ground-truth submission per assignment because the system would give them a fresh ground-truth assignment when they logged out without finishing assessment and returned to the website after a long time).This article's grading procedure assigns the median grade from a small number of randomly selected peers (e.g., four to five).We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.To simulate the median-grade approach, we randomly sampled (with replacement) five student assessments for each ground-truth submission and compared the sample's median to the staff grade.5 We present results for 1,000 samples of five assessments per submission.This sampling method is essentially a bootstrapped statistical analysis [Efron and Tibshirani 1993].It allows staff to only evaluate a small set of randomly selected submissions and still provides an estimate for every peer rater's agreement with his or her grade (since all peers see at least one staff-graded submission).Repeatedly sampling five grades from the pool of peer grades provides an approximate distribution of agreement between staff and peer grades. We also compared students' self-grade with their median peer grade to measure whether students rate themselves differently than their peers.
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The general similarity in accuracy across both iterations (with improvements in the second) suggests that the peer assessment process produces robust results.”
§ Peer and Self Assessment in Massive Online Classes > 3. HOW ACCURATE WAS PEER ASSESSMENT? 3.1. Methods To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).Across both iterations, staff graded 99 ground-truth submissions.Each student graded at least one ground-truth submission per assignment; a groundtruth assignment had a median of 160 assessments.(Some students graded more than one ground-truth submission per assignment because the system would give them a fresh ground-truth assignment when they logged out without finishing assessment and returned to the website after a long time).This article's grading procedure assigns the median grade from a small number of randomly selected peers (e.g., four to five).We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.To simulate the median-grade approach, we randomly sampled (with replacement) five student assessments for each ground-truth submission and compared the sample's median to the staff grade.5 We present results for 1,000 samples of five assessments per submission.This sampling method is essentially a bootstrapped statistical analysis [Efron and Tibshirani 1993].It allows staff to only evaluate a small set of randomly selected submissions and still provides an estimate for every peer rater's agreement with his or her grade (since all peers see at least one staff-graded submission).Repeatedly sampling five grades from the pool of peer grades provides an approximate distribution of agreement between staff and peer grades. We also compared students' self-grade with their median peer grade to measure whether students rate themselves differently than their peers.
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
None of the sources cited for this claim have been checked yet.
In the paper
“Evaluating peers' work also exposes students to solutions, strategies, and insights that they otherwise would likely not see [Chinn 2005;Tinapple et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Chinn 2005
- Not checked — no full text of the sourcecites Tinapple et al. 2013
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“By symmetry, this means that at least four randomly selected raters saw each student's submission, and that each student saw one staff-assessed submission per assignment.”
§ Peer and Self Assessment in Massive Online Classes > 2.3. Peer Assessment
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
In the paper
“Second, consensus among raters serves as a useful indicator of quality [Huang and Fu 2013].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Huang and Fu 2013 source
Strength:plain· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Third, interfaces like FoldIt [Khatib et al. 2011] and NASA Clickworkers [Szpir 2002] demonstrate that short, well-crafted training exercises can enable legions of motivated amateurs to perform work previously thought to require years of training.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Khatib et al. 2011
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Szpir 2002
Strength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“Peer feedback in design classes also creates an audience that provides honest feedback and multiple perspectives [Tinapple et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Tinapple et al. 2013
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“To date, large-scale online classes, including our own, have primarily emphasized learning, rather than certification [Widom 2012].”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Widom 2012
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Cheating may also arise if the value of officially recorded performance in these classes increases (e.g., Kurhila [2012] and Lewin [2013b]).”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Kurhila [2012]
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Lewin [2013b]
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“When peer assessment provides the primary evaluative function, the staff role shifts to emphasize coaching [Kuebli et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Kuebli et al. 2008 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“To the extent that participation in the online classroom is based on intrinsic motivations (such as a desire to learn), students rarely blatantly cheat [Mazar et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not checked — no full text of the sourcecites Mazar et al. 2008 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
In the paper
“By contrast, when teachers coach but do not grade, students focus more on conceptual understanding [Perry 1970].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Perry 1970
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes

It may state two separate claims as one. 56
Probably not a substantial claim. 56
The quote only partly states this claim. 1

A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Load-bearingStrength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was also useful for identifying mistakes and reflection (Figure 8).”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating peers was useful for inspiration and to see other perspectives.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
In the paper
“Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Lewin 2013c
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Zimmerman and Schunk 2001 source
- Not checked — no full text of the sourcecites Pintrich 1995 source
- Not checked — no full text of the sourcecites Pintrich and Zusho 2007 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Hearst 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Chi 2009
- Not checked — no full text of the sourcecites Carter et al. 2008 source
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Galinsky and Moskowitz 2000 source
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Anderson and Bower 1972 source
- Not checked — no full text of the sourcecites Nielsen 1994 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
What the sources say
- Not checked — no full text of the sourcecites Amabile 1982 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:hedged· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
In the paper
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
§ Peer and Self Assessment in Massive Online Classes > 4. PROVIDING STUDENTS FEEDBACK ON GRADING ACCURACY IMPROVES SUBSEQUENT PERFORMANCE
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
Load-bearingStrength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
In the paper
“students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Nielsen 1993 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Palen 1999 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Little and Bjork 2012
Load-bearingStrength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Pendleton-Jullian 2010
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Kaufman et al. 2008 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Klemmer et al. 2006 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Andrade [2005] source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Conti et al. 1996 source
Strength:boosted· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites Martin 2012 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One has to do with enrollment and retention [Kizilcec et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Kizilcec et al. 2013 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.

93 of 93

The paper relies on this without support.
Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
“To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
“We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”
The paper relies on this without support.
Survey respondents are representative of the class.
“At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”
The authors state this as an assumption.
Grades in different assignments are uncorrelated.
“(assuming grades in different assignments are uncorrelated)”
The paper relies on this without support.
Staff grades are a valid ground truth for assessing peer grading accuracy.
“Holding out 10% of ground truth grades”
“compared to 7.74% in the no-feedback condition”
The paper relies on this without support.
At a physical university, the completion rate for an equivalent class is much higher than in the online class.
“Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”
The paper relies on this without support.
Peers can help instruction itself.
“Fourth, peers can help instruction itself.”

6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

Inconsistent p-value across alpha = .05: F(1, 39) = 2.07, p < .05
Major
The paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .158. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.
Reported p < .05 · recomputed p = .158

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05
“F(1, 39) = 2.07, p < .05”
Consistentt(24) = 19.9reported p < .05 · recomputed p < .001
“t(24) = 19.9,p < .05”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Figure 5 caption names the wrong comparison grade
Minor
The results describe Figure 5 as comparing median peer grades with self-assessed grades, but its caption says the percentages measure agreement with staff grades.

1 of 1

An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Formative assessment and self-regulated learning: A model and seven principles of good feedback practice
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08
(2008) · doi:10.3115/1613715 · Cited by 2,216
Peer Assessment between Students in Colleges and Universities
Topping, K. (1998) · Review of Educational Research · doi:10.2307/1170598 · Cited by 2,173
Self-Regulated Learning and Academic Achievement
Zimmerman, B. J., & Schunk, D. H. (1989) · Springer series in cognitive development · doi:10.1007/978-1-4612-3618-4 · Cited by 1,926
How Designers Think: The Design Process Demystified
Lawson, B. (1990) · Cited by 1,906

Flagged first, then by citations

Show all 97 references

Research works printed but not linked: 11

H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165.
O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.

Sorted by citations

How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
“Peer assessment is a proven method for scaling the grading of a large number of assessments, such as is required in massive open online courses (MOOCs) [14, 19].”
BackgroundMethod
Massive open online course completion rates revisited: Assessment, length and attrition
Jordan, K. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i3.2112 · Cited by 510
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
“Datasets from massive open online courses are a potential source of large-scale data (Kulkarni et al., 2015).”
MOOCs: Expectations and Reality
Tirthali, F. M. H. A. D. (2014) · Scholarly Commons (University of Pennsylvania) · Cited by 303
Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation
Blau, I., & Shamir‐Inbal, T. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.014 · Cited by 227
“…self-assessment that offer new opportunities for ﬂ ipped learning in academia, has shown inaccurate grading e on average 7% higher than staff assessment and prone to be biased e students from the same country received higher grades compared to peers from different countries (Kulkarni et al., 2015).”
Result

See all 390

Authors: Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
Affiliations: Stanford University (US)
Stanford University, and Coursera, Inc., Stanford, CA
Coursera, Inc., Mountain View, CA
ORCID: Kulkarni, Chinmay
Cheng, Justin
Koller, Daphne
Klemmer, Scott R.
Venue: ACM Transactions on Computer-Human Interaction
Publisher: Association for Computing Machinery (ACM)
Published: 2013-12
Volume: 20
Issue: 6
Pages: 1-31
ISSN: 1073-0516 (ISSN-L)
1557-7325
DOI: 10.1145/2505057
OpenAlex: W2088700588
Cited by: 390
References: 97
Citations by year: 2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
FWCI: 199.15
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computer-Human Interaction (publishedVersion)
Funders: Division of Information and Intelligent Systems
Hasso Plattner Design Thinking Program
Topics: Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Social Sciences › Education › Online and Blended Learning
Keywords: peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Peer and self assessment in massive online classes

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Statistics that don’t add up

Statistics recomputed

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks