The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 52 sources its claims cite were checked for support.
Peer and self assessment in massive online classes
Open accessAbstract
Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 of 52 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Supportedcites Nicol and Macfarlane-Dick 2006 source
In “Delivers high quality information to students about their learning” (p. 9), the paper defines good-quality feedback as information that helps students identify problems and take action to improve their performance. It also recommends timely feedback and corrective advice (p. 10), supporting the claim’s emphasis on accuracy and actionability. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites Boud 2000 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Partly supportedcites Cheshire and Antin 2008 sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A between-subjects experiment found a 0.97% decrease in mean error (6.77% in the experimental group vs. 7.74% in the control group).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students also reported that they felt their peers put less effort into peer assessment than they did (Table III).On a Mann-Whitney U-test, mean ratings were 4.57 for peer effort and 5.46 for their own effort (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that giving students feedback about their grading bias increased subsequent accuracy.”
- Also stated in: § 1.4. Contributions · plain
“First, we found that giving students feedback about whether they scored peers high or low increased their subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1.4. Contributions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Over both iterations of the class, 5,876 students submitted at least one assignment and participated in peer assessment.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, students assessed their work 7% higher than staff did.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, to help students provide peers with high-quality personalized feedback, we introduce short, customizable feedback snippets that address common issues with assignments; 67% of students obtained open-ended peer feedback using this method.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- It may state two separate claims as one. 56
- Probably not a substantial claim. 56
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The in-person class had 32.8% of submissions with a self-grade within 5% of the staff grade, and 60.8% of submissions within 10% (Figure 6).”
§ Peer and Self Assessment in Massive Online Classes > 3.4. Comparison to In-Person Classes
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was also useful for identifying mistakes and reflection (Figure 8).”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating peers was useful for inspiration and to see other perspectives.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, 55 students left comments expressing their appreciation or receptiveness to this feedback; none expressed resentment.”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of raters per submission helps accuracy but quickly yields diminishing returns (Figure 9).”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, the arithmetic mean, the geometric mean, and a clipped arithmetic mean (that only considers the middle three grades) all do worse than the median.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Creating fortune cookies took a teaching assistant 3 to 4 hours per assignment.”
§ Peer and Self Assessment in Massive Online Classes > 5.1. Methods: Creating Fortune Cookies
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, errors are approximately evenly spread across the median, so adding a constant correction term to the median grade does not significantly improve accuracy either.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because submissions were assessed by multiple students, 94.9% of submissions received at least one piece of written feedback (compared to 83% without cookies);”
§ Peer and Self Assessment in Massive Online Classes > 5.2. Results: How Well Do Fortune Cookies Work?
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect is consistent when the raters and submitters from the largest student enrollment (United States) are removed, but is smaller (the mean difference drops to 1.98%, t(12, 863) = 2.0, p < 0.05).”
§ Peer and Self Assessment in Massive Online Classes > 3.10. "Patriotic" Grading?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Some rubrics sometimes used a nonparallel grammatical structure across sentences.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Still, some students probably attempted to game their score by strategically overreporting their grade (Figure 18).”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, our early experiments suggest that greater accuracy is possible by weighting different raters' grades differently, an important topic for future work.”
§ Peer and Self Assessment in Massive Online Classes > 6.2. Going Beyond Pass/Fail
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Lewin 2013c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Five students from across the world participated.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Together, community TAs answered 547 questions on the forum, while staff (three local TAs and the instructor) answered 582 questions.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Anecdotally, we have noticed that students are generally satisfied with their overall grade but are frustrated by inaccurate qualitative feedback from some peers.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“She also participated heavily in the forums and gathered staff-like respect from her peers.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In iteration 1, questions with parallel structure had lesser disagreement, both among peer graders and between the median grade and the self-assessed grade.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The age distribution of the class is remarkably similar across both iterations.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(Recall that the median agreement was 42.% and 65.5%, respectively).”
§ Peer and Self Assessment in Massive Online Classes > Additional information
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Most fortune cookie feedback was personalized.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Staff rated it low, because”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Peers rated the prototype highly for being”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Students copied snippets of feedback (fortune cookies), pasted them in a textbox, and optionally added an explanation.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Staff praised it as”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(b) Fall 2012 (iteration 2): 17,915 participants”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedn = 17,915Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students were required to assess five submissions, and 20% of students assessed more than required.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Even though this was a completely online class, students self-organized to meet up in 10 locations around the world including London, San Francisco, New York City, Buenos Aires, Aachen (Germany), and Bangladesh.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Students in the second (Fall 2012) iteration of the class reported a self-grade >5% higher than peer grade more frequently, and so got their self-grade less frequently.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(a) Spring 2012 (iteration 1): 10,190 participants”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedn = 10,190Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Feedback on grading accuracy reduced the overall error in assessment and made the range of errors smaller.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“26.3% of grades were within 5% of staff grades, and 46.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > Additional information
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(b) Same graph for second iteration of the class: 24.0% of such samples were within 5% of the staff grade, and 40.63% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some peers rated it”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Hearst 2000 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matchedcites Boud 1995
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matchedcites Chi 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Carter et al. 2008 source
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
- Not matched
In the paper
“Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Galinsky and Moskowitz 2000 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
What the sources say
- Not checked — no full text of the sourcecites Amabile 1982 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We remind the reader that grading was double-blind, so raters did not see the names of submitters.”
§ Peer and Self Assessment in Massive Online Classes > 3.10. "Patriotic" Grading?
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
In the paper
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
§ Peer and Self Assessment in Massive Online Classes > 4. PROVIDING STUDENTS FEEDBACK ON GRADING ACCURACY IMPROVES SUBSEQUENT PERFORMANCE
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that students expend the same amount of effort writing feedback, and using fortune cookies allows this effort to be used to add to the fortune cookie text.”
§ Peer and Self Assessment in Massive Online Classes > 5.2. Results: How Well Do Fortune Cookies Work?
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that both these numbers are based on repeated sampling from a smaller number of staff-graded assignments.As such, they are more susceptible to variations in staff grades for a particular submission.”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students sometimes believe that teachers grade on personal taste and focus on currying favor.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lastly, while peer assessment works well for the large majority of students, students who receive an unfair assessment may lose motivation.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many types of cheating are currently possible and unchecked in online classes.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Nielsen 1993 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These differences require rethinking the design of the classroom.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, providing explicit grading criteria (especially in advance) helps convey to students that grading is fair, consistent, and based on the quality of their work.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The large scale of online classes allows data-driven iterative improvements of classroom materials in ways that small classes may not.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
In the paper
“A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matchedcites Boud 1995
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Palen 1999 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, because community TAs had done well in the class, they possessed enough knowledge to effectively offer information and guidance.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Peer assessment addresses this problem by providing raters with expert-made rubrics, but some differences may persist.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, because they had taken the class recently, they could easily empathize with issues students faced and also could effectively offer social support.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
Tests✗ Asserted: fails✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. The authors may not put it forward as true. It may state two separate claims as one.
In the paper
“platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The role of teaching staff (TAs) changes too.Instead of spending a majority of their time grading, they spend a large fraction of their time fielding student questions, mentoring students, boosting student morale and autonomous perspective, and making data-driven revisions to class materials.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not matchedcites Little and Bjork 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because of the diversity of submissions, this can be extremely difficult to do a priori.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Schön 1985
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Pendleton-Jullian 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After revision, these rubrics can scale well for both students and other teachers to use.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, students often have work commitments, and holidays are at different times around the world.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Schön 1985
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Kaufman et al. 2008 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Klemmer et al. 2006 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As online classes count for more benefits, such gaming may increase.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, peer assessment imposes a particular schedule on class and limits student flexibility.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Andrade [2005] source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and gaming may help students develop this skill.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Conti et al. 1996 source
Strength:boosted· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites Martin 2012 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One has to do with enrollment and retention [Kizilcec et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Kizilcec et al. 2013 source
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Online classes attract students who cannot use traditional universities, such as those working full time.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Human subjects research was reviewed by the Stanford Institutional Review Board through protocol 25001.”
§ Peer and Self Assessment in Massive Online Classes > Acknowledgements
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More generally, online education requires us to reconceptualize what it means to be a student in many ways.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By removing the constraints of the physical classroom, online classes have made education accessible to many new kinds of students-the new mother, the full-time professional, and the retiree.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typing one's email address into a webpage is not the same as showing up for the first day of a registrar-enrolled class.It's more like peeking through the window, and what the large number of signups tells us is that lots of people are curious.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
Strength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
“To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
“We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”
The paper relies on this without support.
Survey respondents are representative of the class.
“At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”
The authors state this as an assumption.
Grades in different assignments are uncorrelated.
“(assuming grades in different assignments are uncorrelated)”
The paper relies on this without support.
Staff grades are a valid ground truth for assessing peer grading accuracy.
“Holding out 10% of ground truth grades”
“compared to 7.74% in the no-feedback condition”
The paper relies on this without support.
At a physical university, the completion rate for an equivalent class is much higher than in the online class.
“Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”
The paper relies on this without support.
Peers can help instruction itself.
“Fourth, peers can help instruction itself.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Techniques such as involvement of community teaching assistants (TAs),21 peer assessment tools,22 well-designed elaborated rubrics with specific content feedback23 and video feedbacks24 are available to help MOOC instructors scale up the provision of perceived personal attention to students.”
From Towards AI-Powered Personalization in MOOC Learning (2017) · cites this paper as [22]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Inconsistent p-value across alpha = .05: F(1, 39) = 2.07, p < .05
MajorThe paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .158. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.
Reported p < .05 · recomputed p = .158
“F(1, 39) = 2.07, p < 0.05”
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05
“F(1, 39) = 2.07, p < .05”
- Consistentt(24) = 19.9reported p < .05 · recomputed p < .001
“t(24) = 19.9,p < .05”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Figure 5 caption names the wrong comparison grade
MinorThe results describe Figure 5 as comparing median peer grades with self-assessed grades, but its caption says the percentages measure agreement with staff grades.
“In the first iteration of the class, 28.7% of submissions had their median peer grade within 5% of the self-assessed grade, and 44.9% within 10% (Figure 5).”
“Figure: 5 Fig. 5 . Fig. 5. (a) Comparison of median peer grades against self-grades.In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.”
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Peer Assessment between Students in Colleges and Universities
Topping, K. (1998) · Review of Educational Research · doi:10.2307/1170598 · Cited by 2,173
Self-Regulated Learning and Academic Achievement
Zimmerman, B. J., & Schunk, D. H. (1989) · Springer series in cognitive development · doi:10.1007/978-1-4612-3618-4 · Cited by 1,926
How Designers Think: The Design Process Demystified
Lawson, B. (1990) · Cited by 1,906
Flagged first, then by citations
Research works printed but not linked: 11
- H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
- J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
- T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
- T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
- T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
- T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
- T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
- T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
- M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
- J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165.
- O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.
Cited by
Sorted by citations
Items 301–388 of 388
IT-gestütztes Peer-Assessment zur Schulung von Kompetenzen in der Lehre
Lehmann, K., & Söllner, M. (2019) · Kompetenzmanagement in Organisationen · doi:10.1007/978-3-662-59390-5_6 · Cited by 1
Exploring Factors Influencing the Adoption of Massive Open Online Courses for Open Distance Learning
Olugbara, T. C., & Letseka, M. (2019) · Cited by 1
L’évaluation comme objet de recherche dans les MOOC : prise en compte du contenu
Zaïd, A., Babori, A., & Fassi-Fihri, H. (2019) · Questions vives recherches en éducation · doi:10.4000/questionsvives.3807 · Cited by 1
Anvari, F., Tran, H. M. T., & Richards, D. (2021) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet52601.2021.00016 · Cited by 1
“To train the students in assessing their peers’ work, Kulkarni et al. [43] used a system in which the students initially assessed teachers’ assessed materials and hence, their competence was assessed before they could assess other students.”
BackgroundIntegrating online assessment into blended training form: a case study at Lac Hong University
Le, T. P., & Aves, G. E. (2021) · Technical Education Science/Giáo dục Kỹ thuật · doi:10.54644/jte.62.2021.68 · Cited by 1
A Graph Analysis Method to Improve Peer Grading Accuracy for Blended Teaching Courses
Du, X., Wang, X., & Ma, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3136352 · Cited by 1
Measuring Cultural Dimensions of Learning in Online Courses
Ji, Y.-C., Li, Y., Krasny, M., & Kizilcec, R. F. (2022) · ACM Conference on Learning @ Scale · doi:10.1145/3491140.3528333 · Cited by 2
“3528333 levels of cultural diversity in these online course can offer valuable opportunities for social learning and feedback [10, 10].”
BackgroundCLASSROOM ASSESSMENT PRACTICES OF ENGLISH TEACHERS AT SECONDARY SCHOOL LEVEL IN PUNJAB PAKISTAN
Ajmal, A., Khalid, A., & Ahmad, S. N. (2022) · Pakistan Journal of Social Research · doi:10.52567/pjsr.v4i04.846 · Cited by 2
Iriani, T., Anisah, A., & Luthfiana, Y. (2023) · Advances in Social Science, Education and Humanities Research/Advances in social science, education and humanities research · doi:10.2991/978-2-38476-056-5_24 · Cited by 1
Ihejirika, N. O., Ogana, M., & OYISE, U. E. (2023) · Direct Research Journal of Social Science and Educational Studies · doi:10.26765/drjsses19591767 · Cited by 1
The role of peer dialogue as disruptor in critical ethical analysis for computing students
Ciminski, A., & Wallace, C. (2023) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342929 · Cited by 1
“For others, it is a means of scaling up massive online courses with minimal instructor involvement [18].”
BackgroundExploring the Integration of Social Practice into MOOC Peer Assessment
Perdue, M., Sandland, J. G., Joshi, A. A., & Liu, J. (2024) · IEEE Digital Education and MOOCS Conference (DEMOcon) · doi:10.1109/democon63027.2024.10747943 · Cited by 1
Designing AI-driven Curriculum and Assessment
(2024) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/979-8-3693-7190-9.ch012 · Cited by 1
Paul, U., Shah, S., Mylavarapu, S. V., et al. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15743-0_27 · Cited by 1
Pessoa, A. S. M., Coelho, L., & Fernandes, R. (2015) · Advances in higher education and professional development book series · doi:10.4018/978-1-4666-7304-5.ch005 · Cited by 2
Simulated Learners in Peers Assessment for Introductory Programming Courses.
Barbosa, A. D. A., & Costa, E. D. B. (2015) · AIED Workshops · Cited by 1
Various Aspects of the Human Factor in Online Education
Nikolaeva, S. G., & Nathtan-Roberts, D. (2016) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601297 · Cited by 1
Primary assessment activity and learner perceptions of attitude change in four MOOCs
Watson, S. L., Watson, W. R., & Kim, W. (2017) · Educational Media International · doi:10.1080/09523987.2017.1384165 · Cited by 1
“However, much like other automated approaches, this type of assessment can be difficult to develop and could still be limiting for certain disciplines as it does not fully support the “open-ended work that is a hallmark of education in creative fields like design” (Kulkarni et al., 2015, p. 33:2).”
BackgroundMethodToward the Design of a Cognitive Tutor for Algebra with Gamification: A Survey of State-of-the-Art
Blanca-Estela, P.-M., González-Calleros, J. M., Guerrero-García, J., & Reyes-García, C.-A. (2017) · Research in Computing Science · doi:10.13053/rcs-145-1-5 · Cited by 1
Student Perception of Scalable Peer-Feedback Design in Massive Open Online Courses
Kasch, J., van Rosmalen, P., Löhr, A., Ragas, A. M. J., & Kalz, M. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-97807-9_5 · Cited by 1
“…ratios and more diverse student pro fi les challenge higher education and in fl uence the curriculum design in several ways such as a decrease in personal teacher feedback and a decrease of creative assignments in which students require personal feedback on their text and or design [1, 5, 11, 12].”
BackgroundInfluentialImplementation of methodological strategies, attitudes and instruments as a PBL resource
Valdez, M. T., Ferreira, C. M., & Barbosa, F. P. M. (2018) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet.2018.8424794 · Cited by 1
“The teacher evaluated the different group performance and took into consideration the peer evaluation for the group final grade [17-18].”
BackgroundScaling up Student Assessment: Issues and Solutions
van Vliet, P. (2018) · Cited by 1
Aqui, J., & Hosein, M. (2021) · IEEE International Conference on Mobile Networks and Wireless Communications (ICMNWC) · doi:10.1109/icmnwc52512.2021.9688358 · Cited by 1
Integrating online assessment into blended training form: a case study at Lac Hong University
Le, T. P., & Aves, G. E. (2021) · Technical Education Science/Giáo dục Kỹ thuật · doi:10.54644/jte62202168 · Cited by 1
“In recent years, many researchers have studied online assessment (Robles & Braathen, 2002; Vonderwell & Alderman 2007; Blender, 2012; Swan et al, 2006; Kulkarni et al, 2013).”
BackgroundCollaborative Calibrated Peer Assessment in Massive Open Online Courses
Boudria, A., Lafifi, Y., & Bordjiba, Y. (2019) · IGI Global eBooks · doi:10.4018/978-1-7998-0420-8.ch065 · Cited by 1
Perceptions of MOOC Learning for Employability: Public Education as Microcosm
Webb, M. K. (2015) · iDEA - Drexel University Institutional Repository · doi:10.17918/etd-6645 · Cited by 1
Spam elimination and bias correction : ensuring label quality in crowdsourced tasks.
Lyu, L. (2018) · ThinkIR: The University of Louisville's Institutional Repository (University of Louisville) · doi:10.18297/etd/3054 · Cited by 2
“[161] found in their research that well-designed rubrics can help lower the variance and improve accuracy in the grading process, with peer and self assessment, for massive online classes.”
BackgroundThe impact of student facilitation on student engagement in asynchronous online discussion
Bowen, R. T. (2019) · doi:10.33915/etd.3779 · Cited by 1
“Peer assessment can enhance face-to-face discussion (Boud, 1995), so it is important to explore how that translates to an online discussion environment (Kulkarni et al., 2013).”
BackgroundMechanical TA 2: A System for Peer Grading with TA SupportPreprint
Zarkoob, H., Abdolhosseini, F., & Leyton-Brown, K. (2021) · arXiv.org · doi:10.48550/arxiv.2101.10078 · Cited by 2
“This is appropriate for courses with extremely inadequate numbers of TAs, such as free massive, open, online courses (MOOCs) (Kulkarni et al. , 2013).”
BackgroundEducating Lawyers in the Digital Age: From Theory to PracticePreprint
Kryvoi, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5356738 · Cited by 1
Assessing the Creativity of Designs at Scale
MacLellan, C. J. (2015) · ACM SIGCHI Conference on Creativity and Cognition (C&C) · doi:10.1145/2757226.2764770
MOOCs in business administration - An overview of assessment practice
Gherib, C., Dujardin, J.-M., & Verpoorten, D. (2016)
Strategyproof Peer Selection.Preprint
Aziz, H., Lev, O., Mattei, N., et al. (2016) · arXiv (Cornell University)
Non-native text analysis with Syntactic Diff, a general comparative text mining framework
Massung, S. (2015) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Automated and Scalable Assessment: Present and Future
Gehringer, E. F. (2015) · ASEE Annual Conference and Exposition · doi:10.18260/p.23609
“[13] found that 40% of assignment grades derived from peer P ge 26270.”
BackgroundPartner in Speak: Peer Coaching and Rating Through Online Synchronous Communication for ESL Learners
Krishnan, S., & Abdullah, M. R. (2016) · Springer eBooks · doi:10.1007/978-981-10-0908-2_11
Teaching and Learning Methods at the Open University of Japan
Aoki, K. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-803637-2.00006-3
Matrix completion for SPOC student peer-grading evaluation
Bin, X., Li, K., Gao, K., et al. (2017) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3063955.3063963
“The common way to deal with so many submissions is peer grading[5, 11, 14] where assignment submission of any student is assigned to 5 or more students in the same course to be graded.”
BackgroundIncorporating SPOCs Platform in Innovation and Entrepreneurship Education
Shengbo, S. (2017) · doi:10.2991/emcs-17.2017.108
“(2013)[8] verified that the hybrid of self-assessment and peer-assessment mechanism can improve online course performance and enhance self-efficacy, which played a significant role to cultivate the students’ abilities.”
BackgroundMotivational Design in Chemistry MOOCs: Applying the ARCS Model
Li, K. (2017) · ACS symposium series · doi:10.1021/bk-2017-1261.ch003
Reputation Systems and Incentives Schemes for Quality Control in Crowdsourcing
Shavlovsky, M. (2017) · eScholarship (California Digital Library)
The Right Content at the Right Time
Fraser, C. A. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3266037.3266127
“good feedback can be hard to generate, and people are not consistently skilled in doing so [4].”
BackgroundMethods for Ordinal Peer GradingPreprint
Raman, K., & Joachims, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.3656
“In peer grading, students — not instructors or TAs — provide feedback on the work of other students in their class [14, 22], meaning that the number of “graders” naturally grows with the number of students.”
BackgroundWhat You Sow, So Shall You Reap! Toward Preselection Mechanisms for Macrotask Crowdsourcing
Gadiraju, U., & Zhuang, M. (2019) · Human-computer interaction series · doi:10.1007/978-3-030-12334-5_6
Students’ perception of summative peer review grading
Zrnić, L., Korov, L., Petrović, J., et al. (2020) · doi:10.23919/mipro48935.2020.9245112
Fostering Student Engagement in a Mobile Formative Assessment System for High-School Economics
Lazarinis, F., & Kanellopoulos, D. N. (2021) · Social Education Research · doi:10.37256/ser.222021747
Applying Lexical Density and Lexical Diversity for developing Writing Quality and Quantity
Mohamed, F. S., & Abdallah, D. T. (2021) · مجلة کلیة التربیة بالإسماعیلیة · doi:10.21608/jfes.2021.173721
“Pupils can have a more personalized learning experience through peer feedback and, as such, they can perform better on assessments (Kulkarni et al., 2015).”
BackgroundInsights from Peer Reviewing in Large University Courses
Luckner, N., & Purgathofer, P. (2021) · doi:10.1145/3507923.3507955
“studies have shown that peer reviewing builds problem solving skills and encourages reflection, self awareness [35] and critical thinking [21].”
BackgroundInfluentialEasier Said or Easier Done? Exploring the Relative Merits of Common Feedback Presentations
Crain, P. A., & Bailey, B. P. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512933
“Templates [3, 14, 39] and rubrics [37, 69] have been shown to help non-experts write feedback comparable in quality to that of domain experts.”
BackgroundFisher, D. (2014) · Springer eBooks · doi:10.1007/978-3-319-04033-2_60-1
Assessment Activities in Massive Open On-Line Courses
Muñoz‐Merino, P. J., Ruipérez‐Valiente, J. A., Moreno, J. L. S., & Kloos, C. D. (2019) · IGI Global eBooks · doi:10.4018/978-1-7998-0420-8.ch029
Suarez-Saet, M. S. R. (2022) · South Florida Journal of Development · doi:10.46932/sfjdv3n6-010
“(Kulkarni et al., 2013). This also coincides with Meek et al., (2016) who pointed out that doing the evaluation of the work of their peers has advantages on the part of the learners because they are being expose to solutions, strategies and points of view that they would not otherwise see.”
BackgroundInfluentialWaduge, N. B. (2022) · Psych-Talk · doi:10.53841/bpstalk.2022.1.103.30
Febrianto, T., Fitriady, G., Sari, Z. N., & Tomi, A. (2023) · Sport Science and Health · doi:10.17977/um062v5i12023p45-52 · Cited by 1
“…hasil yang persentase rata rata nilai siswa 7% lebih tinggi dari penilaian guru, dan kesalahan penilaian yang dilakukan oleh siswa yaitu 9,9% (Kulkarni et al., 2015) 2) penelitian berjudul Online Self-Assessment and PeerAssessment as a Tool to Enhance Student-Teachers’ Assessment Skills…”
BackgroundInteractive Worksheet Development in Mobile Learning Environment
Cholifah, P. S., & Nafsi, V. S. (2021) · Advances in Social Science, Education and Humanities Research/Advances in social science, education and humanities research · doi:10.2991/assehr.k.211210.033
“Related to the assessment procedure, studies show some positive impacts of peer assessment, even more so at the college level [16], [17].”
Peer support dan career decision making self-efficacy pada mahasiswa tingkat akhir di Malang
Ikrima, J., & Tantiani, F. F. (2022) · Jurnal Psikologi Terapan dan Pendidikan · doi:10.26555/jptp.v4i2.24517
COMPARISON OF STUDENTS’ SATISFACTION REGARDING BLENDED LEARNING AND CLASSROOM LEARNING
Hussain, M. A. (2023) · Pakistan Journal of Educational Research · doi:10.52337/pjer.v6i2.785
“E-learning, often called “remote learning” is facilitated by widespread computing and networking gear and software (Kulkarni et al., 2013).”
BackgroundKarishma, K., & Raghuwaiya, K. (2023) · American Journal of Education and Technology · doi:10.54536/ajet.v2i4.1998
“Peers grades correlated highly with staff-assigned grades (Kulkarni, et al ., 2013) and pre-service teachers could observe different performances consistently (Kurnaz, 2021).”
BackgroundDreifuss-Serrano, C., Herrera, P. C., Reategui, J. L., & Braida, F. (2023) · IEEE Learning With MOOCS (LWMOOCS) · doi:10.1109/lwmoocs58322.2023.10306238
“• Case study: focuses on one specific MOOC and its several components, with emphasis on assessment [44], or course design [45].”
Backgroundde Campos, D. S., & Ferreira, D. J. (2024) · Journal of Engineering Research · doi:10.22533/at.ed.1317412409017
de Campos, D. S., & Ferreira, D. J. (2024) · doi:10.22533/at.ed.4332405029
MÃO ELETROMECÂNICA: PESQUISA E CONSTRUÇÃO DE PROTOTIPO
Mendes, A. E. M., Simas, G. W., Serezina, I. R., Fantoni, J. V., Paz, J. O. M., Lorena, L. G., Marciano, L. P., & da Costa, S. S. (2024) · doi:10.22533/at.ed.4332405025
A BIOTECNOLOGIA E A QUALIDADE DO SOLO
Barreto, A. V. R., dos Santos, E. L., de Farias, M. T. P., & Cavalcante, R. F. (2024) · doi:10.22533/at.ed.4332405028
Chaves, J. A. F., & da Silva, G. F. (2024) · doi:10.22533/at.ed.4332405021
ANÁLISE DE PERFIS DE MANCHA DE SANGUE: ACIDENTE OU HOMICÍDIO?
Riella, H. L., & Carneiro, L. A. L. (2024) · doi:10.22533/at.ed.4332405023
Panosso, C. E., & Araújo, A. K. D. (2024) · doi:10.22533/at.ed.43324050210
POTENCIAL CICATRIZANTE DE Passiflora ssp.: UMA REVISÃO SISTEMÁTICA
Fragoso, M. B. T., de Mendonça, E. L. S. S., da Silva, M. O., de Oliveira, R. L. B., Lopes, R. P. C., Valentim, I. B., Goulart, M. O. F., & Xavier, J. A. (2024) · doi:10.22533/at.ed.4332405027
Research on Learner Evaluations of Online Chinese Courses
Jia, Y., & Na, R. (2025) · 教育学文摘 · doi:10.63313/edu.8027
Self-Reflective Crowds: Surfacing Wisdom through Emergent Scaffolding
Qiu, R. N., Vadaparty, A., Vintha, S., & Dow, S. P. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737478
Assess or Discuss: Comparing Peer Assessment and Online Discussion for Enhancing Learning at Scale
Ou, C., & Joyner, D. A. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3733927
“Comparative studies have examined peer assessment alongside other feedback modalities, such as self-assessment [19], expert assessment [21], online vs. offline contexts [18], and automated systems [24, 28].”
Enhancing Accuracy in Peer Assessment by Addressing Bias and Outliers Through Calibration Method
Brkić, L., Mekterović, I., & Fertalj, M. (2025) · MIPRO ICT and Electronics Convention (MIPRO) · doi:10.1109/mipro65660.2025.11131879
Cognitive-Aware Peer Assessment: Design Implications from a Classroom Deployment
Bouskila, N., & Dery, L. (2025) · Annals of Computer Science and Information Systems · doi:10.15439/2025f8584
“Previous studies in large-scale peer review, such as in MOOCs, have documented variability in rater performance over time [13, 17].”
ReSAN: Relation-Sensitive Graph Representation Learning for Peer Assessment in Educational Scenarios
Ma, X., Fang, Y., Gu, Y., Zhou, S., & Yang, S. (2025) · Mathematics · doi:10.3390/math13223664
“Peer assessment networks are characterized by nuanced relation types [46], sparse interactions [47], and evolving grading behaviors—conditions [48] under which generic relation-aware GNNs may not directly transfer.”
23312 Artificial intelligence impact on education: Opportunities and risks
Blazheska-Tabakovska, N., & Savoska, S. (2025) · De Gruyter eBooks · doi:10.1515/9783111595306-012
Bakken, J. P., & Estes, T. S. (2024) · Emerald Group Publishing Limited eBooks · doi:10.1108/979-8-88730-618-620251010
Paul, U., Mantravadi, A., Shah, J., Shah, S., Mylavarapu, S. V., Rashid, M. P., & Gehringer, E. F. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328205
“In this context, aggregating rating or category-based responses is typically straightforward and can be accomplished using simple statistical methods such as mean or median [96], [97], or through linear models [38], [98].”
Li, J., Ahooie, N. N., Paananen, V., Visuri, A., Szabó, D., Kuutila, M., Yatani, K., Mäntylä, M., & Hosio, S. J. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2619100
“It helps individuals to reflect on their understanding and progress (Kulkarni et al. 2013).”
Gusev, S., & Sadykova, M. V. (2026) · Open Education · doi:10.21686/1818-4243-2026-1-23-36
Children With ADHD Diagnoses and Adverse Childhood Experiences in A Public Mental Health System
Cassidy, J., & Altschuler, M. R. (2026) · Neurodiversity · doi:10.1177/27546330261430021
PAStudio: A Peer Assessment Pedagogical Tool for Open-ended Assignments
Zhang, L., Zong, Y., Chen, T., Ma, Y., Zhu, J., Li, J., & Zhang, L. (2026) · ACM Transactions on Computing Education · doi:10.1145/3811024
“Kulkarni et al. [24–26] built a system, PeerStudio, that helps with assessment tasks in online courses.”
Wirgler, C., Duncan, A. S., Lytle, N., et al. (2026) · doi:10.1145/3774398.3811585
‘Ghostcrafting AI’: Under the Rug of Platform Labor
Rahman, A. T. M. M., & Sultana, S. (2026) · ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3811242.3819087 · Cited by 2
Metadata
- Authors
- Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
- Affiliations
- Stanford University (US)
- Stanford University, and Coursera, Inc., Stanford, CA
- Coursera, Inc., Mountain View, CA
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2013-12
- Volume
- 20
- Issue
- 6
- Pages
- 1-31
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2088700588
- Cited by
- 390
- References
- 97
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
- FWCI
- 199.15
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Funders
- Division of Information and Intelligent Systems
- Hasso Plattner Design Thinking Program
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Online and Blended Learning
- Keywords
- peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
