The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 52 sources its claims cite were checked for support.
Peer and self assessment in massive online classes
Open accessAbstract
Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
89 Own result · 44 Own interpretation · 22 No full text of the sources · 16 No basis given · 12 Sources not found · 6 Own method · 4 Not checked yet · 1 Supported so far · 1 Partly supported · accuracy not yet measured
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Supportedcites Nicol and Macfarlane-Dick 2006 source
In “Delivers high quality information to students about their learning” (p. 9), the paper defines good-quality feedback as information that helps students identify problems and take action to improve their performance. It also recommends timely feedback and corrective advice (p. 10), supporting the claim’s emphasis on accuracy and actionability. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites Boud 2000 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Partly supportedcites Cheshire and Antin 2008 sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A between-subjects experiment found a 0.97% decrease in mean error (6.77% in the experimental group vs. 7.74% in the control group).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students also reported that they felt their peers put less effort into peer assessment than they did (Table III).On a Mann-Whitney U-test, mean ratings were 4.57 for peer effort and 5.46 for their own effort (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
plainn = 3,550 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that giving students feedback about their grading bias increased subsequent accuracy.”
- Also stated in: § 1.4. Contributions · plain
“First, we found that giving students feedback about whether they scored peers high or low increased their subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1.4. Contributions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Over both iterations of the class, 5,876 students submitted at least one assignment and participated in peer assessment.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, students assessed their work 7% higher than staff did.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, to help students provide peers with high-quality personalized feedback, we introduce short, customizable feedback snippets that address common issues with assignments; 67% of students obtained open-ended peer feedback using this method.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 56
- Probably not a substantial claim. 56
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
plainn = 3,550- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The in-person class had 32.8% of submissions with a self-grade within 5% of the staff grade, and 60.8% of submissions within 10% (Figure 6).”
§ Peer and Self Assessment in Massive Online Classes > 3.4. Comparison to In-Person Classes
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was also useful for identifying mistakes and reflection (Figure 8).”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating peers was useful for inspiration and to see other perspectives.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, 55 students left comments expressing their appreciation or receptiveness to this feedback; none expressed resentment.”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of raters per submission helps accuracy but quickly yields diminishing returns (Figure 9).”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, the arithmetic mean, the geometric mean, and a clipped arithmetic mean (that only considers the middle three grades) all do worse than the median.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Creating fortune cookies took a teaching assistant 3 to 4 hours per assignment.”
§ Peer and Self Assessment in Massive Online Classes > 5.1. Methods: Creating Fortune Cookies
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, errors are approximately evenly spread across the median, so adding a constant correction term to the median grade does not significantly improve accuracy either.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because submissions were assessed by multiple students, 94.9% of submissions received at least one piece of written feedback (compared to 83% without cookies);”
§ Peer and Self Assessment in Massive Online Classes > 5.2. Results: How Well Do Fortune Cookies Work?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This effect is consistent when the raters and submitters from the largest student enrollment (United States) are removed, but is smaller (the mean difference drops to 1.98%, t(12, 863) = 2.0, p < 0.05).”
§ Peer and Self Assessment in Massive Online Classes > 3.10. "Patriotic" Grading?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Some rubrics sometimes used a nonparallel grammatical structure across sentences.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Still, some students probably attempted to game their score by strategically overreporting their grade (Figure 18).”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, our early experiments suggest that greater accuracy is possible by weighting different raters' grades differently, an important topic for future work.”
§ Peer and Self Assessment in Massive Online Classes > 6.2. Going Beyond Pass/Fail
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Lewin 2013c
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Five students from across the world participated.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Together, community TAs answered 547 questions on the forum, while staff (three local TAs and the instructor) answered 582 questions.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Anecdotally, we have noticed that students are generally satisfied with their overall grade but are frustrated by inaccurate qualitative feedback from some peers.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“She also participated heavily in the forums and gathered staff-like respect from her peers.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In iteration 1, questions with parallel structure had lesser disagreement, both among peer graders and between the median grade and the self-assessed grade.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The age distribution of the class is remarkably similar across both iterations.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(Recall that the median agreement was 42.% and 65.5%, respectively).”
§ Peer and Self Assessment in Massive Online Classes > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Most fortune cookie feedback was personalized.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Staff rated it low, because”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Peers rated the prototype highly for being”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Students copied snippets of feedback (fortune cookies), pasted them in a textbox, and optionally added an explanation.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Staff praised it as”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(b) Fall 2012 (iteration 2): 17,915 participants”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plainn = 17,915- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students were required to assess five submissions, and 20% of students assessed more than required.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Even though this was a completely online class, students self-organized to meet up in 10 locations around the world including London, San Francisco, New York City, Buenos Aires, Aachen (Germany), and Bangladesh.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Students in the second (Fall 2012) iteration of the class reported a self-grade >5% higher than peer grade more frequently, and so got their self-grade less frequently.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(a) Spring 2012 (iteration 1): 10,190 participants”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plainn = 10,190- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Feedback on grading accuracy reduced the overall error in assessment and made the range of errors smaller.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“26.3% of grades were within 5% of staff grades, and 46.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(b) Same graph for second iteration of the class: 24.0% of such samples were within 5% of the staff grade, and 40.63% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while some peers rated it”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Hearst 2000 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matchedcites Boud 1995
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matchedcites Chi 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Carter et al. 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Not matched
In the paper
“Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Galinsky and Moskowitz 2000 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
What the sources say
- Not checked — no full text of the sourcecites Amabile 1982 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We remind the reader that grading was double-blind, so raters did not see the names of submitters.”
§ Peer and Self Assessment in Massive Online Classes > 3.10. "Patriotic" Grading?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
§ Peer and Self Assessment in Massive Online Classes > 4. PROVIDING STUDENTS FEEDBACK ON GRADING ACCURACY IMPROVES SUBSEQUENT PERFORMANCE
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that students expend the same amount of effort writing feedback, and using fortune cookies allows this effort to be used to add to the fortune cookie text.”
§ Peer and Self Assessment in Massive Online Classes > 5.2. Results: How Well Do Fortune Cookies Work?
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that both these numbers are based on repeated sampling from a smaller number of staff-graded assignments.As such, they are more susceptible to variations in staff grades for a particular submission.”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students sometimes believe that teachers grade on personal taste and focus on currying favor.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Lastly, while peer assessment works well for the large majority of students, students who receive an unfair assessment may lose motivation.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many types of cheating are currently possible and unchecked in online classes.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Nielsen 1993 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These differences require rethinking the design of the classroom.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, providing explicit grading criteria (especially in advance) helps convey to students that grading is fair, consistent, and based on the quality of their work.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The large scale of online classes allows data-driven iterative improvements of classroom materials in ways that small classes may not.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matchedcites Boud 1995
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Palen 1999 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, because community TAs had done well in the class, they possessed enough knowledge to effectively offer information and guidance.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Peer assessment addresses this problem by providing raters with expert-made rubrics, but some differences may persist.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, because they had taken the class recently, they could easily empathize with issues students faced and also could effectively offer social support.”
§ Peer and Self Assessment in Massive Online Classes > 6.6. The Changing Roles of Students
hedgeddirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may state two separate claims as one.
In the paper
“platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The role of teaching staff (TAs) changes too.Instead of spending a majority of their time grading, they spend a large fraction of their time fielding student questions, mentoring students, boosting student morale and autonomous perspective, and making data-driven revisions to class materials.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not matchedcites Little and Bjork 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because of the diversity of submissions, this can be extremely difficult to do a priori.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Schön 1985
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Pendleton-Jullian 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After revision, these rubrics can scale well for both students and other teachers to use.”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, students often have work commitments, and holidays are at different times around the world.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matchedcites Schön 1985
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Kaufman et al. 2008 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Klemmer et al. 2006 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As online classes count for more benefits, such gaming may increase.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, peer assessment imposes a particular schedule on class and limits student flexibility.”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Andrade [2005] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and gaming may help students develop this skill.”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Conti et al. 1996 source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites Martin 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One has to do with enrollment and retention [Kizilcec et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Kizilcec et al. 2013 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Online classes attract students who cannot use traditional universities, such as those working full time.”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Human subjects research was reviewed by the Stanford Institutional Review Board through protocol 25001.”
§ Peer and Self Assessment in Massive Online Classes > Acknowledgements
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More generally, online education requires us to reconceptualize what it means to be a student in many ways.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By removing the constraints of the physical classroom, online classes have made education accessible to many new kinds of students-the new mother, the full-time professional, and the retiree.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Typing one's email address into a webpage is not the same as showing up for the first day of a registrar-enrolled class.It's more like peeking through the window, and what the large number of signups tells us is that lots of people are curious.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
“To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
“We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”
The paper relies on this without support.
Survey respondents are representative of the class.
“At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”
The authors state this as an assumption.
Grades in different assignments are uncorrelated.
“(assuming grades in different assignments are uncorrelated)”
The paper relies on this without support.
Staff grades are a valid ground truth for assessing peer grading accuracy.
“Holding out 10% of ground truth grades”
“compared to 7.74% in the no-feedback condition”
The paper relies on this without support.
At a physical university, the completion rate for an equivalent class is much higher than in the online class.
“Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”
The paper relies on this without support.
Peers can help instruction itself.
“Fourth, peers can help instruction itself.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Techniques such as involvement of community teaching assistants (TAs),21 peer assessment tools,22 well-designed elaborated rubrics with specific content feedback23 and video feedbacks24 are available to help MOOC instructors scale up the provision of perceived personal attention to students.”
From Towards AI-Powered Personalization in MOOC Learning (2017) · cites this paper as [22]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“F(1, 39) = 2.07, p < 0.05”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
Reported p < .05 · recomputed p = .158
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05
“F(1, 39) = 2.07, p < .05”
- Consistentt(24) = 19.9reported p < .05 · recomputed p < .001
“t(24) = 19.9,p < .05”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Peer and Self Assessment in Massive Online Classes > 3.2. Results: Grading Agreement
“In the first iteration of the class, 28.7% of submissions had their median peer grade within 5% of the self-assessed grade, and 44.9% within 10% (Figure 5).”
§ Peer and Self Assessment in Massive Online Classes > B. Sample Rubric
“Figure: 5 Fig. 5 . Fig. 5. (a) Comparison of median peer grades against self-grades.In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.”
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Peer Assessment between Students in Colleges and Universities
Topping, K. (1998) · Review of Educational Research · doi:10.2307/1170598 · Cited by 2,173
Self-Regulated Learning and Academic Achievement
Zimmerman, B. J., & Schunk, D. H. (1989) · Springer series in cognitive development · doi:10.1007/978-1-4612-3618-4 · Cited by 1,926
How Designers Think: The Design Process Demystified
Lawson, B. (1990) · Cited by 1,906
Research works printed but not linked: 11
- H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
- J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
- T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
- T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
- T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
- T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
- T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
- T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
- M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
- J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165.
- O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.
Cited by
Sorted by citations
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
“Peer assessment is a proven method for scaling the grading of a large number of assessments, such as is required in massive open online courses (MOOCs) [14, 19].”
BackgroundMethodMassive open online course completion rates revisited: Assessment, length and attrition
Jordan, K. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i3.2112 · Cited by 510
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
“Datasets from massive open online courses are a potential source of large-scale data (Kulkarni et al., 2015).”
MOOCs: Expectations and Reality
Tirthali, F. M. H. A. D. (2014) · Scholarly Commons (University of Pennsylvania) · Cited by 303
Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation
Blau, I., & Shamir‐Inbal, T. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.014 · Cited by 227
“…self-assessment that offer new opportunities for fl ipped learning in academia, has shown inaccurate grading e on average 7% higher than staff assessment and prone to be biased e students from the same country received higher grades compared to peers from different countries (Kulkarni et al., 2015).”
Result
Metadata
- Authors
- Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
- Affiliations
- Stanford University (US)
- Stanford University, and Coursera, Inc., Stanford, CA
- Coursera, Inc., Mountain View, CA
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2013-12
- Volume
- 20
- Issue
- 6
- Pages
- 1-31
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2088700588
- Cited by
- 390
- References
- 97
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
- FWCI
- 199.15
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Funders
- Division of Information and Intelligent Systems
- Hasso Plattner Design Thinking Program
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Education › Online and Blended Learning
- Keywords
- peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
