Cited by 390Student Assessment and FeedbackOnline Learning and AnalyticsOnline and Blended Learning

The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 52 sources its claims cite were checked for support.

Peer and self assessment in massive online classes

Open access

Kulkarni, Chinmay, Wei, Koh Pang, Le, Huy, Chia, Daniel, Papadopoulos, Kathryn, Cheng, Justin, Koller, Daphne, Klemmer, Scott R.·2013·ACM Transactions on Computer-Human Interaction

DOI 10.1145/2505057

Open this paper in CoChat — free account Open PDF

Abstract

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Integrity signals

Registry

Crossref · 10.1145/2505057

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

2 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

2 of 52 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Supported
  In “Delivers high quality information to students about their learning” (p. 9), the paper defines good-quality feedback as information that helps students identify problems and take action to improve their performance. It also recommends timely feedback and corrective advice (p. 10), supporting the claim’s emphasis on accuracy and actionability. (LLM, conf=0.93)
  cites Nicol and Macfarlane-Dick 2006 source
- Not checked — no full text of the sourcecites Boud 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Partly supported
  The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)
  cites Cheshire and Antin 2008 sourceExperimental — accuracy not yet measured
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students also reported that they felt their peers put less effort into peer assessment than they did (Table III).On a Mann-Whitney U-test, mean ratings were 4.57 for peer effort and 5.46 for their own effort (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that giving students feedback about their grading bias increased subsequent accuracy.”
- Also stated in: § 1.4. Contributions · plain
  “First, we found that giving students feedback about whether they scored peers high or low increased their subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, students assessed their work 7% higher than staff did.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.

Show all 102 claims

It may state two separate claims as one. 56
Probably not a substantial claim. 56
The quote only partly states this claim. 1

A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Load-bearingStrength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was also useful for identifying mistakes and reflection (Figure 8).”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating peers was useful for inspiration and to see other perspectives.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
In the paper
“Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Lewin 2013c
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Zimmerman and Schunk 2001 source
- Not checked — no full text of the sourcecites Pintrich 1995 source
- Not checked — no full text of the sourcecites Pintrich and Zusho 2007 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Hearst 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Chi 2009
- Not checked — no full text of the sourcecites Carter et al. 2008 source
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Galinsky and Moskowitz 2000 source
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Anderson and Bower 1972 source
- Not checked — no full text of the sourcecites Nielsen 1994 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
What the sources say
- Not checked — no full text of the sourcecites Amabile 1982 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:hedged· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
In the paper
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
§ Peer and Self Assessment in Massive Online Classes > 4. PROVIDING STUDENTS FEEDBACK ON GRADING ACCURACY IMPROVES SUBSEQUENT PERFORMANCE
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
Load-bearingStrength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
In the paper
“students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Nielsen 1993 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Palen 1999 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Little and Bjork 2012
Load-bearingStrength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Pendleton-Jullian 2010
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Kaufman et al. 2008 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Klemmer et al. 2006 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Andrade [2005] source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Conti et al. 1996 source
Strength:boosted· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites Martin 2012 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One has to do with enrollment and retention [Kizilcec et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Kizilcec et al. 2013 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.

93 of 93

The paper relies on this without support.
Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
“To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
“We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”
The paper relies on this without support.
Survey respondents are representative of the class.
“At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”
The authors state this as an assumption.
Grades in different assignments are uncorrelated.
“(assuming grades in different assignments are uncorrelated)”
The paper relies on this without support.
Staff grades are a valid ground truth for assessing peer grading accuracy.
“Holding out 10% of ground truth grades”
“compared to 7.74% in the no-feedback condition”
The paper relies on this without support.
At a physical university, the completion rate for an equivalent class is much higher than in the online class.
“Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”
The paper relies on this without support.
Peers can help instruction itself.
“Fourth, peers can help instruction itself.”

6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

Inconsistent p-value across alpha = .05: F(1, 39) = 2.07, p < .05
Major
The paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .158. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.
Reported p < .05 · recomputed p = .158

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05
“F(1, 39) = 2.07, p < .05”
Consistentt(24) = 19.9reported p < .05 · recomputed p < .001
“t(24) = 19.9,p < .05”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Figure 5 caption names the wrong comparison grade
Minor
The results describe Figure 5 as comparing median peer grades with self-assessed grades, but its caption says the percentages measure agreement with staff grades.

1 of 1

An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Formative assessment and self-regulated learning: A model and seven principles of good feedback practice
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08
(2008) · doi:10.3115/1613715 · Cited by 2,216
Peer Assessment between Students in Colleges and Universities
Topping, K. (1998) · Review of Educational Research · doi:10.2307/1170598 · Cited by 2,173
Self-Regulated Learning and Academic Achievement
Zimmerman, B. J., & Schunk, D. H. (1989) · Springer series in cognitive development · doi:10.1007/978-1-4612-3618-4 · Cited by 1,926
How Designers Think: The Design Process Demystified
Lawson, B. (1990) · Cited by 1,906

Flagged first, then by citations

Show all 97 references

Research works printed but not linked: 11

H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165.
O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.

Sorted by citations

Items 201–300 of 388

Theory-Driven Design of an IT-Based Peer Assessment to Assess High Cognitive Levels of Educational Objectives in Large-Scale Learning Services
Lehmann, K., & Leimeister, J. M. (2015) · Alexandria (UniSG) (University of St.Gallen) · Cited by 7
Designing Scalable and Sustainable Peer Interactions Online
Kulkarni, C. E., Kotturi, Y., Bernstein, M. S., & Klemmer, S. (2016) · Understanding innovation · doi:10.1007/978-3-319-40382-3_14 · Cited by 7
“Rubrics effectively provide students feedback on the current state of their work for many open-ended assignments, such as writing (Andrade 2001; Andrade 2005), design (Kulkarni et al. 2013), and art (Tinapple et al. 2013).”
BackgroundInfluentialSelf-citation
The MOOC and student assessment: systematic review (2012-2016)
Vera, M. D. M. S., Calatayud, V. G., & Espinosa, M. P. P. (2017) · tic revista d innovació educativa · doi:10.7203/attic.18.10013 · Cited by 7
How Features of a Civic Design Competition Influences the Collective Understanding of a Problem
McInnis, B. J., Xu, X. X., & Dow, S. P. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274389 · Cited by 7
“systems developed to crowdsource design critique might be adapted to help people articulate their understanding of a design problem [38, 45, 46], while active listening systems, like Reflect [37], might be adapted to provide conversational support for mentoring.”
Background
Using image recognition to automatically assess programming tasks with graphical output
Muuli, E., Tõnisson, E., Lepp, M., Luik, P., Palts, T., Suviste, R., Papli, K., & Säde, M. (2020) · Education and Information Technologies · doi:10.1007/s10639-020-10218-z · Cited by 7
“As the organizers of MOOC are usually unable to assess and give feedback in person, it is possible to organize peer and self-assessment (Huisman et al. 2016; Kulkarni et al. 2013), which could be also used in programming courses (Wang et al. 2016).”
Background
Adjacent Display of Relevant Discussion Helps Resolve Confusion
Yarmand, M., Palani, S., & Klemmer, S. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449217 · Cited by 7
“Besides, in online education students rely on peer-based interactions for motivation and learning [20].”
BackgroundSelf-citation
Scaling Up Data Science Course Projects: A Case Study
Bhavya, B., Xiao, J., & Zhai, C. (2021) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3430895.3460168 · Cited by 7
“To continuously guide students and provide rapid feedback [6], we divided the project into three stages: 1) initial stage where students were asked to form groups, choose their topics and submit a short proposal on the feasibility, significance, and”
BackgroundMethod
How Effective Can Simple Ordinal Peer Grading Be?
Caragiannis, I., Krimpas, G. A., & Voudouris, A. A. (2016) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2940716.2940748 · Cited by 7
“The only solution that seems consistent to the MOOCs vision is known as peer grading [10, 13, 16], according to which the grading task is outsourced to the students that participated to the exam themselves.”
Background
Motivating Learners in Massive Open Online Courses: A Design-Based Research Approach.
Li, K. (2015) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 7
Design and Evaluation of an IT-Based Peer Assessment to Increase Learner Performance in Large-Scale LecturesPreprint
Lehmann, K., Söllner, M., & Leimeister, J. M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.3159160 · Cited by 7
“First observations show that peers are indeed able to provide valuable feedback (Dochy et al. 1999; Falchikov and Goldfinch 2000) and that evaluations by the peer group agree with the lecturer’s evaluation (Kulkarni et al. 2013).”
Background
“I Know It When I See It”: How Experts and Novices Recognize Good Design
Tanner, K., & Landay, J. A. (2018) · Understanding innovation · doi:10.1007/978-3-319-97082-0_13 · Cited by 6
A Systematic Quantitative and Qualitative Analysis of Participants’ Opinions on Peer Assessment in Surveys and Course Forum Discussions of MOOCs
Staubitz, T., & Meinel, C. (2020) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon45650.2020.9125089 · Cited by 6
“PA has been introduced to MOOCs in 2013 in Scott Klemmer’s course on human-centered interaction design on the coursera platform [7].”
MethodResultInfluential
REFLEKSI PENILAIAN KONTRIBUSI PADA TEAM-BASED PROJECT SECARA DARING
Nasir, R., & Maknun, C. L. (2022) · Aksioma · doi:10.22487/aksioma.v11i1.1905 · Cited by 11
“Mahasiswa pun mengevaluasi hasil proyeknya sendiri, usaha, motivasi, ketertarikan dan tingkat produktivitas (Kulkarni et al., 2013).”
Background
How Personality Traits Affect Peer Assessment in Distance Learning
Alves, T., Sousa, F., Gama, S., Jorge, J., & Gonçalves, D. (2023) · Technology, Knowledge and Learning · doi:10.1007/s10758-023-09694-2 · Cited by 9
“…specific criteria to support the assessment, students’ different backgrounds and knowledge levels question the eligibility and grading accuracy (Kulkarni et al., 2013; Na and Liu, 2019), leading peer assessment itself, as a valid assessment form, to be sometimes challenged by course…”
Influential
A Peer Grading Tool for MOOCs on Programming
Wei, Z., & Wu, W. (2015) · International Conference of Young Computer Scientists, Engineers and Educators · doi:10.1007/978-3-662-46248-5_46 · Cited by 6
Negotiating Individual Learner Models in Contexts of Peer Assessment and Group Learning
Susan Bull, A., & Al-Shanfari, L. (2015) · University of Birmingham Research Portal (University of Birmingham) · Cited by 6
Assessment Activities in Massive Open On-Line Courses
Muñoz‐Merino, P. J., Ruipérez‐Valiente, J. A., Moreno, J. L. S., & Kloos, C. D. (2015) · Advances in higher education and professional development book series · doi:10.4018/978-1-4666-8279-5.ch008 · Cited by 6
Crafting Policy Discussion Prompts as a Task for Newcomers
McInnis, B. J., Leshed, G., & Cosley, D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274390 · Cited by 6
“Looking back, the designs share many characteristics with rubrics and peer feedback systems designed for MOOCs [35]: the instructions align closely with the sub-criteria; they give definitions, considerations, and examples to illustrate each of the criteria; and the interface provides people with a…”
Background
Der Wert von IT-gestütztem Peer Assessment zur Unterstützung des Lernens in einer Universitären Massenlehrveranstaltung
Lehmann, K., Söllner, M., & Leimeister, J. M. (2015) · Alexandria (UniSG) (University of St.Gallen) · Cited by 6
Crowd Research: Open and Scalable University Laboratories
Vaish, R., Gaikwad, S. S., Kovacs, G., Veit, A., Krishna, R., Arrieta-Ibarra, I., Simoiu, C., Wilber, M. J., Belongie, S., Goel, S., et al. (2018) · Understanding innovation · doi:10.1007/978-3-319-97082-0_7 · Cited by 8
The Impact of Rubric-Based Peer Assessment on Feedback Quality in Blended MOOCs
Yousef, A. M. F., Wahid, U., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-29585-5_27 · Cited by 6
JAG: A Crowdsourcing Framework for Joint Assessment and Peer Grading
Labutov, I., & Studer, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10631 · Cited by 6
“The advantage of peer-grading is its flexibility to students’ submissions, which may range from short answers, to essays, diagrams, code, or entire projects (Kulkarni et al. 2015).”
Background
A quantitative revision method to improve usability of self-and peer assessment in MOOCs
Na, J., & Liu, Y. (2019) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3321408.3322846 · Cited by 6
“In [11], self-and peer assessment was firstly used in an online massive class, where 30,000+ students participated from 124 countries.”
Method
A cognitive diagnosis framework based on peer assessment
He, Y., Hu, X.-Y., & Sun, G. (2019) · ACM Turing Celebration Conference · doi:10.1145/3321408.3322850 · Cited by 6
Untitled work
(2019) · Educational Data Mining · Cited by 7
Improving the accuracy of middle school students' self-assessment, peer assessment, and mathematics achievement.
Popelka, E. (2015) · doi:10.18297/etd/2324 · Cited by 6
“…the research that has produced relatively higher levels of accurate self- and peer assessment, including that obtained subsequent to successful implementation of interventions, has taken place in a higher-education setting (Kulkarni et al., 2013; Lawson et al., 2012; Wagner, Suh, & 11 Cruz, 2011).”
BackgroundMethodResultInfluential
Supporting Workers in Developing Effective Collaboration Skills for Complex Work
Huang, E. J., Garg, K., Gómez-Zará, D., Hui, J. S., Kulkarni, C. E., Massimi, M., Churchill, E. F., & Gerber, E. M. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3611290 · Cited by 5
“This includes developing supportive relationships with peers and mentors [6, 17, 23], identifying and exchanging resources [15, 16], communicating what support is needed [15, 17, 18], and reflecting on how one’s work practices can be improved [25].”
BackgroundSelf-citation
Creating a network between community college students in first-year seminars and in capstone courses using writing assignments
Jaafar, R., Boumlik, H., & Alberts, I. L. (2018) · Cogent Education · doi:10.1080/2331186x.2018.1548051 · Cited by 5
“Cooperative learning is a successful strategy in improving students’ understanding (Falchikov, 2013; Kulkarni et al., 2015; Peng, 2010).”
Background
Design of peer assessment rubrics for ICT topics
Bowen-Mendoza, L. E., Pinargote-Ortega, M., Meza, J., & Ventura, S. (2021) · Journal of Computing in Higher Education · doi:10.1007/s12528-021-09297-9 · Cited by 8
The strategy to sustainable sharing resources repository for Massive Open Online Courses in Thailand
Chunwijitra, S., Tummarattananont, P., Laokok, S., Krairaksa, K., Junlouchai, C., Chai, W. N., & Wutiwiwatchai, C. (2015) · International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) · doi:10.1109/ecticon.2015.7206980 · Cited by 5
“The systematic knowledge of them are explained as following;”
Background
The PASCA: A Mail Based Randomized Blinded Peer Assessment System for Complex Artifacts
Neznanov, A. A., & Maksimenkova, O. (2016) · Procedia Computer Science · doi:10.1016/j.procs.2016.08.261 · Cited by 5
Stagewise learning for noisy k-ary preferences
Pan, Y., Han, B., & Tsang, I. W.-H. (2018) · Machine Learning · doi:10.1007/s10994-018-5716-2 · Cited by 5
The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study
Babik, D., Stevens, S. P., Waters, A. E., & Tinapple, D. (2020) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2020.2971495 · Cited by 5
“Such development is necessitated by the move toward large-scale online teaching and learning and the declining feasibility of single-expert assessment [44], [45].”
Background
The Impact of the Coronavirus (COVID-19) Pandemic on Education
Dang, M. Y., & Zhang, Y. G. (2021) · International Journal of Information and Communication Technology Education · doi:10.4018/ijicte.292481 · Cited by 5
“Without the time and location constrains, e-learning has opened the door of higher education to those who could not take classes on site (Kulkarni et al., 2013; Teo et al., 2018).”
Background
Enhancing active learning in remote collaboration: An experience in teaching functional materials
Marinelli, A., Papile, F., Sossini, L., & Curto, B. D. (2022) · International Journal of Mechanical Engineering Education · doi:10.1177/03064190221143312 · Cited by 7
“given specific and unambiguous terms are used.(22,23) Regarding the obtained results in Figure 6, future work may focus on the evaluation parameters that achieved higher standard deviation, trying to figure out if due to ambiguous terms.”
Result
On the Supervision of Peer Assessment Tasks: An Efficient Instructor Guidance Technique
Hernández-González, J., & Herrera, P. J. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3319733 · Cited by 5
“Peer assessment has been applied in all levels of formal education [23], from primary education to university [20], [24], [25], [26], [27], [28], and in informal education, such as massive online open courses (MOOCs) [5], [22], [29].”
BackgroundMethodInfluential
Adaptive Clustering of Codes for Assessment in Introductory Programming Courses
Barbosa, A. D. A., Costa, E. D. B., & Brito, P. H. S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-91464-0_2 · Cited by 5
Mooqita
Krause, M., Schiöberg, D., & Smeddinck, J. D. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3174351 · Cited by 5
Calibrated Self-Assessment
Labutov, I., & Studer, C. (2016) · ETH Zurich · doi:10.3929/ethz-b-000460330
Mechanical TA 2: Peer Grading with TA and Algorithmic Support
Zarkoob, H., & Leyton-Brown, K. (2024) · Technical Symposium on Computer Science Education · doi:10.1145/3626252.3630891 · Cited by 4
“Unless peer grading systems teach students to grade consistently and accurately, different students are likely to assign different grades to the same work [2, 12, 21, 23].”
Background
A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss System
Zhang, L., Chen, T., Zong, Y., et al. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630767 · Cited by 4
“Researchers have shown that peer review offers many benefits for students and instructors [1, 5, 15, 19].”
Background
Improving Grading Fairness and Transparency with Decentralized Collaborative Peer Assessment
Alipour, S., Elahimanesh, S., Jahanzad, S., Mohammadi, I., Morassafar, P., Neshaei, S. P., & Tefagh, M. (2024) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3637350 · Cited by 4
“…the computer-supported peer assessment systems currently in use suffer from some drawbacks, such as low reliability of the scores given by the peers [15, 52], lack of transparency [98], the effect of personal bias [9–11, 39, 65], high grading burden [40], and lack of intense motivation [49, 86].”
Background
Organic Peer Assessment
Komarov, S., Seas, H., & Gajos, K. Z. (2014) · Cited by 4
Including Content-Based Methods in Peer-Assessment of Open-Response Questions
Luaces, O., Díez, J., Alonso, A., Troncoso, A., & Bahamonde, A. (2015) · 2015 IEEE International Conference on Data Mining Workshop (ICDMW) · doi:10.1109/icdmw.2015.256 · Cited by 8
“We adopt the approach called peer-assessment [1], [2], [3], [4], [5], [6], [7], [8] as the basic strategy.”
BackgroundMethod
Collaborative Calibrated Peer Assessment in Massive Open Online Courses
Boudria, A., Lafifi, Y., & Bordjiba, Y. (2017) · International Journal of Distance Education Technologies · doi:10.4018/ijdet.2018010105 · Cited by 4
Exploring the Effectiveness of a SPOC Learning Analytics System Based on Attribution Theory: Evaluation Framework and Educational Experiment
Lin, X., Zhan, Z., Zhang, X., & Xiong, J. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3268276 · Cited by 5
“Kulkarni et al. [31] focused on optimizing SPOC teaching by strengthening learners’ self-evaluation and peer evaluation, improving online course performance, and enhancing self-efﬁcacy.”
Background
Lessons learned from the first worldwide accessible e-learning in Landscape Ecology
Kienast, F., Gosteli, S., Edwards, T. C., et al. (2020) · Landscape Online · doi:10.3097/lo.202083 · Cited by 4
“Kulkarni et al. (2013) also identifies some difficulties with peer assessment, i.e. “peers’ working from their own country were rated 3.6% higher than those from elsewhere”.”
BackgroundInfluential
Current Trends in Online Programming Languages Learning Tools: A Systematic Literature Review
Alaqsam, A., Ghabban, F. M., Ameerbakhsh, O. Z., Alfadli, I., & Fayez, A. (2021) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2021.147017 · Cited by 4
Technology enhanced peer learning and peer assessment
Henriksen, C. B., Bregnhøj, H., Rosthøj, S., Ceballos, A., Kaas, H., Harker-Schuch, I., Andersen, I., Larsen, S., & May, M. (2016) · Tidsskriftet Læring og Medier (LOM) · doi:10.7146/lom.v9i16.24415 · Cited by 4
“In a MOOC course on human centered interaction design Kulkarni et al (2013) compared student assessments with teacher assessments and found that that even though the majority of students assessed their peers within 10% of the teacher assessments, students generally assessed their peers higher than…”
Result
An analysis of quiz in MOOC
Chauhan, J., & Goel, A. (2016) · International Conference on Contemporary Computing · doi:10.1109/ic3.2016.7880245 · Cited by 4
“We have studied University of Auckland, New Zealand5 that is running on CourseBuilder platform, using guest account.”
Method
A MOBILE APPLICATION FOR USER REGULATED SELF-ASSESSMENTS
Lazarinis, F., Verykios, V. S., & Panagiotakopoulos, C. T. (2017) · International Association for Development of the Information Society · Cited by 4
Los nano-MOOC como herramienta de formación en competencia digitales de los docentes de la Universidad Técnica del Norte
Basantes-Andrade, A. (2020) · Gredos (University of Salamanca) · doi:10.14201/gredos.144006 · Cited by 7
Studying how social relationships affect peer assessment in an E-learning environment
Sousa, F., Alves, T., Gama, S. P., Jorge, J. A., & Gonçalves, D. J. V. (2024) · Learning Environments Research · doi:10.1007/s10984-024-09505-0 · Cited by 3
“The most common method to determine the quality of the peer assessment is by measuring the differences between grades provided by students and faculty members (AlFallay, 2004; Kulkarni et al., 2013; Yan et al., 2022).”
Helping Students to Provide Effective Peer Feedback
Gehringer, E. F. (2018) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--28434 · Cited by 3
“[10] found that students in several Coursera MOOCs said they learned more by assessing their peers (4.”
Background
Uncovering the practices, challenges, and incentives for engineering design faculty
Mandala, M., Schunn, C. D., Dow, S. P., Goldberg, M. R., & Pearlman, J. L. (2018) · International journal of engineering education · Cited by 3
Animal sciences in the secondary classroom: Considering didactic strategies available through distance learning
Gomez, N. A. (2020) · Natural sciences education · doi:10.1002/nse2.20037 · Cited by 3
Probabilistic Models for Competence Assessment in Education
Gómez, A. L. D. A., Sabater-Mir, J., & Sierra, C. (2022) · Applied Sciences · doi:10.3390/app12052368 · Cited by 3
“Moreover, according to these voices, this methodology may provide other potential benefits such as helping students see the task from an assessor’s perspective and boosting self-reflection, as well as providing valuable feedback [3,12].”
Background
Undergraduate dental students’ perspectives on teaching and learning during the COVID-19 pandemic: Results from an online survey conducted at a South African university using a mixed-methods approach
Moodley, R., Singh, S., & Moodley, I. (2022) · African Journal of Health Professions Education · doi:10.7196/ajhpe.2022.v14i1.1482 · Cited by 3
Exploring Potential Contributions of Social Learning to Adaptive Learning Systems
Gautam, S., Rosson, M. B., & Akgun, M. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585758 · Cited by 4
“A re-lated application is the research on peer learning, wherein students interact with peers to attain educational goals [19] [24].”
Background
Offline and online peer assessment in computer engineering: Insights from a 5-year experience
Santander‐Jiménez, S., Vega-Rodríguez, M. A., Granado-Criado, J. M., Rubio‐Largo, Á., Gomez-Pulido, J. A., Gómez‐Martín, C., & Durán-Domínguez, A. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11989-x · Cited by 3
“For example, peer assessment was used in (Kulkarni et al., 2013) to support human-centered interaction design MOOCs, achieving high correlation between peer grades and staff-assigned grades by implementing a calibration-assessment system.”
Um Estudo Exploratório do uso de Rubricas Pedagógicas para Avaliar a Aprendizagem Corregulada e a Regulação Compartilhada de Estudantes de Disciplinas Introdutórias de Programação
de Campos, D. S., & Ferreira, D. J. (2023) · Congresso sobre Tecnologias na Educação (Ctrl+e) · doi:10.5753/ctrle.2023.232953 · Cited by 3
Research to Practice of Blended Learning in Computer Programming Course
Zhang, L., Chen, T., Zong, Y., & Zhang, L. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342940 · Cited by 4
“Adopting peer review can motivate students to complete exercises with higher quality and independence, build their confidence and provide them with a good sense of achievement while also solving the problem of limited teaching team energy [11], [12].”
Background
Teaching UI Design at Global Scales
Cheng, H.-F., Yu, B., Fu, S., et al. (2019) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3330430.3333635 · Cited by 3
“among students in large MOOC classes, such as designing peer assessment tools [19] and group discussion environments [18].”
Background
Peer-review in architecture education: Application in Geographic Information Systems learning
Garcia-Almirall, M. P., Domínguez, E. R., Dalmau, F. V., & Fonseca, D. (2015) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.1109/cisti.2015.7170497 · Cited by 3
“En la literatura encontramos referencias que demuestran que en estos cursos la calificación otorgada por el propio participante a sí mismo es muy parecida a la otorgada por los otros participantes [2], especialmente cuando se tiene el soporte de una rúbrica.”
Background
The conditions of peak empiricism in big data and interaction design
Marcinkowski, M., & Fonseca, F. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23497 · Cited by 4
What Peer-Review Systems Can Learn from Online Rating Sites
Gehringer, E. F., Ma, K., & Duong, V. T. (2015) · Lecture notes in educational technology · doi:10.1007/978-981-287-868-7_42 · Cited by 3
“Kulkarni [14] reports that 20 % of the students in Scott Klemmer’s HCI MOOC did more peer reviews than required.”
Background
Optimising Peer Marking with Explicit Training: from Superficial to Deep Learning
Caldwell, S., & Gedeon, T. (2015) · doi:10.4995/head15.2015.441 · Cited by 3
“Kulkarni et al. (2015) used a “fortune cookie” approach to provide qualitative and personalised feedback. This was found to have no effect on the amount of feedback returned, however, they noted that multiple assignments assessed via peer assessments provided incremental improvements on the quality of assessments prod…”
Method
Peer Reviewing in Participatory Guarantee Systems: Modelisation and Algorithmic Aspects
Barrot, N., Lemeilleur, S., Paget, N., et al. (2020) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/bfzg4089 · Cited by 3
Improving Student Engagement in Online Learning: How the Online Learning Readiness Scale and Self-Assessment Matter?
Cholifah, P. S., Rini, T. A., Nuraini, N. L. S., Satriyani, F. Y., & Saidah, K. (2020) · doi:10.1109/icet51153.2020.9276570 · Cited by 4
“It is related to the case different from the discussion process faceto-face, where sometimes psychological factors, courage in expressing opinions [28].”
BackgroundResult
An Approach to Establishing Simultaneous Server-Side Connections for NFC/Bluetooth enabled Quiz Management Systems
Aqui, J., & Hosein, M. (2021) · Global Journal of Computer Science and Technology · doi:10.34257/gjcstavol20is1pg1 · Cited by 3
“Numerous papers and contributions have been made with respect to online classes/sessions (Kulkarni et al., 2013 and Frydenberg, 2007) and (Grandon, Alshare, and Kwun,2005).”
Background
A Conceptual Framework for Identifying Risks of AI Integration in Higher Education
Al-araibi, A. A. M., & Abdalla, R. (2025) · International Journal of Computer Applications · doi:10.5120/ijca2025925258 · Cited by 2
Non-calibrated Peer Assessment: An Effective Assessment Method for Student Creative Works
Li, J., Zhang, Y., & Gao, K. (2015) · Communications in computer and information science · doi:10.1007/978-3-319-21383-5_47 · Cited by 2
“Peer assessment and self assessment, is an calibrated assessment,which offer an opportunity to scale both assessment and learning to global classrooms, has been adopted in many classes, including 79 MOOCs on the Coursera platform alone [1].”
Background
Data Science Course Projects with Peer Challenges
Wu, W. (2022) · doi:10.1145/3502718.3524743 · Cited by 2
“[23] used self and peer assessment to scale the assessment to large classes and reported that peer grades correlated highly with staff-assigned grades.”
BackgroundMethod
E-Portfolio for Online Assessment: Padlet Implementation
Puspita, N., & Hasyim, U. A. A. (2022) · Jurnal Al-Qiyam · doi:10.33648/alqiyam.v3i2.242 · Cited by 2
“In addition, there are two kinds of online assessment that can be completely online (such as online exams) or just require online submission (such as essays) (Kulkarni et al, 2013).”
Background
SmartGroup: A Tool for Small-Group Learning Activities
Zhu, H., Li, N., Rai, N. K., & Carroll, J. M. (2022) · Future Internet · doi:10.3390/fi15010007 · Cited by 5
A Peer Review Approach to Grading Projects in Computer Courses
Zhang, L., Zong, Y., Chen, T., et al. (2023) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343176 · Cited by 2
“Researchers have provided evidence that peer review offers many benefits for students and instructors [1], [2], [6], [7].”
Background
TeachNow: Enabling Teachers to Provide Spontaneous, Realtime 1:1 Help in Massive Online Courses
Malik, A., Woodrow, J., Wang, C., & Piech, C. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653629 · Cited by 2
Rater behaviors in peer evaluation: Patterns and early detection with learner model
Liang, C., Horikoshi, I., Majumdar, R., & Ogata, H. (2024) · Research and Practice in Technology Enhanced Learning · doi:10.58459/rptel.2025.20012 · Cited by 2
“Validity is assessed by comparing the student evaluation with a standard, such as a teacher’s score (Kulkarni et al., 2013).”
Method
EXPLORING THE INTEGRATION OF DIGITAL COMMUNICATION TOOLS IN LANGUAGE EDUCATION: A COMPREHENSIVE REVIEW
Pongen, L. (2024) · ShodhGyan-NU Journal of Literature and Culture Studies · doi:10.29121/shodhgyan.v2.i2.2024.20 · Cited by 3
“Online Course A study by Kulkarni et al. (2013) explored the use of peer assessment to improve learning in a massive open online course (MOOC). the study, 42, • Peer assessment led to higher levels of learning • Peer assessment also helped students to develop more critical thinking skills.”
Background
The Implementation of Artificial Intelligence in Moroccan Higher Education: Benefits and Challenges From the Perception of Academics
Mjahad, R. A., Boukranaa, A. M., Karfa, A. E., & Sandy, K. (2025) · SAGE Open · doi:10.1177/21582440251381402 · Cited by 3
“As a reaction to this risk, many academics and educators have called for neglecting traditional tasks to assess students and adopting more sophisticated and innovative open-ended tasks that machines cannot solve (Kulkarni et al., 2013).”
Peer Review and Response: Supporting Improved Writing Skills in Environmental Chemistry
Samarasekara, D., Mlsna, T., & Mlsna, D. (2020) · The journal of college science teaching · doi:10.1080/0047231x.2020.12290686 · Cited by 2
The impact of integrated wireless and mobile communication technologies on the corporate world
Durkin, B. J., & Lokshina, I. V. (2015) · Wireless Telecommunications Symposium · doi:10.1109/wts.2015.7117242 · Cited by 2
“This is an environment more open to experimentation and the user population, such as staff and students, are usually early adopters of technology [2, 4, 6, 16].”
Background
Making distance matter
Kulkarni, C. (2014) · doi:10.1145/2658779.2661169 · Cited by 2
“For instance, by aggregating grades from five peers, we estimate 80% of students in the HCI class got grades within 15% of the staff grade [11].”
BackgroundSelf-citation
Source Effects in Online Education
Gupta, N., Neill, J. O., Cross, A., et al. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2728671 · Cited by 2
О бизнес-процессах коллаборативного обучения и его компьютерной поддержки: Разработка системы взаимного оценивания[1]
Kolomiets, A., Maksimenkova, O., & Neznanov, A. A. (2016) · Бизнес-информатика · doi:10.17323/1998-0663.2016.4.35.46 · Cited by 2
New Approaches to the Assessment of Learning: New Possibilities for Business Education
Murgatroyd, S. (2017) · Springer eBooks · doi:10.1007/978-3-319-62776-2_12 · Cited by 2
Increasing Peer Review Quality in Online Learning Systems
Luckner, N., Purgathofer, P., & Fitzpatrick, G. (2018) · International Conference on Higher Education Advances (HEAd) · doi:10.4995/head18.2018.7987 · Cited by 2
“A number of papers discuss different influences of qualitative (e.g. text-based) and quantitative (e.g. numeric or categorized) feedback in reviewing (Hicks et al. 2016), (Kulkarni et al. 2015a), (Kulkarni et al. 2015b).”
BackgroundMethod
Re-imagining Peer Assessment in Self-Paced Online Learning Environments
Corrin, L., & Bakharia, A. (2020) · The enabling power of assessment · doi:10.1007/978-3-030-41956-1_14 · Cited by 1
Quality Control Method for Peer Assessment System Based on Multi-dimensional Information
Li, P., Yin, Z., & Li, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60029-7_17 · Cited by 2
Optimization of movement skills assessment in physical education learning using online self and peer assessment
Fitriady, G., Alfarizi, M. A. M., & Saputra, S. A. (2022) · Journal of Science and Education (JSE) · doi:10.56003/jse.v3i2.168 · Cited by 2
Alternative Assessment for Movement Skills in Physical Education: The Effectiveness of Online Self and Peer Assessment
Fitriady, G., Nurrochmah, S., Yudasmara, D. S., et al. (2022) · doi:10.2991/978-94-6463-072-5_40 · Cited by 2
Investigating Simultaneous Wireless Connections for a Quiz Management System-A Case Study
Aqui, J., & Hosein, M. (2021) · Global Journal of Computer Science and Technology · doi:10.34257/gjcstevol20is4pg1 · Cited by 2
Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms do not Improve over Simple BaselinesPreprint
Sajjadi, M. S. M., Alamgir, M., & von Luxburg, U. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00852 · Cited by 2
“This problem is bound to come up when using peer grading in a course such as algorithms and data structures and might be much less of an issue when grading is used to evaluate project reports (Raman & Joachims, 2014) or design questions (Kulkarni et al., 2015).”
BackgroundMethod
Inventive Scaffolds Catalyze Creative Learning
Ngoon, T. J. (2019) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299073 · Cited by 1
“However, people are not consistently skilled at providing good structural feedback, instead focusing on surface features because they require less analysis to see and critique [6].”
Background
Interactions Beyond the Pandemic: Lessons Learned from Large-scale Emergency Remote Teaching in Higher Education
Yarmand, M., Li, H., & Weibel, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713995 · Cited by 1
“Peer learning (e.g., peer feedback [75] and peer assessment [74]) and commentary discussions among peers about online courses content [145] provide valuable information that can inform scaffolding learning, an instructional practice in which guidance is given to learners in the beginning but is…”
Toward Scalable, Ethical, and AI-Supported Peer Assessment in Higher Education
Bouhdid, B., & Douss, R. (2025) · International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346723
“In the flagship “Circuits and Electronics” MOOC, studies reported reliable peer grading correlations with instructor marks and a positive impact on metacognitive engagement and motivation [22], [23].”
Understanding concepts and values for the future of online education through the practice of design
Marcinkowski, M., & Fonseca, F. (2015) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 1
Supporting Face-to-Face Like Communication Modalities for Asynchronous Assignment Feedback in Math Education
Randles, B., Yoon, D., Cheatle, A., Jung, M. F., & Guimbretière, F. (2015) · ACM Conference on Learning @ Scale · doi:10.1145/2724660.2728684 · Cited by 1
Standing on the Shoulders of Peers
Cambre, J., & Klemmer, S. R. (2017) · doi:10.1145/3022198.3026353 · Cited by 1
“Prior studies in massive online courses found that learners cited seeing others’ work as the most valuable aspect of the peer evaluation process [3].”
BackgroundSelf-citation
Score-Group Framing Negatively Impacts Peer Evaluations
Durkin, C., Rossano, F., & Klemmer, S. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274314 · Cited by 1
Self-citation

Authors: Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
Affiliations: Stanford University (US)
Stanford University, and Coursera, Inc., Stanford, CA
Coursera, Inc., Mountain View, CA
ORCID: Kulkarni, Chinmay
Cheng, Justin
Koller, Daphne
Klemmer, Scott R.
Venue: ACM Transactions on Computer-Human Interaction
Publisher: Association for Computing Machinery (ACM)
Published: 2013-12
Volume: 20
Issue: 6
Pages: 1-31
ISSN: 1073-0516 (ISSN-L)
1557-7325
DOI: 10.1145/2505057
OpenAlex: W2088700588
Cited by: 390
References: 97
Citations by year: 2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
FWCI: 199.15
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computer-Human Interaction (publishedVersion)
Funders: Division of Information and Intelligent Systems
Hasso Plattner Design Thinking Program
Topics: Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Social Sciences › Education › Online and Blended Learning
Keywords: peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Peer and self assessment in massive online classes

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Statistics that don’t add up

Statistics recomputed

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks