Cited by 390Student Assessment and FeedbackOnline Learning and AnalyticsOnline and Blended Learning

The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 52 sources its claims cite were checked for support.

Peer and self assessment in massive online classes

Open access

Kulkarni, Chinmay, Wei, Koh Pang, Le, Huy, Chia, Daniel, Papadopoulos, Kathryn, Cheng, Justin, Koller, Daphne, Klemmer, Scott R.·2013·ACM Transactions on Computer-Human Interaction

DOI 10.1145/2505057

Open this paper in CoChat — free account Open PDF

Abstract

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Integrity signals

Registry

Crossref · 10.1145/2505057

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

2 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

2 of 52 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5. PROVIDING PERSONALIZED, QUALITATIVE FEEDBACK ON ASSIGNMENTS
What the sources say
- Supported
  In “Delivers high quality information to students about their learning” (p. 9), the paper defines good-quality feedback as information that helps students identify problems and take action to improve their performance. It also recommends timely feedback and corrective advice (p. 10), supporting the claim’s emphasis on accuracy and actionability. (LLM, conf=0.93)
  cites Nicol and Macfarlane-Dick 2006 source
- Not checked — no full text of the sourcecites Boud 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Partly supported
  The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)
  cites Cheshire and Antin 2008 sourceExperimental — accuracy not yet measured
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, students also reported that they felt their peers put less effort into peer assessment than they did (Table III).On a Mann-Whitney U-test, mean ratings were 4.57 for peer effort and 5.46 for their own effort (6-point Likert scale, 6:”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that giving students feedback about their grading bias increased subsequent accuracy.”
- Also stated in: § 1.4. Contributions · plain
  “First, we found that giving students feedback about whether they scored peers high or low increased their subsequent accuracy.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, students assessed their work 7% higher than staff did.”
§ Peer and Self Assessment in Massive Online Classes > Abstract
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.

Show all 102 claims

It may state two separate claims as one. 56
Probably not a substantial claim. 56
The quote only partly states this claim. 1

A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Load-bearingStrength:plain· not yet calibratedn = 3,550
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was also useful for identifying mistakes and reflection (Figure 8).”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
§ Peer and Self Assessment in Massive Online Classes > 1.4. Contributions
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluating peers was useful for inspiration and to see other perspectives.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
§ Peer and Self Assessment in Massive Online Classes > 4.2. Results: Feedback Reduces Grading Errors
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
§ Peer and Self Assessment in Massive Online Classes > 3.7. Would More Raters Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
§ Peer and Self Assessment in Massive Online Classes > 3.6. Does a Different Weighting of Peer Grades Help?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
§ Peer and Self Assessment in Massive Online Classes > 3.8. Do Students Become Better Graders Over Time?
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
- Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.
In the paper
“Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Lewin 2013c
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Zimmerman and Schunk 2001 source
- Not checked — no full text of the sourcecites Pintrich 1995 source
- Not checked — no full text of the sourcecites Pintrich and Zusho 2007 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
§ Peer and Self Assessment in Massive Online Classes > 1. INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Hearst 2000 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
§ Peer and Self Assessment in Massive Online Classes > 1.2. The Promise of Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
§ Peer and Self Assessment in Massive Online Classes > 1.3. Scaling Peer Assessment
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Chi 2009
- Not checked — no full text of the sourcecites Carter et al. 2008 source
Load-bearingStrength:plain· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Galinsky and Moskowitz 2000 source
Load-bearingStrength:hedged· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not checked — no full text of the sourcecites Anderson and Bower 1972 source
- Not checked — no full text of the sourcecites Nielsen 1994 source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.
In the paper
“Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
§ Peer and Self Assessment in Massive Online Classes > 3.3. Results: Grading Agreement Between Staff
What the sources say
- Not checked — no full text of the sourcecites Amabile 1982 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
§ Peer and Self Assessment in Massive Online Classes > 3.5. Results: Student Reactions
Strength:hedged· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Survey respondents are representative of the class.
In the paper
“Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
§ Peer and Self Assessment in Massive Online Classes > 4. PROVIDING STUDENTS FEEDBACK ON GRADING ACCURACY IMPROVES SUBSEQUENT PERFORMANCE
What the sources say
- Not checked — no full text of the sourcecites Dow et al. 2012 source
Load-bearingStrength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
In the paper
“students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Nielsen 1993 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
§ Peer and Self Assessment in Massive Online Classes > 6.3. Inflating Self-Grades and Other Gaming
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Boud 1995
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
§ Peer and Self Assessment in Massive Online Classes > 6.5. The Changing Role of Teachers
What the sources say
- Not checked — no full text of the sourcecites Palen 1999 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Breslow et al. 2013
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
§ Peer and Self Assessment in Massive Online Classes > 5.3. Discussion
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Little and Bjork 2012
Load-bearingStrength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Pendleton-Jullian 2010
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not matched
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
  cites Schön 1985
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Kaufman et al. 2008 source
Strength:hedged· not yet calibratedCausal:conditional· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Klemmer et al. 2006 source
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
§ Peer and Self Assessment in Massive Online Classes > 6.1. Using Data to Improve Assessment Materials
What the sources say
- Not checked — no full text of the sourcecites Andrade [2005] source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
§ Peer and Self Assessment in Massive Online Classes > 6.4. Limitations of Peer Assessment
What the sources say
- Not checked — no full text of the sourcecites Conti et al. 1996 source
Strength:boosted· not yet calibratedCausal:correlational· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
It may state two separate claims as one.
In the paper
“Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
§ Peer and Self Assessment in Massive Online Classes > 7. CONCLUSIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites Martin 2012 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“One has to do with enrollment and retention [Kizilcec et al. 2013].”
§ Peer and Self Assessment in Massive Online Classes > 6.7. The Changing Classroom
What the sources say
- Not checked — no full text of the sourcecites Kizilcec et al. 2013 source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.

93 of 93

The paper relies on this without support.
Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
“To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
“We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”
The paper relies on this without support.
Survey respondents are representative of the class.
“At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”
The authors state this as an assumption.
Grades in different assignments are uncorrelated.
“(assuming grades in different assignments are uncorrelated)”
The paper relies on this without support.
Staff grades are a valid ground truth for assessing peer grading accuracy.
“Holding out 10% of ground truth grades”
“compared to 7.74% in the no-feedback condition”
The paper relies on this without support.
At a physical university, the completion rate for an equivalent class is much higher than in the online class.
“Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”
The paper relies on this without support.
Peers can help instruction itself.
“Fourth, peers can help instruction itself.”

6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

Inconsistent p-value across alpha = .05: F(1, 39) = 2.07, p < .05
Major
The paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .158. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.
Reported p < .05 · recomputed p = .158

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05
“F(1, 39) = 2.07, p < .05”
Consistentt(24) = 19.9reported p < .05 · recomputed p < .001
“t(24) = 19.9,p < .05”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Figure 5 caption names the wrong comparison grade
Minor
The results describe Figure 5 as comparing median peer grades with self-assessed grades, but its caption says the percentages measure agreement with staff grades.

1 of 1

An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Formative assessment and self-regulated learning: A model and seven principles of good feedback practice
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08
(2008) · doi:10.3115/1613715 · Cited by 2,216
Peer Assessment between Students in Colleges and Universities
Topping, K. (1998) · Review of Educational Research · doi:10.2307/1170598 · Cited by 2,173
Self-Regulated Learning and Academic Achievement
Zimmerman, B. J., & Schunk, D. H. (1989) · Springer series in cognitive development · doi:10.1007/978-1-4612-3618-4 · Cited by 1,926
How Designers Think: The Design Process Demystified
Lawson, B. (1990) · Cited by 1,906

Flagged first, then by citations

Show all 97 references

Research works printed but not linked: 11

H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/?p=165.
O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.

Sorted by citations

How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
“Peer assessment is a proven method for scaling the grading of a large number of assessments, such as is required in massive open online courses (MOOCs) [14, 19].”
BackgroundMethod
Massive open online course completion rates revisited: Assessment, length and attrition
Jordan, K. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i3.2112 · Cited by 510
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
“Datasets from massive open online courses are a potential source of large-scale data (Kulkarni et al., 2015).”
MOOCs: Expectations and Reality
Tirthali, F. M. H. A. D. (2014) · Scholarly Commons (University of Pennsylvania) · Cited by 303
Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation
Blau, I., & Shamir‐Inbal, T. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.014 · Cited by 227
“…self-assessment that offer new opportunities for ﬂ ipped learning in academia, has shown inaccurate grading e on average 7% higher than staff assessment and prone to be biased e students from the same country received higher grades compared to peers from different countries (Kulkarni et al., 2015).”
Result
Students’ patterns of engagement and course performance in a Massive Open Online Course
Phan, T., McNeil, S., & Robin, B. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.11.015 · Cited by 199
PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724670 · Cited by 192
“Prior work shows peer-based critique is effective both for in-person [8,29] and online classes [25], and can provide students accurate numeric grades and comments [13,25].”
BackgroundSelf-citation
Autonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. J. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724668 · Cited by 139
“At what point is a student ready to consciously reflect on the challenges that they faced? Peer grading is one of the most common solutions for giving feedback to complex assignments at scale in massive open online courses [8]; however, it can only be used to give feedback on a final submissions, not hints to a stuck…”
Background
How technology shapes assessment design: Findings from a study of university teachers
Bennett, S., Dawson, P., Bearman, M., Molloy, E., & Boud, D. (2017) · British Journal of Educational Technology · doi:10.1111/bjet.12439 · Cited by 130
“Technologies to support assessment have a long history in higher education – from the early days of programmed instruction and computerbased quizzes, to richer forms of interaction and content creation underpinned by constructivist approaches, and more recent tools that support online assignment submission, peer- and…”
BackgroundMethod
Assessment approaches in massive open online courses: Possibilities, challenges and future directions
Xiong, Y., & Suen, H. K. (2018) · International Review of Education · doi:10.1007/s11159-018-9710-5 · Cited by 101
“Among all assessment methods within MOOCs, peer assessment has received considerable attention (e.g. Admiraal et al. 2014; Goldin 2012; Kulkarni et al. 2015; Luo et al. 2014; Piech et al. 2013; Raman and Joachims 2014; Suen 2014; Xiong et al. 2014).”
BackgroundMethodInfluential
Towards AI-Powered Personalization in MOOC Learning
Yu, H., Miao, C., Leung, C., & White, T. J. (2017) · npj Science of Learning · doi:10.1038/s41539-017-0016-3 · Cited by 131
“Techniques such as involvement of community teaching assistants (TAs), 21 peer assessment tools, 22 well-designed elaborated rubrics with speci ﬁ c content feedback 23 and video feedbacks 24 are available to help MOOC instructors scale up the provision of perceived personal attention to students.”
Method
Building engagement for MOOC students
Nawrot, I., & Doucet, A. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2567948.2580054 · Cited by 137
“Furthermore, MOOC platforms take advantage of the newest research ﬁndings to provide their students with as engaging an experience as possible (eg. retrieval questions in videos [19, 18] or peer assessment [25, 36, 34]).”
Background
Informed Truthfulness in Multi-Task Peer Prediction
Shnayder, V., Agarwal, A., Frongillo, R., et al. (2016) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2940716.2940790 · Cited by 114
“2013], as well as helping students self-adjust for bias and provide better feedback [Kulkarni et al. 2013].”
BackgroundMethod
Building Engagement for MOOC Students - Introducing Support for Time Management on Online Learning Platforms
Nawrot, I., & Doucet, A. (2014) · HAL · Cited by 116
Methods for ordinal peer grading
Raman, K., & Joachims, T. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623654 · Cited by 111
“In peer grading, students — not instructors or TAs — provide feedback on the work of other students in their class [14, 22], meaning that the number of “graders” naturally grows with the number of students.”
Background
Learner Behaviour in a MOOC Practice-oriented Course: In Empirical Study Integrating TAM and TPB
Yang, H.-H., & Su, C.-H. (2017) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v18i5.2991 · Cited by 111
“One common approach to overcome these issues is the mechanism of peer assessment (Kulkarni et al., 2013; Suen, 2014).”
Background
Understanding online interaction in language MOOCs through learning analytics
Martín-Monje, E., Castrillo, M. D., & Mañana-Rodríguez, J. (2018) · Computer Assisted Language Learning · doi:10.1080/09588221.2017.1378237 · Cited by 98
“According to several researchers, structured self-assessment provides a suitable path in an LMOOC (Sokolik, 2014; Kulkarni et al., 2013).”
Background
Talkabout: Making Distance Matter with Small Groups in Massive Classes
Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., & Klemmer, S. R. (2015) · Conference on Computer Supported Cooperative Work · doi:10.1145/2675133.2675166 · Cited by 120
“One example is the use of rater redundancy and short exercises that create microexpertise in peer review: with this structure, peers can provide expert-quality assessment and feedback [32], and act as mentors [45].”
BackgroundSelf-citation
Why do Institutions Offer MOOCs?
Hollands, F. M., & Tirthali, D. (2014) · Online Learning · doi:10.24059/olj.v18i3.464 · Cited by 101
“…al. (2013); Halawa, Greene, and Mitchell (2014); and Nguyen, Piech, Huang, and Guibas (2014) • natural language processing • human–computer interaction—for example, Kulkarni et al. (2013) and Cambre, Kulkarni, Bernstein, and Klemmer (2014) • personalized and adaptive learning—for example,…”
Background
Almost an Expert
Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819953 · Cited by 98
Scaling short-answer grading by combining peer assessment with algorithmic scoring
Kulkarni, C. E., Socher, R., Bernstein, M. S., & Klemmer, S. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2566238 · Cited by 95
“Repeating this process multiple times estimates peer agreement with staff [17].”
BackgroundResultInfluentialSelf-citation
Does peer assessment in on-line learning environments work? A systematic review of the literature
Tenório, T., Bittencourt, I., Isotani, S., & Silva, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.06.020 · Cited by 82
Multimodal Technologies in Precision Education: Providing New Opportunities or Adding More Challenges?
Qushem, U. B., Christopoulos, A., Oyelere, S. S., Ogata, H., & Laakso, M. (2021) · Education Sciences · doi:10.3390/educsci11070338 · Cited by 114
“In addition to these functionalities, recent implementations discuss the integration of features that involve the engagement of educators or community assistants [62] who aim at providing participants with either video feedback [63] or peer assessment [64].”
Background
Structuring Interactions for Large-Scale Synchronous Peer Learning
Coetzee, D., Lim, S., Fox, A., Hartmann, B., & Hearst, M. A. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675251 · Cited by 95
“Great strides have been made in peer grading in MOOCs, where students improve their own understanding by assessing the work of others [29].”
Background
Methodological approaches in MOOC research: Retracing the myth of P roteus
Raffaghelli, J. E., Cucchiara, S., & Persico, D. (2015) · British Journal of Educational Technology · doi:10.1111/bjet.12279 · Cited by 89
Improving Crowd Innovation with Expert Facilitation
Chan, J., Dang, S., & Dow, S. P. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2820023 · Cited by 91
“For example, peer-review systems have successfully employed scaffolding techniques, such as preauthored templates to improve feedback at scale [27,34].”
Method
Neuro-symbolic program corrector for introductory programming assignments
Bhatia, S., Kohli, P., & Singh, R. K. (2018) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/3180155.3180219 · Cited by 84
“These approaches can be categorized into two broad categories – peer-grading [14, 15] and automated grading techniques [8, 17, 26, 27, 29].”
Background
Massive open online courses (MOOCs) as an enabler for competent employees and innovation in industry
Karnouskos, S. (2017) · Computers in Industry · doi:10.1016/j.compind.2017.05.001 · Cited by 83
“mediately followed by short quizzes and potentially larger assignments/homework, all of which are graded either by increasingly sophisticated software [18], engaged alumni, tutors or even peer-students [19].”
Background
Artificial Intelligence in Education: Exploring the Potential Benefits and Risks
Ayala-Pazmiño, M. (2023) · 593 Digital Publisher CEIT · doi:10.33386/593dp.2023.3.1827 · Cited by 122
2 What Constitutes an Effective Language MOOC?
Sokolik, M. (2014) · doi:10.2478/9783110420067.2 · Cited by 74
Reinforcement Learning for the Adaptive Scheduling of Educational Activities
Bassen, J., Balaji, B., Schaarschmidt, M., Thille, C., Painter, J., Zimmaro, D. M., Games, A., Fast, E., & Mitchell, J. C. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376518 · Cited by 85
“As a very different form of automation, learnersourcing has been used to manage hard-to-scale instructor tasks such as grading [19], providing feedback [20], giving explanations [40] and hints [10], and even creating new educational materials [25].”
Background
Enhancing peer assessment with artificial intelligence
Topping, K. J., Gehringer, E. F., Khosravi, H., Gudipati, S., Jadhav, K., & Susarla, S. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00501-1 · Cited by 68
“Kulkarni et al. (2013) reported on large online classes where peer assessment showed high correspondence with staff assessments.”
Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural NetworksPreprint
Bhatia, S., & Singh, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06129 · Cited by 103
“These approaches can be categorized into two broad categories – peer-grading [11] and automated grading techniques [18, 13].”
Background
Insights about large-scale online peer assessment from an analysis of an astronomy MOOC
Formánek, M., Wenger, M., Buxner, S., Impey, C., & Sonam, T. (2017) · Comput. Educ. · doi:10.1016/j.compedu.2017.05.019 · Cited by 71
“Unlike in the study by Kulkarni et al. (2015) our peer graders turned out to be more harsh graders than instructors.”
Result
Crowd Guilds
Whiting, M. E., Gamage, D., Gaikwad, S. S., Gilbee, A., Goyal, S., Ballav, A., Majeti, D., Chhibber, N., Richmond-Fuller, A., Vargus, F., et al. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998234 · Cited by 66
“Our infrastructure for crowd guilds enables workers to engage in continuous doubleblind peer assessment [32] of a random sample of members’ task submissions on the crowdsourcing platform, rating the quality of the submission and providing critiques for further improvement.”
BackgroundInfluential
Improving the Peer Assessment Experience on MOOC Platforms
Staubitz, T., Petrick, D., Bauer, M., Renz, J., & Meinel, C. (2016) · ACM Conference on Learning @ Scale · doi:10.1145/2876034.2876043 · Cited by 66
“Comparing the survey results of OSAP1 and OSAP2 (in terms of perceived learning impact of assessing peers or one’s own) with Kulkarni’s results [16] (see Figure 5 and 6) shows at least very similar tendencies.”
BackgroundMethodResultInfluential
A framework for designing, implementing, communicating and researching peer assessment
Adachi, C., Tai, J., & Dawson, P. (2017) · Higher Education Research & Development · doi:10.1080/07294360.2017.1405913 · Cited by 74
“Cluster VI: contextual elements (17) Technology use (Kulkarni et al., 2015; Li et al., 2015) What technology facilitates it? What support is required to use technology? Nil – face to face Assignment exchange done online through Learning Management System; manual group creation Simple online tool created”
BackgroundMethod
Designing moocs to facilitate participant self-monitoring for self-directed learning
Zhu, M., & Bonk, C. J. (2019) · Online Learning · doi:10.24059/olj.v23i4.2037 · Cited by 64
“Such Online Learning Journal – Volume 23 Issue 4 – December 2019 5 126 results align with the findings of Kulkarni et al. (2013).”
Result
Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features
Zedelius, C. M., Mills, C., & Schooler, J. W. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1137-1 · Cited by 67
Critique Style Guide
Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025883 · Cited by 60
“Luther et al. compares differences between design iterations [33, 53], while others contrast critiques with feedback produced by experts [33, 31].”
BackgroundMethodResult
Design Exposition with Literate Visualization
Wood, J., Kachkaev, A., & Dykes, J. A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864836 · Cited by 61
“Primarily it is a form of scaffolding [39] to direct design exposition, analogous to the use of rubrics and ‘fortune cookies’ [16] in pedagogy.”
Uncovering Trajectories of Informal Learning in Large Online Communities of Creators
Yang, S., Domeniconi, C., Revelle, M., Sweeney, M., Gelman, B. U., Beckley, C., & Johri, A. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724674 · Cited by 60
“investigated ways to scale up peer assessment [12].”
Background
"I was afraid, but now I enjoy being a streamer!"
Chen, X., Chen, S., Wang, X., & Huang, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432936 · Cited by 85
“, MOOCs [49, 105, 106], and online peer learning systems [54, 55].”
A Systematic Review of the Socio-Ethical Aspects of Massive Online Open Courses
Rolfe, V. E. (2015) · European Journal of Open Distance and E-Learning · doi:10.1515/eurodl-2015-0004 · Cited by 59
“Six articles carried out a comparative analysis across two or more groups or interventions (Li et al., 2014; Kop & Carroll, 2012; Stewart, 2010; DeBoer, Stump, Seaton & Breslow, 2013; Kulkarni et al., 2013; Root, 2014).”
Background
University of Toronto instructors’ experiences with developing MOOCs
Najafi, H., Rolheiser, C., Harrison, L., et al. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i3.2073 · Cited by 58
Mapping the Factors Influencing Success of Massive Open Online Courses (MOOC) in Higher Education
Albelbisi, N. A., Yusop, F. D., & Salleh, U. K. M. (2018) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/91486 · Cited by 58
E-assessment process: giving a voice to online learners
Fontanillas, T. R., Carbonell, M. R., & Catasús, M. G. (2016) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-016-0019-9 · Cited by 56
“In Higher Education (HE), the construction of an integrated European space has been an adaptation challenge for universities (González & Wagenaar, 2003).”
Background
Instructors’ Experience of Designing MOOCs in Higher Education: Considerations and Challenges
Zhu, M., Bonk, C., & Sari, A. R. (2018) · Online Learning · doi:10.24059/olj.v22i4.1495 · Cited by 34
“participation (Anders, 2015), peer assessment (Kulkarni et al., 2015; Phan et al., 2016), interaction, and the heavy demands of time and money (Hew & Cheung, 2014; Milligan et al.”
Background
Mudslide
Glassman, E. L., Kim, J., Monroy-Hernández, A., et al. (2015) · doi:10.1145/2702123.2702304 · Cited by 56
“This scenario might be particularly relevant for MOOCs, where peer-based interactions sometimes supplant teacher interactions for reasons of scalability [15].”
Background
Juxtapeer
Cambre, J., Klemmer, S., & Kulkarni, C. E. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173868 · Cited by 63
Self-citation
Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P., & Dietze, S. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3119930 · Cited by 54
“Kulkarni et al. showed that in an online course addressing a large number of students (MOOC), the students graded their work 7% higher than those assigned by the staff on average [27].”
Background
Peer assessment in MOOCs: Systematic literature review
Gamage, D., Staubitz, T., & Whiting, M. E. (2021) · Distance Education · doi:10.1080/01587919.2021.1911626 · Cited by 59
“We highlight the studio model of human-computer interaction education, which provides an open, shared environment in which students can work (Reimer & Douglas, 2003) and an open feedback process that enhances reflective learning (Kulkarni et al., 2013).”
Background
Interactive Guidance Techniques for Improving Creative Feedback
Ngoon, T. J., Fraser, C. A., Weingarten, A. S., Dontcheva, M., & Klemmer, S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173629 · Cited by 55
“Viewing examples of past designs can lead to greater creativity and insights [21,26]; thus, showing examples of good feedback may spark ideas reviewers would not have otherwise considered [12,22,25].”
BackgroundInfluentialSelf-citation
Critiki
Greenberg, M. D., Easterday, M. W., & Gerber, E. M. (2015) · Creativity and Cognition · doi:10.1145/2757226.2757249 · Cited by 51
“In this work they describe how structured peer training and grading mechanisms can perform at the level of expert graders and achieve high levels of agreement with previously established answers [24].”
Background
The Future of Adaptive Learning: Does the Crowd Hold the Key?
Heffernan, N. T., Ostrow, K., Kim, K. N., Selent, D., Inwegen, E. G. V., Xiong, X., & Williams, J. J. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0094-z · Cited by 51
“^ Not only might the process of explaining their actions help to solidify their understanding of the content, but the feedback they provide can in turn be connected to the problem for the benefit of future students (Kulkarni et al. 2015).”
Background
Crowd Coach
Chiang, C., Kasunic, A., & Savage, S. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274306 · Cited by 59
“Yet most related work has studied peer learning within formal educational settings [17, 52, 78].”
Background
Framing Feedback
Hicks, C. M., Pandey, V., Fraser, C. A., & Klemmer, S. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858195 · Cited by 54
“g: learners may tend to inflate assessments for peers from their own country [32].”
BackgroundSelf-citation
The Pandemic Shift to Remote Learning under Resource Constraints
Ravi, P., Ismail, A., & Kumar, N. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476055 · Cited by 48
Crowd Research
Vaish, R., Gaikwad, S. S., Kovacs, G., Veit, A., Krishna, R., Arrieta-Ibarra, I., Simoiu, C., Wilber, M. J., Belongie, S., Goel, S., et al. (2017) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3126594.3126648 · Cited by 48
Automatic detection of inconsistencies between numerical scores and textual feedback in peer-assessment processes with machine learning
Rico-Juan, J. R., Gallego, A., & Calvo-Zaragoza, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.103609 · Cited by 50
An Analysis of the use of Good Feedback Practices in Online Learning Courses
Cavalcanti, A. P., de Mello, R. F. L., Rolim, V., André, M., Freitas, F., & Gašević, D. (2019) · International Conference on Advanced Learning Technologies · doi:10.1109/icalt.2019.00061 · Cited by 32
“of providing sufficiently detailed and personalized feedback [12].”
Background
Designing planning and control interfaces to support user collaboration with flying robots
Szafir, D. J., Mutlu, B., & Fong, T. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916688256 · Cited by 46
“This process was inspired by grading systems for courses (Kulkarni et al., 2015), which may create aggregate scores by assigning weights across a variety of different tasks that may have unknown difficulties.”
Method
Strategyproof peer selection using randomization, partitioning, and apportionment
Aziz, H., Lev, O., Mattei, N., Rosenschein, J. S., & Walsh, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2019.06.004 · Cited by 53
“The peer selection problem is closely related to peer-based grading/marking [1, 28, 32, 47, 51, 58, 60] especially when students are graded based on percentile scores.”
Background
Students’ perceptions of the peer-feedback experience in MOOCs
Kasch, J., van Rosmalen, P., Löhr, A., Klemke, R., Antonaci, A., & Kalz, M. (2021) · Distance Education · doi:10.1080/01587919.2020.1869522 · Cited by 51
Peer Assessment Enhances Student Learning: The Results of a Matched Randomized Crossover Experiment in a College Statistics Class
Sun, D. L., Harris, N., Walther, G., & Baiocchi, M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0143177 · Cited by 60
“It has already made personalized feedback feasible in a number of settings—most notably MOOCs—where it otherwise would be impossible [9].”
Example-based learning in computer-aided STEM education
Gulwani, S. (2014) · Communications of the ACM · doi:10.1145/2634273 · Cited by 42
“We can leverage large student populations to crowdsource tasks that are difficult to automate [31] as is done in peer grading [21].”
Background
Structure and messaging techniques for online peer learning systems that increase stickiness
Kotturi, Y., Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724676 · Cited by 42
“The second platform, PeerStudio (Figure 3), provides fast feedback on in-progress open-ended work, such as essays [23].”
BackgroundSelf-citation
UpGrade
Wang, X., Talluri, S. T., Rosé, C. P., & Koedinger, K. R. (2019) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3330430.3333614 · Cited by 42
Reviewing and analyzing peer review Inter-Rater Reliability in a MOOC platform
García-Loro, F., Martín, S., Ruipérez‐Valiente, J. A., Sancristobal, E., & Castro, M. A. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103894 · Cited by 47
Using Truth Detection to Incentivize Workers in Mobile Crowdsourcing
Huang, C., Yu, H., Berry, R., et al. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2020.3034590 · Cited by 41
Strategyproof Peer Selection: Mechanisms, Analyses, and Experiments
Aziz, H., Lev, O., Mattei, N., Rosenschein, J. S., & Walsh, T. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10038 · Cited by 40
“The peer selection problem is also related to peer-based grading/marking (Alfaro and Shavlovsky 2014; Joachims and Raman 2015; Kulkarni et al. 2013; Piech et al. 2013; Robinson 2001; Walsh 2014; Wright, Thornton, and Leyton-Brown 2015) especially when students are graded based on percentile scores.”
Background
Nonprofessional Peer Support to Improve Mental Health: Randomized Trial of a Scalable Web-Based Peer Counseling Course
Bernecker, S. L., Williams, J. J., Caporale-Berkowitz, N. A., Wasil, A. R., & Constantino, M. J. (2020) · Journal of Medical Internet Research · doi:10.2196/17164 · Cited by 41
“Feedback took the form of self-assessments because of the challenges of providing nuanced human-delivered or machine-coded feedback at scale [79,80] and the risk associated with an untrained peer providing inaccurate or anxiety-provoking feedback [81].”
Non-native text analysis: A survey
Massung, S., & Zhai, C. (2015) · Natural Language Engineering · doi:10.1017/s1351324915000303 · Cited by 36
A peer assessment method to provide feedback, consistent grading and reduce students' burden in massive teaching settings
Luaces, O., Díez, J., & Bahamonde, A. (2018) · Comput. Educ. · doi:10.1016/j.compedu.2018.07.016 · Cited by 35
“Peer assessment has been explored as an eﬃcient procedure to deal with this problem; see for instance (Kulkarni et al., 2015; Piech et al., 2013; Raman and Joachims, 2014, 2015; Sadler and Good, 2006; Shah et al., 2013; Labutov and Studer, 2016; D´ıez et al., 2013; Luaces et al., 2015a,b, 2017;…”
Method
The student self-assessment paradigm in MOOC: An example in Chinese higher education
Duan, T.-T., & Wu, B.-H. (2023) · Comunicar · doi:10.3916/c75-2023-09 · Cited by 3
An Exploration of Automated Grading of Complex Assignments
Geigle, C., Zhai, C., & Ferguson, D. C. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2876049 · Cited by 41
“While there are encouraging ﬁndings about peer assessment and methods proposed to improve it [14, 24], there are still systematic problems with this approach: discrepancy between peer and instructor ratings, variation in ratings over time by the same peer rater, inconsistency across exercises for…”
Background
Critique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community
Cheng, R., Zeng, Z., Liu, M., & Dow, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415232 · Cited by 33
“Many instructors have adopted peer feedback methods [39] but have noted that peers often lack the experience to provide meaningful feedback like experts [23, 31, 56].”
Background
Bayesian Ordinal Peer Grading
Raman, K., & Joachims, T. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724678 · Cited by 33
Analyzing MOOC discussion forum messages to identify cognitive learning information exchanges
Wong, J. A., Pursel, B., Divinsky, A., & Jansen, B. J. J. (2015) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.2015.145052010023 · Cited by 34
“As MOOCs continue to rise in popularity (Russell & Klemmer 2013), researchers are examining specific aspects of MOOCs beyond simply exploring descriptive data and completion rates, including innovative research facets (Kulkarni et al. 2013; Coetzee, Fox, Hearst, & Hartmann, 2014).”
Background
A factorization approach to evaluate open-response assignments in MOOCs using preference learning on peer assessments
Luaces, O., Díez, J., Alonso-Betanzos, A., Troncoso, A., & Bahamonde, A. (2015) · Knowledge-Based Systems · doi:10.1016/j.knosys.2015.05.019 · Cited by 35
“Since it is unpractical that an instructor evaluates all the assignments, or even a set of Teaching Assistants (TA), researchers have been exploring the possibilities of using methods previously employed in Journals or Conferences: peer grading or peer assessment [2, 3, 4, 5, 6, 7].”
BackgroundMethod
Distributed Apprenticeship in Online Communities
Hui, J. S., Easterday, M. W., & Gerber, E. M. (2018) · Human-Computer Interaction · doi:10.1080/07370024.2018.1469409 · Cited by 35
“…& Ackerman, 2008), expert recommender systems (McDonald & Ackerman, 2000), social search tools (Morris & Horvitz, 2007), massive open online courses (MOOCS) (Brooks, Thompson, & Teasley, 2015; Kulkarni et al., 2013), and social media sites, like Facebook (Lampe, Wohn, Vitak, Ellison, & Wash, 2011).”
BackgroundInfluential
Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring
Hearst, M. A., Laskowski, P., & Silva, L. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858280 · Cited by 33
“…novice critiques with that of experts; in particular, the use of rubrics has been shown in massive open online courses to bring student ratings within one grade of instructors [18] and novice guideline-based feedback in aggregate can be as acceptable to the receivers as expert feedback [30, 21].”
Background
Aggregating Partial Rankings with Applications to Peer Grading in Massive Online Open Courses
Caragiannis, I., Krimpas, G. A., & Voudouris, A. A. (2015) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/2772879.2773241 · Cited by 28
Peer Grading in a Course on Algorithms and Data Structures
Sajjadi, M. S. M., Alamgir, M., & von Luxburg, U. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2876036 · Cited by 28
“This problem is bound to come up when using peer grading in a course such as algorithms and data structures and might be much less of an issue when grading is used to evaluate project reports (Raman & Joachims, 2014) or design questions (Kulkarni et al., 2015).”
BackgroundMethod
Moving biochemistry and molecular biology courses online in times of disruption: Recommended practices and resources ‐ a collaboration with the faculty community and ASBMB
Procko, K., Bell, J. K., Benore, M. A., et al. (2020) · Biochemistry and Molecular Biology Education · doi:10.1002/bmb.21354 · Cited by 27
Curiosity Notebook: A Platform for Learning by Teaching Conversational Agents
Law, E., Baghaei Ravari, P., Chhibber, N., Kulic, D., Lin, S., Pantasdo, K. D., Ceha, J., Suh, S., & Dillen, N. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3382783 · Cited by 28
“, explaining [23], questioning [9], assessment and feedback [15]—require reflective knowledge building, where students synthesize, structure and reflect [19].”
Background
Decipher: An Interactive Visualization Tool for Interpreting Unstructured Design Feedback from Multiple Providers
Yen, Y.-C. G., Kim, J. O., & Bailey, B. P. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376380 · Cited by 34
Incorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback Quality
Darvishi, A., Khosravi, H., Abdi, S., Sadiq, S., & Gašević, D. (2022) · Proceedings of the Ninth ACM Conference on Learning @ Scale · doi:10.1145/3491140.3528265 · Cited by 31
“Kulkarni et al. [47] provided training for students during the peer review process by offering them feedback about their bias, standard adaptable feedback texts, and indicating crucial items.”
Background
A comparative analysis of the consistency and difference among online self-, peer-, external- and instructor-assessments: The competitive effect
Domínguez, C., Jaime, A., Sánchez, A., Blanco, J. M., & Heras, J. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.02.061 · Cited by 28
Language acquisition through massive open online courses (MOOCs): opportunities and restrictions in educational university environment
Vorobyeva, A. A. (2018) · XLinguae · doi:10.18355/xl.2018.11.02.11 · Cited by 26
Peer Feedback in Learner‑Learner Interaction Practices. Mixed Methods Study on an xMOOC
Elizondo-García, J., & Córdova, K. E. G. (2020) · The Electronic Journal of e-Learning · doi:10.34190/ejel.20.18.2.002 · Cited by 25
iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience
Rocio, V. J. R., Coelho, J., Caeiro, S., et al. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i6.2160 · Cited by 25
“Also iMOOC participants frequented a five-day “boot camp” module prior to the course for familiarization with the online environment and for initiating social bonding, following the best practices specified on Pereira et al. (2008). Furthermore, on the pilot iMOOC, the use of introductory videos in each topic and a we…”
Crowdsourcing programming assignments with CrowdSorcerer
Pirttinen, N., Kangas, V., Nikkarinen, I., Nygren, H., Leinonen, J., & Hellas, A. (2018) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3197091.3197117 · Cited by 31
“Course participants in various institutions and MOOCs evaluate assignment responses that the participants themselves have submitted as a part of their coursework – research suggests that peers can give actionable and accurate feedback [13].”
Background
From in the Class or in the Wild?
Wauck, H. C., Yen, Y.-C. G., Fu, W., Gerber, E. M., Dow, S. P., & Bailey, B. P. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025477 · Cited by 24
“Researchers continue to improve the quality of peer review by studying interface features of the review platforms [15], rubrics [23], and characteristics of the review content [31].”
BackgroundInfluential
Content-based methods in peer assessment of open-response questions to grade students as authors and as graders
Luaces, O., Díez, J., Alonso-Betanzos, A., Troncoso, A., & Bahamonde, A. (2017) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.06.024 · Cited by 21
A peer assessment approach to project based blended learning course in a Vietnamese higher education
Nguyen, V. A. (2016) · Education and Information Technologies · doi:10.1007/s10639-016-9539-0 · Cited by 26
“Although the implementation of evaluation process faced many obstacles especially in classes with a big number of students (Kulkarni et al. 2013; Suen 2014).”
Background
Gut Instinct
Pandey, V., Amir, A., Debelius, J. W., Hyde, E. R., Kościółek, T., Knight, R. D., & Klemmer, S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025769 · Cited by 24
“Similarly, global-scale distributed contributions like peer review have enabled massive online courses to offer creative, openended assignments through peer review [41].”
BackgroundSelf-citation
EFFECTIVE BLENDED LEARNING IN HIGHER EDUCATION DURING COVID-19
N.A., J. P. S., Kalaichelvi, R., Elumalai, K. V., & Alqahtani, M. S. M. (2022) · Information Technologies and Learning Tools · doi:10.33407/itlt.v88i2.4438 · Cited by 23
“Further, the data-driven approach of peer interaction will increase blended learning [6].”
BackgroundInfluential
On the validity of peer grading and a cloud teaching assistant system
Vogelsang, T., & Ruppertz, L. (2015) · International Conference on Learning Analytics and Knowledge · doi:10.1145/2723576.2723633 · Cited by 31
“CPRwasused for two consecutive Coursera oﬀerings of the course Human Computer Interaction, taught by Scott Klem-mer[13] and analyzed afterwards[20] with statistical models close to the Bayesian approach used by Goldin et. al in smaller classrooms[1, 10].”
BackgroundMethod
The Effect of Peer Assessment Rubrics on Learners' Satisfaction and Performance Within a Blended MOOC Environment
Yousef, A. M. F., Wahid, U., Chatti, M. A., Schroeder, U., & Wosnitza, M. (2015) · International Conference on Computer Supported Education · doi:10.5220/0005495501480159 · Cited by 23
“This study included the design of a questionnaire adapted from (Brindley & Scoffield, 1998; Wolf & Stevens, 2007; Kulkarni et al., 2013).”
BackgroundMethod

Authors: Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
Affiliations: Stanford University (US)
Stanford University, and Coursera, Inc., Stanford, CA
Coursera, Inc., Mountain View, CA
ORCID: Kulkarni, Chinmay
Cheng, Justin
Koller, Daphne
Klemmer, Scott R.
Venue: ACM Transactions on Computer-Human Interaction
Publisher: Association for Computing Machinery (ACM)
Published: 2013-12
Volume: 20
Issue: 6
Pages: 1-31
ISSN: 1073-0516 (ISSN-L)
1557-7325
DOI: 10.1145/2505057
OpenAlex: W2088700588
Cited by: 390
References: 97
Citations by year: 2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
FWCI: 199.15
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computer-Human Interaction (publishedVersion)
Funders: Division of Information and Intelligent Systems
Hasso Plattner Design Thinking Program
Topics: Social Sciences › Education › Student Assessment and Feedback
Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Social Sciences › Education › Online and Blended Learning
Keywords: peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Peer and self assessment in massive online classes

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Statistics that don’t add up

Statistics recomputed

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks