CoChat
Sign Up
Cited by 390Student Assessment and FeedbackOnline Learning and AnalyticsOnline and Blended Learning

Peer and self assessment in massive online classes

Open access
Kulkarni, Chinmay, Wei, Koh Pang, Le, Huy, Chia, Daniel, Papadopoulos, Kathryn, Cheng, Justin, Koller, Daphne, Klemmer, Scott R.·2013·ACM Transactions on Computer-Human Interaction
DOI 10.1145/2505057
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students’ grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers’ work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.

Integrity signals

Registry

Crossref · 10.1145/2505057

Cites

101 of 101 references verified0 citation errors · 0 not checkable

Statistics

2 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

2 of 52 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Accurate, actionable feedback helps students improve their work.

    “Accurate, actionable feedback helps students improve their work [Nicol and Macfarlane-Dick 2006;Boud 2000].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Nicol and Macfarlane-Dick 2006 source
    • cites Boud 2000 source
  • Premisecitation

    Crowdworkers perform better when they are intrinsically motivated by the task's importance.

    “First, crowdworkers perform better when they are intrinsically motivated by the task's importance [Cheshire and Antin 2008].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Cheshire and Antin 2008 source

      The Discussion (page unavailable) says that high intrinsic motivation can sustain contributions and that added incentives may reduce them. But the paper discusses contributors to online information pools, not crowdworkers, and does not establish that motivation arising from a task’s importance improves their performance. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    Over both iterations, the average absolute difference between peer and staff grades was 3%.

    “Overall, the correlation between peer grades and staf-assigned grades was r = 0.73, and the average absolute difference between peer and staff grades was 3% (positive and negative errors were approximately balanced).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.
  • Findingown data

    On a 7-point Likert scale, the median rating of peer feedback and peer assessment as learning experiences was 6 (7 = very valuable).

    “On a 7-point Likert scale, the median rating was 6 (7 = very valuable).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Survey respondents are representative of the class.
  • Findingown data

    In a between-subjects experiment, giving students feedback about whether they scored peers high or low decreased mean error by 0.97% (6.77% experimental vs 7.74% control).

    “A between-subjects experiment found a 0.97% decrease in mean error (6.77% in the experimental group vs. 7.74% in the control group).”
    Quote in the paperSubstantial claim
  • May combine two claims. 56
  • Classified as probably not a substantial claim. 56
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.

    “Students reported both self-assessment and peer assessment to be valuable, and that they played different roles.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • May combine two claims.

    Findingown data

    The Fall 2012 in-person class had 32.8% of submissions with a self-grade within 5% of the staff grade, and 60.8% within 10%.

    “The in-person class had 32.8% of submissions with a self-grade within 5% of the staff grade, and 60.8% of submissions within 10% (Figure 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Over the second iteration, 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.

    “Only 4.0% of submissions had all peer grades agreeing within 5%, and 16.9% within 10%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.

  • May combine two claims.

    Findingown data

    Self-assessment was useful for identifying mistakes and reflection.

    “It was also useful for identifying mistakes and reflection (Figure 8).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • May combine two claims.

    Findingown data

    In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.

    “In end-of-course surveys, students reported both receiving peer feedback and performing peer assessment to be valuable learning experiences.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • May combine two claims.

    Findingown data

    Evaluating peers was useful for inspiration and to see other perspectives.

    “Evaluating peers was useful for inspiration and to see other perspectives.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • May combine two claims.

    Findingown data

    Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.

    “Self-assessment provided students an opportunity to look at their own work again and encouraged comparing it with others' work they had assessed.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • May combine two claims.

    Findingown data

    In the third iteration, 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.

    “For these submissions, the average disagreement between staff raters (defined as the median difference between a staff grade and the mean staff grade) was 6.7%; 28% of submissions had all staff grades within 5% of the assignment grade, and 42% within 10%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    55 students left comments expressing appreciation or receptiveness to the feedback; none expressed resentment.

    “Notably, 55 students left comments expressing their appreciation or receptiveness to this feedback; none expressed resentment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10%.

    “The five assignments in order had 23.7%, 29.4%, 38.4%, 39.5%, and 37.1% within 5% of staff, and 47.4%, 63.8%, 61.8%, 63.3%, and 64.2% within 10% (Figure 10b).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.

  • May combine two claims.

    Findingown data

    Increasing the number of raters per submission helps accuracy but quickly yields diminishing returns.

    “Increasing the number of raters per submission helps accuracy but quickly yields diminishing returns (Figure 9).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The arithmetic mean, geometric mean, and a clipped arithmetic mean that only considers the middle three grades all do worse than the median.

    “Similarly, the arithmetic mean, the geometric mean, and a clipped arithmetic mean (that only considers the middle three grades) all do worse than the median.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Creating fortune cookies took a teaching assistant 3 to 4 hours per assignment.

    “Creating fortune cookies took a teaching assistant 3 to 4 hours per assignment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Errors are approximately evenly spread across the median, so adding a constant correction term to the median grade does not significantly improve accuracy.

    “In addition, errors are approximately evenly spread across the median, so adding a constant correction term to the median grade does not significantly improve accuracy either.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    94.9% of submissions received at least one piece of written feedback, compared to 83% without cookies.

    “Because submissions were assessed by multiple students, 94.9% of submissions received at least one piece of written feedback (compared to 83% without cookies);”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The difference between the median peer grade from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition.

    “The difference between the median peer grade obtained by sampling from the feedback condition and the staff grade was 6.77%, compared to 7.74% in the no-feedback condition (Figure 13).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.

  • May combine two claims.

    Findingown data

    The own-country grading effect is consistent when raters and submitters from the United States are removed, but is smaller (mean difference 1.98%, t(12,863)=2.0, p<0.05).

    “This effect is consistent when the raters and submitters from the largest student enrollment (United States) are removed, but is smaller (the mean difference drops to 1.98%, t(12, 863) = 2.0, p < 0.05).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Increasing the number of assessments per submission from 5 to 11 increases the number of assignments graded within 5% of the staff grade by 3.8% and those graded within 10% by 3.6%.

    “Increasing the number of assessments per submission from 5 to 11 increases the number of assignments that were graded within 5% of the staff grade by 3.8%, and those graded within 10% by 3.6%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.

  • May combine two claims.

    Findingown data

    Using the median yields an accuracy of 35% of samples within 5% and 58.7% within 10%.

    “In contrast, using the median yields an accuracy of 35% of samples within 5%, and 58.7% within 10%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.

  • May combine two claims.

    Findingown data

    The increase in peer-staff agreement is seen both for the class as a whole and for students who submit all assignments.

    “This increase is seen both for the class as a whole and for students who submit all assignments (i.e., excluding students who drop out).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Staff grades are a valid ground truth for assessing peer grading accuracy.

  • Classified as probably not a substantial claim.

    Findingnone given

    Some rubrics sometimes used a nonparallel grammatical structure across sentences.

    “Some rubrics sometimes used a nonparallel grammatical structure across sentences.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Some students probably attempted to game their score by strategically overreporting their grade.

    “Still, some students probably attempted to game their score by strategically overreporting their grade (Figure 18).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Our early experiments suggest that greater accuracy is possible by weighting different raters' grades differently.

    “In addition, our early experiments suggest that greater accuracy is possible by weighting different raters' grades differently, an important topic for future work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although university-like deadlines helped generate interest in online classes, campus-paced deadlines are too rigid online.

    “Although universitylike deadlines helped generate interest in online classes [Lewin 2013c], we found that campus-paced deadlines are too rigid online.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lewin 2013c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Five students from across the world participated as community TAs.

    “Five students from across the world participated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Together, community TAs answered 547 questions on the forum, while staff answered 582 questions.

    “Together, community TAs answered 547 questions on the forum, while staff (three local TAs and the instructor) answered 582 questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Anecdotally, students are generally satisfied with their overall grade but are frustrated by inaccurate qualitative feedback from some peers.

    “Anecdotally, we have noticed that students are generally satisfied with their overall grade but are frustrated by inaccurate qualitative feedback from some peers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    She also participated heavily in the forums and gathered staff-like respect from her peers.

    “She also participated heavily in the forums and gathered staff-like respect from her peers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In iteration 1, questions with parallel structure had lesser disagreement, both among peer graders and between the median grade and the self-assessed grade.

    “In iteration 1, questions with parallel structure had lesser disagreement, both among peer graders and between the median grade and the self-assessed grade.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The age distribution of the class is remarkably similar across both iterations.

    “The age distribution of the class is remarkably similar across both iterations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The median agreement was 42.% and 65.5%, respectively.

    “(Recall that the median agreement was 42.% and 65.5%, respectively).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Most fortune cookie feedback was personalized.

    “Most fortune cookie feedback was personalized.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Staff rated the prototype low because while fully functional, the design did not seem appropriate to the goal, and the diary aspect seemed to be the main aspect of the app yet was hidden behind a search bar.

    “Staff rated it low, because”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Peers rated the prototype highly for being interactive.

    “Peers rated the prototype highly for being”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Students copied snippets of feedback (fortune cookies), pasted them in a textbox, and optionally added an explanation.

    “Students copied snippets of feedback (fortune cookies), pasted them in a textbox, and optionally added an explanation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Staff praised the application as fully interactive, with complete page flow.

    “Staff praised it as”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.

    “In the first iteration, 28.7% of such samples were within 5% of the staff grade, and 44.9% within 10%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Fall 2012 iteration of the class had 17,915 participants.

    “(b) Fall 2012 (iteration 2): 17,915 participants”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    20% of students assessed more than the required five submissions.

    “Students were required to assess five submissions, and 20% of students assessed more than required.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Even though this was a completely online class, students self-organized to meet up in 10 locations around the world including London, San Francisco, New York City, Buenos Aires, Aachen (Germany), and Bangladesh.

    “Even though this was a completely online class, students self-organized to meet up in 10 locations around the world including London, San Francisco, New York City, Buenos Aires, Aachen (Germany), and Bangladesh.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Students in the second (Fall 2012) iteration of the class reported a self-grade >5% higher than peer grade more frequently, and so got their self-grade less frequently.

    “Students in the second (Fall 2012) iteration of the class reported a self-grade >5% higher than peer grade more frequently, and so got their self-grade less frequently.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Spring 2012 iteration of the class had 10,190 participants.

    “(a) Spring 2012 (iteration 1): 10,190 participants”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Feedback on grading accuracy reduced the overall error in assessment and made the range of errors smaller.

    “Feedback on grading accuracy reduced the overall error in assessment and made the range of errors smaller.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    26.3% of peer grades were within 5% of staff grades, and 46.7% within 10%.

    “26.3% of grades were within 5% of staff grades, and 46.7% within 10%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the second iteration, 24.0% of such samples were within 5% of the staff grade, and 40.63% within 10%.

    “(b) Same graph for second iteration of the class: 24.0% of such samples were within 5% of the staff grade, and 40.63% within 10%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    Some peers rated the application unpolished and asked the student to try to make the UI less coloured.

    “while some peers rated it”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers.

    “Similarly, self-assessment helps students reflect on gaps in their understanding, making them more resourceful, confident, and higher achievers [Zimmerman and Schunk 2001;Pintrich 1995;Pintrich and Zusho 2007], and provides learning gains not seen with external evaluation [Dow et al. 2012].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Zimmerman and Schunk 2001 source
    • cites Pintrich 1995 source
    • cites Pintrich and Zusho 2007 source

  • May combine two claims.

    Premisecitation

    Human grading more easily provides context-appropriate responses and better handles ill-specified constraints.

    “Furthermore, human grading more easily provides context-appropriate responses and better handles ill-specified constraints [Hearst 2000].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hearst 2000 source

  • May combine two claims.

    Premisecitation

    Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion.

    “Peer assessment can increase student involvement and maturity, lower the grading burden on staff, and enhance classroom discussion [Boud 1995].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Boud 1995

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Massive online classes provide a valuable living lab for exploring peer-sourcing approaches.

    “Massive online classes provide a valuable living lab [Chi 2009;Carter et al. 2008] for exploring peer-sourcing approaches, and our hope is that peer-sourcing insights from massive classes will contribute techniques that apply more broadly.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chi 2009

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Carter et al. 2008 source

  • May combine two claims.

    Premisecitation

    Showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically.

    “Second, showing a list of common, assignment-specific problems that the submission could have potentially reduces inhibition and encourages peers to think critically [Galinsky and Moskowitz 2000].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Galinsky and Moskowitz 2000 source

  • May combine two claims.

    Premisecitation

    Providing raters a list of potential feedback items changes a recall/identification task into a recognition task, reducing the cost of giving feedback.

    “First, providing raters a list of potential feedback items changes a recall/identification task into a recognition task.This reduces the cost of giving feedback [Anderson and Bower 1972;Nielsen 1994].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Anderson and Bower 1972 source
    • cites Nielsen 1994 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Consensus-based mechanisms that encourage sharing perspectives may improve agreement.

    “Consensus-based mechanisms that encourage sharing perspectives may improve agreement [Amabile 1982].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Amabile 1982 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Students found rating their peers valuable or enjoyable and/or believed it would help their peers.

    “We infer from this that students found rating their peers valuable or enjoyable and/or they believed it would help their peers.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Survey respondents are representative of the class.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Grading was double-blind, so raters did not see the names of submitters.

    “We remind the reader that grading was double-blind, so raters did not see the names of submitters.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior work has demonstrated that feedback improves the quality of crowd work.

    “Prior work has demonstrated that feedback improves the quality of crowd work [Dow et al. 2012]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Dow et al. 2012 source

  • May combine two claims.

    Interpretationown data

    Students expend the same amount of effort writing feedback, and using fortune cookies allows this effort to be used to add to the fortune cookie text.

    “This suggests that students expend the same amount of effort writing feedback, and using fortune cookies allows this effort to be used to add to the fortune cookie text.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The repeated-sampling numbers for the five assignments are based on a smaller number of staff-graded assignments and are more susceptible to variations in staff grades for a particular submission.

    “Note that both these numbers are based on repeated sampling from a smaller number of staff-graded assignments.As such, they are more susceptible to variations in staff grades for a particular submission.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Students sometimes believe that teachers grade on personal taste and focus on currying favor.

    “Students sometimes believe that teachers grade on personal taste and focus on currying favor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Students in online classrooms are much more diverse both demographically and in their objectives in taking the class.

    “students in online classrooms are much more diverse both demographically and in their objectives in taking the class”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Breslow et al. 2013

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    While peer assessment works well for the large majority of students, students who receive an unfair assessment may lose motivation.

    “Lastly, while peer assessment works well for the large majority of students, students who receive an unfair assessment may lose motivation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Many types of cheating are currently possible and unchecked in online classes.

    “Many types of cheating are currently possible and unchecked in online classes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation

    Iterative design often pays big dividends, and assessment systems are no exception.

    “Iterative design often pays big dividends [Nielsen 1993], and assessment systems are no exception.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nielsen 1993 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    These differences require rethinking the design of the classroom.

    “These differences require rethinking the design of the classroom.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Providing explicit grading criteria, especially in advance, helps convey to students that grading is fair, consistent, and based on the quality of their work.

    “Also, providing explicit grading criteria (especially in advance) helps convey to students that grading is fair, consistent, and based on the quality of their work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The large scale of online classes allows data-driven iterative improvements of classroom materials in ways that small classes may not.

    “The large scale of online classes allows data-driven iterative improvements of classroom materials in ways that small classes may not.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A valuable skill for success is the theory of mind to intuit how others perceive one's performance.

    “A valuable skill for success is the theory of mind to intuit how others perceive one's performance [Boud 1995]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Boud 1995

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse, and defaults have a powerful impact on behavior.

    “In key areas, online education encodes pedagogy into software, which increases consistency and supports reuse-and defaults have a powerful impact on behavior [Palen 1999].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Palen 1999 source

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Community TAs possessed enough knowledge to effectively offer information and guidance because they had done well in the class.

    “First, because community TAs had done well in the class, they possessed enough knowledge to effectively offer information and guidance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Peer assessment addresses the problem of differing interpretations by providing raters with expert-made rubrics, but some differences may persist.

    “Peer assessment addresses this problem by providing raters with expert-made rubrics, but some differences may persist.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

    Interpretationreasoning

    Community TAs could easily empathize with issues students faced and effectively offer social support because they had taken the class recently.

    “Second, because they had taken the class recently, they could easily empathize with issues students faced and also could effectively offer social support.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible.

    “platforms make some kinds of data, such as engagement with course material, more plentiful and finer grained, while making other information, such as facial expressions of confusion, completely inaccessible [Breslow et al. 2013].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Breslow et al. 2013

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The role of teaching staff (TAs) changes: instead of spending a majority of their time grading, they spend a large fraction of their time fielding student questions, mentoring students, boosting student morale and autonomous perspective, and making data-driven revisions to class materials.

    “The role of teaching staff (TAs) changes too.Instead of spending a majority of their time grading, they spend a large fraction of their time fielding student questions, mentoring students, boosting student morale and autonomous perspective, and making data-driven revisions to class materials.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts, leading to more conceptual comments.

    “Third, because fortune cookies sometimes used terminology learned in class, they may have triggered cued recall of these concepts [Little and Bjork 2012], leading to more conceptual comments.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Little and Bjork 2012

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Because of the diversity of submissions, articulating assessment criteria a priori can be extremely difficult.

    “Because of the diversity of submissions, this can be extremely difficult to do a priori.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    The design studio offers inspiration for supporting student leadership and community learning in the online classroom.

    “Again, the design studio offers inspiration [Schön 1985;Pendleton-Jullian 2010].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Schön 1985

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Pendleton-Jullian 2010

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    After revision, rubrics can scale well for both students and other teachers to use.

    “After revision, these rubrics can scale well for both students and other teachers to use.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Students often have work commitments, and holidays are at different times around the world.

    “For instance, students often have work commitments, and holidays are at different times around the world.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems.

    “In addition, a studio facilitates dialogue between students, instructors, and artifacts that helps students collaboratively learn difficult concepts and solve problems [Schön 1985].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Schön 1985

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Novices and experts differ in their tacit understanding of value.

    “This may be because novices and experts differ in their tacit understanding of value [Kaufman et al. 2008].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kaufman et al. 2008 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    By making not only the results of work but also the process of creation highly visible, the design studio helps students learn and build awareness through observation.

    “By making not only the results of work but also the process of creation highly visible, it helps students learn and build awareness through observation [Klemmer et al. 2006].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Klemmer et al. 2006 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    As online classes count for more benefits, such gaming may increase.

    “As online classes count for more benefits, such gaming may increase.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Peer assessment imposes a particular schedule on class and limits student flexibility.

    “Second, peer assessment imposes a particular schedule on class and limits student flexibility.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Even examples in prior work on using rubrics suffer from nonparallel grammatical structure.

    “This is not uncommon: even examples in prior work on using rubrics suffer from this problem (e.g., Andrade [2005]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Andrade [2005] source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Gaming may help students develop the theory of mind skill.

    “and gaming may help students develop this skill.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other.

    “experts and novices both robustly reach consensus about creativity, but their consensual judgments differ from each other [Conti et al. 1996].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Conti et al. 1996 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several universities are teaching physical classes augmented with online materials.

    “Already, several universities are teaching physical classes augmented with online materials [Martin 2012].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Martin 2012 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One aspect of reconceptualizing the student has to do with enrollment and retention.

    “One has to do with enrollment and retention [Kizilcec et al. 2013].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kizilcec et al. 2013 source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Online classes attract students who cannot use traditional universities, such as those working full time.

    “Online classes attract students who cannot use traditional universities, such as those working full time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Human subjects research was reviewed by the Stanford Institutional Review Board through protocol 25001.

    “Human subjects research was reviewed by the Stanford Institutional Review Board through protocol 25001.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Online education requires us to reconceptualize what it means to be a student in many ways.

    “More generally, online education requires us to reconceptualize what it means to be a student in many ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    By removing the constraints of the physical classroom, online classes have made education accessible to many new kinds of students-the new mother, the full-time professional, and the retiree.

    “By removing the constraints of the physical classroom, online classes have made education accessible to many new kinds of students-the new mother, the full-time professional, and the retiree.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Typing one's email address into a webpage is not the same as showing up for the first day of a registrar-enrolled class; it is more like peeking through the window, and the large number of signups indicates that lots of people are curious.

    “Typing one's email address into a webpage is not the same as showing up for the first day of a registrar-enrolled class.It's more like peeking through the window, and what the large number of signups tells us is that lots of people are curious.”
    Quote in the paperNot a substantial claim
93 of 93

  • The paper relies on this without support.

    Staff-assigned grades are a valid ground truth for assessing peer and self grading accuracy.

    “To establish a ground-truth comparison of self-and staff grades, each assignment included four to 10 staff-graded submissions in the peer assessment pool (these were randomly selected).”
    “We evaluated the accuracy of this grading process using the 99 assignments with a staff grade.”

  • The paper relies on this without support.

    Survey respondents are representative of the class.

    “At the end of the class, students were invited to participate in a survey; 3,550 students participated in all.Participation was voluntary, students were not compensated, and the survey did not count toward course credit.”

  • The authors state this as an assumption.

    Grades in different assignments are uncorrelated.

    “(assuming grades in different assignments are uncorrelated)”

  • The paper relies on this without support.

    Staff grades are a valid ground truth for assessing peer grading accuracy.

    “Holding out 10% of ground truth grades”
    “compared to 7.74% in the no-feedback condition”

  • The paper relies on this without support.

    At a physical university, the completion rate for an equivalent class is much higher than in the online class.

    “Needless to say, at a physical university, the completion rate for an equivalent class is much higher.”

  • The paper relies on this without support.

    Peers can help instruction itself.

    “Fourth, peers can help instruction itself.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Techniques such as community TAs, peer assessment, rubrics, video feedback are available to help MOOC instructors scale up provision of perceived personal attention.

    “Techniques such as involvement of community teaching assistants (TAs),21 peer assessment tools,22 well-designed elaborated rubrics with specific content feedback23 and video feedbacks24 are available to help MOOC instructors scale up the provision of perceived personal attention to students.”

    From Towards AI-Powered Personalization in MOOC Learning (2017) · cites this paper as [22]

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value across alpha = .05: F(1, 39) = 2.07, p < .05

    Major

    The paper reports p < .05 here; recomputing from the test statistic and its degrees of freedom gives p = .158. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p < .05 · recomputed p = .158

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(1, 39) = 2.07reported p < .05 · recomputed p = .158The two fall on opposite sides of .05

    “F(1, 39) = 2.07, p < .05”

  • Consistentt(24) = 19.9reported p < .05 · recomputed p < .001

    “t(24) = 19.9,p < .05”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure 5 caption names the wrong comparison grade

    Minor

    The results describe Figure 5 as comparing median peer grades with self-assessed grades, but its caption says the percentages measure agreement with staff grades.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 11

  1. H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.
  2. J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http://www.cs.helsinki.fi/en/uutiset/72025.
  3. T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.
  4. T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.
  5. T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.
  6. T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.
  7. T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.
  8. T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.
  9. M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002).
  10. J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/&quest;p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http://wp.sigmod.org/&quest;p=165.
  11. O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.

Cited by

Sorted by citations

Related

Authors
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R.
Affiliations
  • Stanford University (US)
  • Stanford University, and Coursera, Inc., Stanford, CA
  • Coursera, Inc., Mountain View, CA
ORCID
Venue
ACM Transactions on Computer-Human Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2013-12
Volume
20
Issue
6
Pages
1-31
ISSN
  • 1073-0516 (ISSN-L)
  • 1557-7325
DOI
10.1145/2505057
OpenAlex
W2088700588
Cited by
390
References
97
Citations by year
2026: 7 · 2025: 15 · 2024: 27 · 2023: 21 · 2022: 22 · 2021: 30 · 2020: 34 · 2019: 27 · 2018: 46 · 2017: 37 · 2016: 43 · 2015: 56 · 2014: 24 · 2013: 1
FWCI
199.15
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM Transactions on Computer-Human Interaction (publishedVersion)
Funders
  • Division of Information and Intelligent Systems
  • Hasso Plattner Design Thinking Program
Topics
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Education › Online and Blended Learning
Keywords
peer assessment, self-assessment, online learning, grading accuracy, rubric development, peer feedback, student self-assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review