CoChat
Sign Up
Cited by 98Student Assessment and FeedbackOnline Learning and AnalyticsEvaluation of Teaching Practices

Using learning analytics to understand student perceptions of peer feedback

Open access
Misiejuk, Kamila, Wasson, Barbara, Egelandsdal, Kjetil·2021·Computers in Human Behavior
DOI 10.1016/j.chb.2020.106658
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer assessment (PA) is the process of students grading and giving feedback to each other's work. Learning analytics is a field focused on analysing educational data to understand and improve learning processes. Using learning analytics on PA data has the potential to gain new insights into the feedback giving/receiving process. This exploratory study focuses on backward evaluation, an under researched aspect of peer assessment, where students react to the feedback that they received on their work. Two aspects are analysed: 1) backward evaluation characteristics depending on student perception of feedback that they receive on their work, and 2) the relationship between rubric characteristics and backward evaluation. A big dataset (N = 7,660 records) from an online platform called Peergrade was analysed using both statistical methods and Epistemic Network Analysis. Students who found feedback useful tended to be more accepting by acknowledging their errors, intending to revise their text, and praising its usefulness, while students who found the feedback less useful tended to be more defensive by expressing that they were confused about its meaning, critical towards its form and focus, and in disagreement with the claims. Moreover, students mostly suggested feedback improvement in terms of feedback specificity, justification and constructivity, rather than kindness. The paper concludes by discussing the potential and limitations of using LA methods to analyse big PA datasets.

AI snapshot

Written from the paper’s full text.

TL;DR

This exploratory learning-analytics study analyses 7,660 Peergrade peer-assessment records to relate students’ backward evaluation comments and improvement suggestions to perceived feedback usefulness and rubric characteristics. It finds that useful feedback is associated with accepting/grateful comments and positive sentiment, while less useful feedback is associated with defending comments and more improvement suggestions, with rubric characteristics showing only weak relationships.

Key findings

  • Students who found feedback useful tended to be more accepting, grateful, and positive; students who found it less useful tended to be more defensive, confused, critical, or disagreeing. Almost 60% graded feedback extremely useful (FG5 = 0.32) or very useful (FG4 = 0.27), while only 18% graded it not useful at all (FG1 = 0.09) or not very useful (FG2 = 0.1). Defending comments were the most frequent type (def = 0.29), followed by accepting comments (acc = 0.28). In ordinal logistic regression, defending comments predicted lower perceived usefulness (def β = −2.30, p ≤ .001), while accepting-gratitude (β = 0.67, p ≤ .001) and gratitude (β = 0.19, p ≤ .01) predicted higher perceived usefulness relative to accepting comments.
  • Improvement suggestions were mostly not selected: 75% of students chose none, and the most popular suggestions were specificity (25.08%), constructivity (22.23%), and justification (17.26%). Selecting any improvement suggestion predicted a higher likelihood of finding feedback less useful (relevance β = −1.23, constructivity β = −0.84, specificity β = −0.82, kindness β = −0.71, justification β = −0.6, all p ≤ .001). The number of improvement suggestions had a moderate negative relationship with FG5 (rho = −0.453, p < .001).
  • Positive sentiment in backward evaluation comments predicted higher perceived usefulness (β = 0.94, p ≤ .001); sentiment had a weak positive relationship with FG5 (rho = 0.275, p < .001) and a weak negative relationship with FG1 (rho = −0.268, p < .001).
  • Rubric characteristics showed only weak or non-significant relationships with perceived usefulness. More text questions per rubric predicted positive feedback perception (β = 0.001, p ≤ .05), while more questions per rubric was not statistically significant (β = 0.39, p = .093); all reported rubric correlations were very weak.

Methods

  • Exploratory analysis of an anonymised Peergrade dataset collected across many institutions between 2015 and 2017; after retaining English entries and removing missing values, the final dataset was n = 7,660 records. The dataset had 10,197 unique student IDs and 6,329 unique course titles but did not include student hand-ins or contextual information.
  • Variables included feedback grade (1–5 usefulness scale), backward evaluation comments coded as accepting, defending, and/or gratitude via string matching (validated on a random 10% sample with Cohen’s kappa ≥ 0.8 for every code), comment length, part-of-speech tags, Vader sentiment, improvement suggestions, and rubric question type/count.
  • Analyses used descriptive statistics, Spearman rank correlation, backward stepwise regression followed by ordinal logistic regression in R 3.5.0, and Epistemic Network Analysis using the ENA web tool.

Limitations

  • Stated: lack of control variables due to weaknesses in the Peergrade dataset, including absence of student background information, PA context (discipline, educational level, pedagogical approach, course structure), and assignment/final course marks; coding of backward evaluation comments was broad because of the heterogeneous dataset and lack of context; and the dataset did not include the original hand-ins, making it impossible to analyse whether feedback was used to improve work.
  • Stated: the final ordinal regression model showed multicollinearity for text (GVIF = 38.389), BE_comment (GVIF = 8.496), and specificity (GVIF = 15.201), indicating possible bias; BE_comment_length and #_of_qs were not statistically significant.
  • Evident from the text: the data came from a commercial platform not designed for learning analytics, so variables were limited to those available; only English entries were retained, and some feedback may come from groups rather than individual students, though the paper notes these as data constraints.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.chb.2020.106658

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 60 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Students mostly suggested feedback improvement in terms of feedback specificity, justification and constructivity, rather than kindness.

    “Moreover, students mostly suggested feedback improvement in terms of feedback specificity, justification and constructivity, rather than kindness.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The multiple-choice improvement suggestions scale validly indicates how the feedback students receive on their work could have been improved.
    • Rests on an assumption: The variables available in the Peergrade dataset are adequate proxies for the constructs used in the analysis.
  • Findingown data

    Students who found feedback useful tended to be more accepting by acknowledging their errors, intending to revise their text, and praising its usefulness.

    “Students who found feedback useful tended to be more accepting by acknowledging their errors, intending to revise their text, and praising its usefulness”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Peergrade feedback grade scale validly measures students' perception of feedback usefulness.
    • Rests on an assumption: The variables available in the Peergrade dataset are adequate proxies for the constructs used in the analysis.
  • Findingown data

    Students who found feedback less useful tended to be more defensive by expressing confusion about its meaning, being critical towards its form and focus, and disagreeing with its claims.

    “students who found the feedback less useful tended to be more defensive by expressing that they were confused about its meaning, critical towards its form and focus, and in disagreement with the claims”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Peergrade feedback grade scale validly measures students' perception of feedback usefulness.
    • Rests on an assumption: The variables available in the Peergrade dataset are adequate proxies for the constructs used in the analysis.
  • Findingown data

    Only 18% of all feedback grades were not useful at all (FG1 = 0.09) or not very useful (FG2 = 0.1).

    “Only 18% of all feedback grades were not useful at all (FG1 = 0.09), or not very useful (FG2 = 0.1).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.
  • Findingown data

    The majority of students (almost 60%) graded feedback extremely useful (FG5 = 0.32) or very useful (FG4 = 0.27).

    “The majority of students (almost 60%) graded feedback extremely useful (FG5 = 0.32), or very useful (FG4 = 0.27)”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.
  • May combine two claims. 39
  • Classified as probably not a substantial claim. 45

  • May combine two claims.

    Findingown data

    Defending comments are the most frequent type of comment (def = 0.29) followed by accepting comments (acc = 0.28).

    “Defending comments are the most frequent type of comment (def = 0.29) followed by accepting comments (acc = 0.28).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The proportion of gratitude and accepting comments are highest for FG5 (grat = 0.098; acc_grat = 0.078) and FG1 (grat = 0.001; acc_grat = 0.0003).

    “The proportion of gratitude and accepting comments are highest for FG5 (grat = 0.098; acc_grat = 0.078) and FG1 (grat = 0.001; acc_grat = 0.0003) as depicted in Fig. 3.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    The proportion of accepting comments is highest for FG5 (acc = 0.125), whereas the proportion of defending comments is highest for FG1 (def = 0.072).

    “Moreover, the proportion of accepting comments is the highest for FG5 (acc = 0.125), whereas the proportion of defending comments is the highest for FG1 (def = 0.072).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    A longer BE_comment_length indicates that students are more likely to perceive feedback as less useful (β = -0.88), but this result is not statistically significant (p = .077).

    “Furthermore, a longer BE_comment_length indicates that students are more likely to perceive feedback as less useful (β = -0.88),however, this result is not statistically significant (p=.077).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    FG3 has strong connections between fewCat and almost all improvement suggestions (specificity, 0.19; constructivity, 0.14; justification, 0.10; relevance, 0.05), and strong relationships among suggestions: specificity-constructivity (0.09), specificity-justification (0.07), and constructivity-justification (0.05).

    “Fig. 4c shows that FG3 has not only strong connections between fewCat and almost all improvement suggestions (specificity, 0.19; constructivity, 0.14; justification, 0.10; relevance, 0.05), but also some strong relationships among the improvement suggestions themselves are visible: specificity-constructivity (0.09), specificity-justification (0.07), and constructivity-justification (0.05).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.
    • Rests on an assumption: The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

  • May combine two claims.

    Findingown data

    The highest proportion of comments coded with more than one code is for FG3 (acc_def = 0.26; def_grat = 0.06; acc_def_grat = 0.07) and FG4 (acc_def = 0.25; def_grat = 0.05; acc_def_grat = 0.009).

    “The highest proportion of comments coded with more than one code is for FG3 (acc_def = 0.26; def_grat = 0.06; acc_def_grat = 0.07), and FG4 (acc_def = 0.25; def_grat = 0.05; acc_def_grat = 0.009).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    Constructivity, justification and specificity have weak negative relationships with FG5 (constructivity, rho = -0.284, p<.001; justification, rho = -0.231, p<.001; specificity, rho = -0.284, p<.001), while relevance has a weak positive relationship with FG1 (rho = 0.292, p<.001).

    “Constructivity, justification and specificity have weak negative relationships with FG5 (constructivity, rho = -0.284,p=<.001; justification, rho = -0.231,p=<.001; specificity, rho = -0.284,p=<.001), while relevance has a weak positive relationship with FG1 (rho = 0.292, p=<.001).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.
    • Rests on an assumption: The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

  • May combine two claims.

    Findingown data

    Sentiment has a weak positive relationship with FG5 (rho = 0.275, p<.001) and a weak negative relationship with FG1 (rho = -0.268, p<.001).

    “The sentiment has a weak positive relationship with FG5 (rho = 0.275, p=<.001), and a weak negative relationship with FG1 (rho = -0.268,p=<.001).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    More #_of_qs per rubric makes students more likely to perceive feedback as more useful rather than less useful (β = 0.39), but this result is not statistically significant (p = .093).

    “and more #_of_qs per rubric makes students more likely to perceive feedback as more useful rather than less useful (#_of_qs, β = 0.39), however, this result is not statistically significant (p = .093).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • May combine two claims.

    Findingown data

    The most popular improvement suggestion was specificity (25.08%), followed by constructivity (22.23%) and justification (17.26%).

    “The most popular improvement suggestion was specificity (25.08%), followed by constructivity (22.23%), and justification (17.26%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

  • May combine two claims.

    Findingown data

    #_of_impr_suggs has a moderate negative relationship with FG5 (rho = -0.453, p<.001) and a weak positive relationship with FG1 (rho = 0.214, p<.001), FG2 (rho = 0.228, p<.001), and FG3 (rho = 0.236, p<.001).

    “#_of_impr_suggs has a moderate negative relationship with FG5 (rho = -0.453,p=<.001), and a weak positive relationship with FG1 (rho = 0.214, p=<.001),FG2 (rho = 0.228, p=<.001), and FG3 (rho = 0.236, p=<.001).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.
    • Rests on an assumption: The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

  • May combine two claims.

    Findingown data

    Defending coded BE comments (def) show a weak positive relationship with FG1 (rho = 0.362, p<.001) and FG2 (rho = -0.264, p<.001), and a weak negative relationship with FG5 (rho = -0.390, p<.001).

    “Only defending coded BE comments (def) show a weak positive relationship with FG1 (rho = 0.362, p=<.001) and FG2 (rho = -0.264,p=<.001), and a weak negative relationship with FG5 (rho = -0.390,p=<.001).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most used part of speech is verb (mean = 0.205, median = 0.205) followed by noun (mean = 0.168, median = 0.158).

    “Moreover, most used part of speech is verb (mean = 0.205, median = 0.205) followed by noun (mean = 0.168, median = 0.158) (see Fig. 2e).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    75% of students did not choose any improvement suggestion, 3% chose four, and 1% selected all five improvement suggestions.

    “75% of students did not choose any improvement suggestion and only 3% chose four, whereas 1% selected all five improvement suggestions, as shown in Fig. 2f.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

  • May combine two claims.

    Findingown data

    9.5% of the variance on the x-axis and 24.5% of the variance on the y-axis are explained by the ENA model.

    “9.5% of the variance on the x-axis and 24.5% of the variance on the y-axis are explained by this model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Giving feedback is just as effective, or more so, for improving writing performance as receiving feedback.

    “some studies have found that giving feedback is just as effective, or more so, for improving writing performance as receiving feedback (Graner, 1987;Lundstrom & Baker Smemoe, 2009).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Graner, 1987 source

      Not checked in the free run.

    • cites Lundstrom & Baker Smemoe, 2009

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Jonsson's review concluded that the feedback gap relates to students' understanding of the feedback, as well as strategies and opportunities to use the feedback purposefully.

    “A review by Jonsson (2013) concluded that this gap relates to student's understanding of the feedback, as well as strategies and opportunities to use the feedback purposefully.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jonsson (2013) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Peer assessment appears to be just as effective as teacher assessment when it comes to enhancing students' academic achievement.

    “Although PA is performed by the students themselves, studies have found that PA appears to be just as effective as teacher assessment when it comes to enhancing students' academic achievement (Double et al., 2018).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Double et al., 2018

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    A higher number of praise comments predicted lower agreement with the feedback and a lower revision rate.

    “However, a higher number of praise comments predicted lower agreement with the feedback and a lower revision rate.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wu and Schunn (2020)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Van der Pol et al.'s second study showed that students agreed more with the feedback that they perceived as useful.

    “the second study showed that students agreed more with the feedback that they perceived as useful”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van der Pol et al. (2008) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    No research looks at how student backward evaluation might provide insight into a rubric's quality.

    “We found no research that looks at how student BE might provide insight into a rubric's quality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Only a few peer assessment studies include backward evaluation in their analysis, typically as a helpfulness scale or a free-text comment.

    “Only a few PA studies include BE in their analysis, typically as a helpfulness scale or a free-text comment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Feedback with concrete solutions contributed to a higher understanding of feedback.

    “Feedback with concrete solutions contributed to a higher understanding of feedback”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wu and Schunn (2020)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Van der Pol et al.'s first study found that a higher perceived importance of feedback on their work by students corresponded with more revisions in their written work.

    “Their first study found that a higher perceived importance of feedback on their work by students corresponded with more revisions in their written work”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van der Pol et al. (2008) source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Student's agreement with the feedback, and not perceived feedback usefulness, correlated with higher rate of revision.

    “Student's agreement with the feedback, and not perceived feedback usefulness, correlated with higher rate of revision.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van der Pol et al. (2008) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    Some student feedback is given from a group of students and not an individual student, so what appears to be a single feedback might actually come from a group.

    “From our own experience with university instructors using Peergrade we know that some student feedback is given from a group of students and not an individual student, thus what appears to be a single feedback might actually come from a group.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Many commercial online learning environments that students interact with do not track, or log, useful data.

    “many commercial online learning environments that students interact with do not track, or log, useful data”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roschelle and Krumm (2016)

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The dependent variable in the current study, the feedback grade, is an ordinal categorical variable with five levels.

    “The dependent variable in the current study, the feedback grade (FG), is an ordinal categorical variable with five levels (recall Table 1).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Feedback does not always result in student improvement and may in some cases inhibit learning rather than promote it.

    “Nevertheless, feedback does not always result in student improvement, and may in some cases inhibit learning rather than promote it.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hattie & Timperley, 2007 source

      Not checked in the free run.

    • cites Kluger & DeNisi, 1996 source

      Not checked in the free run.

    • cites Shute, 2008 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The quality of the results of learning analytics on big data is dependent on the quality of the questions asked.

    “Researchers agree that the quality of the results of LA on big data is dependent on the quality of the questions asked (e.g., Kitchin, 2013;Prinsloo & Slade, 2017).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Kitchin, 2013 source

      Not checked in the free run.

    • cites Prinsloo & Slade, 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Learning analytics is most often used on large datasets.

    “is most often used on large datasets (Misiejuk & Wasson, 2017).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Misiejuk & Wasson, 2017

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Ordinal logistic regression is the recommended method to model an ordinal dependent variable, since metric methods might distort the analysis results.

    “The recommended method to model an ordinal dependent variable is ordinal logistic regression, since metric methods might distort the analysis results (Liddell & Kruschke, 2018).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liddell & Kruschke, 2018 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Working with exhaust big data collected as a by-product of the primary task can be challenging, and the data generated may be messy and dirty.

    “The data generated may be messy and dirty (Kitchin, 2014), however, for many researchers it is a reality to work this kind of data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kitchin, 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    After data pre-processing and removal of observations with missing values, the final dataset was n = 7,660 records.

    “After data pre-processing and removal of observations with missing values, the final dataset was n = 7,660 records.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The analysis of large datasets poses new challenges, such as the automated coding of written peer feedback or a limited interpretation of analysis results in an educational context due to lack of contextual data.

    “At the same time, the analysis of large datasets poses new challenges, such as the automated coding of written peer feedback, or a limited interpretation of the analysis results in an educational context due to lack of contextual data (Mangaroska & Giannakos, 2018;Xiong, Litmaan, & Schunn, 2012).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Mangaroska & Giannakos, 2018

      Not checked in the free run.

    • cites Xiong, Litmaan, & Schunn, 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The number of students refers only to the number of participants visible in a particular peer assessment activity and may not reflect the overall number of students in a course.

    “It is important to note that the number of students refers only to the number of participants that are visible in a particular PA activity and may not reflect the overall number of students in a course.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Variations in the effect of feedback have been related to content, form and timing of the feedback.

    “Variations in the effect of feedback have been related to content, form and timing of the feedback, and studies have indeed found variations based on these factors”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Hattie & Timperley, 2007 source

      Not checked in the free run.

    • cites Kluger & DeNisi, 1996 source

      Not checked in the free run.

    • cites Shute, 2008 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Spearman rank correlation is more appropriate for correlation of ordinal variables than standard methods such as Pearson correlation.

    “It was decided to conduct Spearman rank correlation, since it is more appropriate for correlation of ordinal variables than standard methods, such as Pearson correlation (Mukaka, 2012).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Mukaka, 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The quality of feedback depends not only on students' skills, but also on the feedback questions that students are asked.

    “As Nilson (2003) noticed the quality of feedback does not only depend on student's skills, but also the feedback questions that students are asked.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nilson (2003) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Big data is often collected by the private sector as an auxiliary function of their core business in order to improve business processes and document organization activities.

    “Big data is often collected by the private sector “as an auxiliary function of their core business” in order to “improve business processes and to document organization activities” (Buchanan, Gesher, & Hammer, 2015, p. 93).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Buchanan, Gesher, & Hammer, 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Previous research on rubrics in peer assessment focused on the amount of guidance necessary in a rubric.

    “Previous research on rubrics in PA focused on the amount of guidance necessary in a rubric.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    In this study the authors analyse a dataset provided by a commercial peer assessment platform, where they did not influence the data collection.

    “In this study we analyse a dataset provided by a commercial PA platform, where we did not influence the data collection.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    The data learning analytics researchers analyse is generated by the tools, as decided by the tool developer and not by the researcher who will use the data.

    “Thus, the data they analyse is that which is generated by the tools, as decided by the tool developer and not by the researcher who will use the data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Assessment enhances performance when self-efficacy is high and impedes performance when self-efficacy is low.

    “there is evidence that assessment enhances performance when self-efficacy is high and impedes performance when self-efficacy is low (Beckmann, Beckmann, & Elliott, 2009;Birney, Beckmann, Beckmann, & Double, 2017;Kluger & DeNisi, 1996).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Beckmann, Beckmann, & Elliott, 2009

      Not checked in the free run.

    • cites Birney, Beckmann, Beckmann, & Double, 2017

      Not checked in the free run.

    • cites Kluger & DeNisi, 1996 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The dataset does not contain the student hand-ins, due to consent issues.

    “but does not contain the student hand-ins, due to consent issues”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Educational big data coming from the private sector often has to be combined with other data sources to enrich the set of attributes to be studied.

    “Another important aspect of educational big data coming from the private sector is that it often has to be combined with other data sources “to enrich the set of attributes to be studied” (Buchanan et al., 2015, p. 94).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Buchanan et al., 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Feedback promoting self-efficacy leads to better self-regulation and more effort devoted to the task.

    “Feedback promoting self-efficacy leads to better selfregulation and more effort devoted to the task (Hattie & Timperley, 2007)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hattie & Timperley, 2007 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Another variation in the effectiveness of feedback is related to how individual students perceive and use feedback.

    “Another variation in the effectiveness of feedback is related to how individual students perceive and use feedback”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bloxham & Campbell, 2010

      Not checked in the free run.

    • cites Carless, Salter, Yang, & Lam, 2010 source

      Not checked in the free run.

    • cites Hattie & Gan, 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The authors do not have any context information about the integration of the peer assessment activity in course structure, nor its pedagogical context.

    “We do not have any context information about the integration of the PA activity in course structure, nor its pedagogical context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Feedback interventions are particularly effective when they raise students' awareness of how to improve in relation to their current level of performance and the learning intentions.

    “Feedback interventions have been found to be particularly effective when they raise the students' awareness of how to improve (feed forward) in relation to their current level of performance (feed back) and the learning intentions (feed up)”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Black & Wiliam, 1998

      Not checked in the free run.

    • cites Hattie & Timperley, 2007 source

      Not checked in the free run.

    • cites Nicol & Macfarlane-Dick, 2006 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The coding scheme was validated by two researchers who coded a random sample of 10% of the whole dataset and achieved an inter-rater reliability of Cohen's kappa of at least κ = 0.8 for every code.

    “The coding scheme was validated by two researchers that coded a random sample of 10% of the whole dataset and achieved an inter-rater reliability of Cohen's kappa of at least κ = 0.8 for every code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The data a researcher eventually analyzes depends upon the business rules of the database as well as the informal rules around how individuals input and make use of data within these systems.

    “The data a researcher eventually analyzes depends upon the business rules of the database as well as the informal rules around how individuals input and make use of data within these systems”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Krumm, Means, and Bienkowski (2018)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a study of peer feedback using 1,073 feedback segments from an online peer review system (SWoRD), Nelson and Schunn (2009) found that students' comprehension of the feedback was the only significant mediator for student implementation.

    “in a study of peer feedback using 1,073 feedback segments from an online peer review system (SWoRD), Nelson and Schunn (2009) found that student's comprehension of the feedback was the only significant mediator for student implementation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nelson and Schunn (2009) source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Student's sensemaking of the feedback correlates with their experience of its usefulness is known from previous studies and relates to the problem of the feedback gap.

    “That student's sensemaking of the feedback correlates with their experience of its usefulness is known from previous studies and relates to the problem of the feedback gap (Jonsson, 2013;Nelson & Schunn, 2009).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Jonsson, 2013 source

      Not checked in the free run.

    • cites Nelson & Schunn, 2009 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Van der Pol et al. (2008) found that the more students agreed with the feedback, the more useful they would grade it.

    “Similarly, Van der Pol et al. (2008) found that the more students agreed with the feedback, the more useful they would grade it.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van der Pol et al. (2008) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The coding of the BE comments in the study is quite broad, as a result of the heterogeneous dataset and the lack of context.

    “Second, the coding of the BE comments is quite broad as a result of the heterogeneous dataset (i.e., there is a wide variety of types of feedback characteristics (length; quality, full sentences, phrases, etc.), and lack of context (e.g., domain information such as are the students writing in their mother tongue, was the feedback assignment obligatory, etc.).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Insights from LA depend significantly on the availability of data and context information, and the quality of the available data.

    “This current exploratory study shows that the insights from LA depend significantly on the availability of data and context information, and the quality of the available data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Adewoyin et al. (2016) found that longer comments do not predict higher BE ratings.

    “This finding corresponds to Adewoyin et al. (2016) who found that longer comments do not predict higher BE ratings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Adewoyin et al. (2016)

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The Peergrade dataset used in the study lacked control variables, specifically background information about the students, the context of the peer assessment activity (discipline, educational level, pedagogical approach, course structure), and the assignment mark and/or final course mark.

    “The first is lack of control variables due to weaknesses in the Peergrade dataset including 1) the absence of background information about the students, 2) the context of the PA activity, such as discipline (e.g., history or art), educational level (e.g., K-12 or college), pedagogical approach, or course structure and 3) assignment mark and/or the final course mark for the students.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The dataset did not include the original hand-in on which feedback was given, which makes it impossible to analyse whether the feedback was used to improve the students' work.

    “Third, the dataset did not include the original work-the”
    • Also stated in: § 6.1. Limitations · plain
      “This lack of essential data makes it impossible to analyse if the feedback was used to improve their work.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Students who found the feedback more useful generally experienced that the feedback made sense to them, appropriately addressed problems in their text, and were useful for improvement of their text and/or their competence as a writer.

    “This shows that students who found the feedback more useful generally experience that the feedback made sense to them, appropriately addressed problems in their text (feedback) and were useful for improvement of their text or/and their competence as a writer (feedforward).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Ashenafi (2017) found that most PA activities are non-iterative and not fully integrated into the whole educational program, which makes it hard to measure the impact of PA on long-term learning.

    “The literature review by Ashenafi (2017) found that most PA activities are non-iterative, and not fully integrated into the whole educational program, which makes it hard to measure the impact of PA on long-term learning (Ashenafi, 2017).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ashenafi, 2017 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Patchan et al. (2018) found that students who believed that their peer feedback was graded based on the perceived helpfulness by feedback receivers gave better quality feedback.

    “Patchan et al. (2018) found that students who believed that their peer feedback was graded based on the perceived helpfulness by feedback receivers, gave better quality feedback.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Patchan et al. (2018) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Studies have found that the affective features of feedback have less impact on student improvement than cognitive features.

    “That students used the improvement category kindness to a lesser extent than the improvement suggestions specificity, justification, and constructivity, resonates well with studies that have found that the affective features of feedback has less impact on student improvement than cognitive features (Hattie & Timperley, 2007;Nelson & Schunn, 2009).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Hattie & Timperley, 2007 source

      Not checked in the free run.

    • cites Nelson & Schunn, 2009 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This suggests that students did not receive sufficient preparation for the PA activity, or they did not take the task seriously.

    “This suggests that students did not receive sufficient preparation for the PA activity, or they did not take the task seriously.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This study confirms a larger question about the meaningfulness of this kind of analysis of big data without the possibility to connect this data with external context information and when data making-sense capabilities are restricted.

    “This study confirms a larger question about the meaningfulness of this kind of analysis of big data without the possibility to connect this data with external context information and when our data making-sense capabilities are restricted.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Three variables in the final model have GVIF values higher than 5 (text, GVIF = 38.389; BE_comment, GVIF = 8.496; specificity, GVIF = 15.201), indicating some multicollinearity and possible bias in the final model.

    “Three variables in the final model have GVIF values higher 5 (text, GVIF = 38.389;BE_comment, GVIF = 8.496; specificity, GVIF = 15.201), which indicates some multicollinearity and possible bias in the final model (see Table 6).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Nelson and Schunn (2009) found that students were more likely to understand feedback if it offered concrete solutions, a location of the problem(s), or a summary.

    “In particular, they found that students were more likely to understand the feedback if it offered concrete solutions, a location of the problem(s), or if the feedback included a summary.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nelson and Schunn (2009) source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Student perception of feedback is affected not only by the feedback message itself, but also by the students' ability to interpret the feedback.

    “Student perception of feedback, however, is not only affected by the feedback message itself, but also by their ability to interpret the feedback.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a review, Jonsson (2013) found that a lack of understanding of academic terminology and assessment criteria was a common problem across many studies on student perception and use of feedback.

    “In a review, Jonsson (2013) found that a lack of understanding of academic terminology and assessment criteria was a common problem across many studies on student perception and use of feedback.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jonsson (2013) source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Learning analytics has the potential to show new insights into the back-evaluation (BE) aspect of peer assessment, although many challenges remain.

    “we have shown that LA has the potential to show new insights into the BE aspect of PA, although there are many challenges as highlighted above.”
    Quote in the paperSubstantial claim
71 of 71

  • The paper relies on this without support.

    The Peergrade feedback grade scale validly measures students' perception of feedback usefulness.

    “In Peergrade the scale measures feedback usefulness”
    “F_grade is the simplest variable to indicate students' perception of the feedback that they received.”

  • The paper relies on this without support.

    The multiple-choice improvement suggestions scale validly indicates how the feedback students receive on their work could have been improved.

    “the multiple-choice improvement suggestions scale-with five suggestions-that indicates how the feedback that students receive on their work could have been improved”

  • The paper relies on this without support.

    The variables available in the Peergrade dataset are adequate proxies for the constructs used in the analysis.

    “Thus, the variables used in our analysis had to be chosen based on their availability in the dataset and their potential for use in LA methods.”

  • The paper relies on this without support.

    The feedback grade (FG1-FG5) is a valid measure of students' perception of the usefulness of feedback.

    “the ordinal dependent variable, feedback grade”

  • The paper relies on this without support.

    The list of improvement suggestions (constructivity, specificity, kindness, justification, relevance) captures the important dimensions of feedback quality.

    “Students could choose none, one, or many from five suggestions: constructivity, specificity, kindness, justification, and relevance.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Brant test is described as successful despite a significant omnibus result

    Major

    The paper says the parallel regression assumption was satisfied, but Table 7 reports an omnibus Brant-test probability of 0. A significant omnibus result indicates that the assumption is violated, calling the reported ordinal-regression model into question.

  • Number of rubric questions has opposite stated regression directions

    Minor

    The regression section says more questions predict more useful feedback, consistent with the positive coefficient in Table 5. The RQ2 results instead say more questions predict less useful feedback.

  • Boolean-question correlation directions are reversed

    Minor

    The stated directions for FG3 and FG5 disagree with the reported rho values: FG3's rho is positive, while FG5's is negative. The RQ2 results repeat the reversed verbal directions.

  • FG2 defending correlation is called positive but is negative

    Minor

    The prose assigns FG2 a positive relationship with defending comments, but its reported correlation coefficient is −0.264.

  • Five BE-comment variables are claimed, but seven are listed

    Minor

    The paper says comments were coded with one of five variables, then names seven distinct coding categories.

5 of 5

The citation lists haven’t been fetched yet

References 101 · Cited by 98

Research works printed but not linked: 1

  1. Liu, 2016, The impact of peer review on creative self-efficacy and learning performance in Web 2.0 learning activities, Journal of Educational Technology & Society, 19

Authors
Misiejuk, K., Wasson, B., & Egelandsdal, K.
Affiliations
University of Bergen (NO)
Corresponding author
Misiejuk, Kamila
ORCID
Venue
Computers in Human Behavior
Publisher
Elsevier BV
Published
2021-04
Volume
117
Pages
106658
ISSN
0747-5632 (ISSN-L)
DOI
10.1016/j.chb.2020.106658
OpenAlex
W3111691479
Semantic Scholar
9507a931705c1e283c7cc3d3927a883b6db30612
Cited by
98
References
101
Citations by year
2026: 11 · 2025: 19 · 2024: 22 · 2023: 23 · 2022: 16 · 2021: 7
FWCI
17.43
Type
journal article
Language
en
Open access
Licence
Locations
  • Computers in Human Behavior (publishedVersion, cc-by)
  • NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
Funders
Universitetet i Bergen
Topics
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › Education › Evaluation of Teaching Practices
Keywords
peer assessment, peer feedback, learning analytics, student perceptions of feedback, epistemic network analysis, feedback acceptance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review