A systematic review of peer assessment design elements
Open accessAbstract
The growing number of peer assessment studies in the last decades created diverse design options for researchers and teachers to implement peer assessment. However, it is still unknown if there are more commonly used peer assessment formats and design elements that could be considered when designing peer assessment activities in educational contexts. This systematic review aims to determine the diversity of peer assessment designs and practices in research studies. A literature search was performed in the electronic databases PsycINFO, PsycARTICLES, Web of Science Core Collection, Medline, ERIC, Academic Search Premier, and EconLit. Using data from 449 research studies (derived from 424 peer-reviewed articles), design differences were investigated for subject domains, assessment purposes, objects, outcomes, and moderators/mediators. Arts and humanities was the most frequent subject domain in the reviewed studies, and two-third of the studies had a formative purpose of assessment. The most used object of assessment was written assessment, and beliefs and perceptions were the most investigated outcomes. Gender topped the list of the investigated moderators/mediators of peer assessment. Latent class analysis of 27 peer assessment design elements revealed a five-class solution reflecting latent patterns that best describe the variability in peer assessment designs (i.e. prototypical peer assessment designs). Only ten design elements significantly contributed to these patterns with an associated effect size R 2 ranging from .204 to .880, indicating that peer assessment designs in research studies are not as diverse as they theoretically can be.
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic review of 449 studies from 424 articles found that peer assessment designs in research are less diverse than theoretically assumed, with five latent prototypical design patterns. Instructional design elements drove the patterns, while context, outcomes, and moderators/mediators contributed little.
Key findings
- Across 449 studies, arts and humanities was the most frequent subject domain (n = 118; 25.71%), followed by natural sciences, mathematics and statistics (n = 87; 18.95%), education (n = 81; 17.65%), and health and welfare (n = 54; 11.76%). Formative purpose was most common (n = 257; 59.08%), then summative (n = 157; 36.09%), and both (n = 21; 4.83%).
- Written assignment was the most used object of assessment (45.76%). Most studied outcomes were beliefs and perceptions (38.46%), performance and skills (21.54%), and peer feedback content (17.62%); reliability (7.69%), validity (11.47%), and peer feedback processing (3.22%) were less studied. Moderators/mediators were most often ability and skills (34.31%) and gender (27.45%).
- Latent class analysis of 27 peer assessment design elements yielded a five-class solution (BIC = 31,462.1; CAIC = 31,864.1; classification error = .036; entropy R² = .938; standard R² = .930). Ten design elements had R² > .20, ranging from .204 to .880; the largest were unit of assessment assessee (R² = .880), group constellation (R² = .681), revision (R² = .476), unit of assessment assessor (R² = .444), and purpose (R² = .370).
- Study designs were mostly pre-experimental (75.28%), with quasi-experimental (16.00%) and experimental (8.24%) much less frequent. Higher education was overrepresented: higher education 73.72%, postgraduate 7.35%, and higher plus postgraduate 2.45%.
Methods
- Systematic review with database searches in March 2015 and an update in October 2018 across PsycINFO, PsycARTICLES, Web of Science Core Collection, Medline, ERIC, Academic Search Premier, and EconLit. Included peer-reviewed English empirical articles from 2011 to October 2018, yielding 424 articles and 449 studies.
- Coding scheme based on Topping (1998) and later refinements (e.g., Gielen et al., 2011), grouped into context, instructional design, outcomes, and moderators/mediators. Inter-rater reliability was assessed with percentage agreement and Krippendorff’s alpha.
- Descriptives and frequency tables addressed RQ1; latent class analysis (Latent GOLD 5.0) addressed RQ2 using 891 cases, estimating one- to eight-class models and evaluating fit with BIC, CAIC, classification error, Wald tests, and R² effect sizes; only design elements with R² > .20 were interpreted.
Limitations
- Stated: the ISCED-F 2013 classification may be too elaborate; inter-rater reliability fell below 70% for three of 28 categories (relation to staff assessment, individual constellation assessor, and individual constellation assessee); many studies were vague, unclear, or inaccurately reported design elements; the five classes may be simpler than real peer assessment practice; and primary, secondary, and vocational subsamples were too small for separate LCA.
- Stated: the R² > .20 cutoff is acknowledged as arbitrary.
- Evident from the text: title and abstract screening was performed by only one reviewer (the first author), with the second author consulted only for unclear full-text records; the search ends in October 2018, so later literature is not covered.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 28 cited sources checked · 0 supported · 0 flagged
- Findingown data
Two-thirds of the reviewed studies had a formative purpose of assessment.
“two-third of the studies had a formative purpose of assessment”
Quote in the paperNot a substantial claim - Findingown data
An inflation of Krippendorff's alpha was observed in the data where alpha = 1 when percentage agreement was merely 80%.
“Furthermore, an inflation of Krippendorff's α was also observed in our data where α = 1 when the percentage agreement was merely 80%.”
Quote in the paperSubstantial claim - Findingown data
Arts and humanities was the most frequent subject domain in the reviewed studies.
“Arts and humanities was the most frequent subject domain in the reviewed studies”
Quote in the paperNot a substantial claim - Findingown data
The associated effect size R2 for the ten design elements ranged from .204 to .880.
“with an associated effect size R 2 ranging from .204 to .880”
Quote in the paperSubstantial claim - Findingown data
Latent class analysis of 27 peer assessment design elements revealed a five-class solution reflecting latent patterns that best describe the variability in peer assessment designs.
“Latent class analysis of 27 peer assessment design elements revealed a five-class solution reflecting latent patterns that best describe the variability in peer assessment designs (i.e.prototypical peer assessment designs).”
Quote in the paperSubstantial claim
- Findingown data
Beliefs and perceptions were the most investigated outcomes.
“beliefs and perceptions were the most investigated outcomes”
Quote in the paperNot a substantial claim - Findingown data
Gender topped the list of the investigated moderators/mediators of peer assessment.
“Gender topped the list of the investigated moderators/mediators of peer assessment.”
Quote in the paperNot a substantial claim - Findingown data
The most used object of assessment was written assessment.
“The most used object of assessment was written assessment”
Quote in the paperNot a substantial claim - Findingown data
Only ten design elements significantly contributed to these patterns.
“Only ten design elements significantly contributed to these patterns”
Quote in the paperNot a substantial claim - Findingcitation
Due to vagueness of peer assessment design descriptions in most published articles, reaching above 80% agreement for all design elements was not feasible.
“Due to the vagueness of the description of peer assessment designs in most of the published articles-an issue that was also previously highlighted by Topping (1998)-reaching an agreement of above 80% for all the peer assessment design elements was not feasible.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Topping (1998)
Not checked in the free run.
- Findingown data
In most studies, the purpose of assessment was formative (n=257; 59.08%).
“In most studies, the purpose of assessment was formative (n = 257; 59.08%).”
Quote in the paperNot a substantial claim - Findingown data
Many studies were unclear on their study details or inaccurately reported them.
“Many studies were also unclear on their study details or inaccurately reported them (e.g.experimental vs. quasi-experimental design).”
Quote in the paperSubstantial claim - Findingown data
In most subject domains, formative assessment was most often used.
“In most subject domains, formative assessment was most often used (formative assessment was also the most abundant category).”
Quote in the paperNot a substantial claim - Findingown data
Moderators and mediators most commonly studied in relation to peer assessment designs were gender (n=28; 27.45%) and ability and skills (n=35; 34.31%).
“Moderators and mediators that were most commonly studied in relation to peer assessment designs were gender (n = 28; 27.45%) and ability and skills (n = 35; 34.31%).”
Quote in the paperNot a substantial claim - Findingreasoning
Most peer assessment design elements were statistically significant at the 0.001 level.
“Most peer assessment design elements were statistically significant at the 0.001 level.”
Quote in the paperNot a substantial claim - Findingown data
The classification error of the 5-class model (0.036) was very similar to the classification errors of the 3-class and 4-class models.
“Furthermore, the classification error of the 5-class model (0.036) was very similar to the classification errors of the 3-class and 4-class models.”
Quote in the paperNot a substantial claim - Findingown data
Almost half of the studies used a written assignment as the object of assessment (45.76%).
“Almost half of the studies used a written assignment (45.76%).”
Quote in the paperNot a substantial claim - Findingown data
The least used object of assessment was test performance (0.89%).
“The least used object of assessment was test performance (0.89%).”
Quote in the paperNot a substantial claim - Findingown data
All other categories of moderators/mediators occurred only in a relatively small number of studies.
“All other categories occurred only in a relatively small number of studies (see Table 8).”
Quote in the paperNot a substantial claim - Findingnone given
Studies in classes 1, 2, and 3 were very likely to use unit of assessment as individual for both assessor and assessee.
“Studies in classes 1, 2, and 3 were very likely to use unit of assessment as individual for both assessor and assessee.”
Quote in the paperSubstantial claim - Findingown data
Few studies included 'psychological factors' despite the attention over the past decade for the role of social and interpersonal factors as relevant moderators and mediators of both peer assessment processes and outcomes.
“it is striking that few studies included 'psychological factors' despite the attention over the past decade for the role of social and interpersonal factors (e.g.psychological safety, trust in self and other as assessor, fairness, friendship, interdependence) as relevant moderators and mediators of both peer assessment processes and outcomes”
Quote in the paperSubstantial claim - Findingown data
Matching based on 'ability and skills' is not dominant among any of the identified classes.
“Furthermore, matching based on 'ability and skills' is not dominant among any of the identified classes which is quite surprising given that ability and year--as an indicator of skill and experience--are the main elements of Topping's (1998) typology that he stated students should be matched on.”
Quote in the paperSubstantial claim - Findingown data
Few of the included studies were in the domains of 'information and communication technologies', 'social sciences, journalism, and information', 'business, administration, and law', and 'engineering, manufacturing, and construction'.
“whereas few of the included studies were in the domains of 'information and communication technologies', 'social sciences, journalism, and information', 'business, administration, and law', and 'engineering, manufacturing, and construction'.”
Quote in the paperNot a substantial claim - Findingown data
Major contributors to class 4 were the subject domains 'education' and 'natural sciences, mathematics, and statistics'.
“Major contributors to class 4 were the subject domains 'education' and 'natural sciences, mathematics, and statistics'.”
Quote in the paperNot a substantial claim - Findingown data
Studies from 'education', 'arts and humanities' and 'natural sciences, mathematics, and statistics' were major contributors to the compositions of class 1.
“Studies from 'education', 'arts and humanities' and 'natural sciences, mathematics, and statistics' were major contributors to the compositions of class 1.”
Quote in the paperNot a substantial claim - Findingown data
Class 5 contained studies from 'education' and 'natural sciences, mathematics, and statistics', as well as 'health and welfare'.
“Finally, class 5 contained studies from 'education' and 'natural sciences, mathematics, and statistics', as well as 'health and welfare'.”
Quote in the paperNot a substantial claim - Findingown data
The ten peer assessment design elements with the highest effect sizes were all instructional design variables.
“The ten peer assessment design elements with the highest effect sizes were all instructional design variables.”
Quote in the paperSubstantial claim - Findingown data
Class 2 was a quite diverse class, as it contained studies from most subject domains.
“Class 2 was a quite diverse class, as it contained studies from most subject domains.”
Quote in the paperNot a substantial claim - Findingown data
The two subject domains 'education' and 'social sciences, journalism, and information' showed a comparatively even contribution to each of the five classes.
“The two subject domains 'education' and 'social sciences, journalism, and information' showed a comparatively even contribution to each of the five classes.”
Quote in the paperNot a substantial claim - Findingown data
Among the factors that are considered as potential moderators and mediators of both peer assessment processes and outcomes, nearly two-thirds concern 'gender' and 'ability and skills'.
“Finally, we found that among the factors that are considered as potential moderators and mediators of both peer assessment processes and outcomes, nearly twothirds concern 'gender' and 'ability and skills' (Table 8).”
Quote in the paperNot a substantial claim - Findingown data
Both reliability and validity were less often treated as an outcome measure compared to the impressions that many review studies provide.
“Regarding peer assessment outcomes, our results showed that both reliability and validity were less often treated as an outcome measure compared to the impressions that many review studies provide”
Quote in the paperSubstantial claim - Findingown data
Class 3 was dominated by studies from 'Arts and humanities'.
“Class 3 was dominated by studies from 'Arts and humanities'.”
Quote in the paperNot a substantial claim - Findingown data
Studies in class 5 were likely to use unit of assessment as both (group and individual) for both assessee and assessor.
“whereas studies in class 5 were likely to use Unit of assessment as both (group and individual) for both assessee and assessor.”
Quote in the paperNot a substantial claim - Findingown data
The interpretations of classes 1, 2, 3, and 5 of the 5-class model, in terms of the design elements, were analogous to the interpretations of the classes of the 4-class model.
“Furthermore, the interpretations of classes 1, 2, and 3 and 5 of the 5-class model, in terms of the design elements, were analogous to the interpretations of the classes of the 4-class model.”
Quote in the paperNot a substantial claim - Findingown data
Close to three-quarters of the included studies were conducted in four subject domains: 'education', 'natural sciences, mathematics, and statistics', 'arts and humanities', and 'health and welfare'.
“We found that close to three-quarters of the included studies were conducted in four subject domains: 'education', 'natural sciences, mathematics, and statistics', 'arts and humanities', and 'health and welfare';”
Quote in the paperNot a substantial claim - Findingown data
The ten most important peer assessment design elements in Table 10 are also the ten most important contributors of the additional LCA using instructional design variables only.
“The ten most important peer assessment design elements in Table 10 are also the ten most important contributors of this additional LCA.”
Quote in the paperNot a substantial claim - Findingown data
The first LCA revealed that variables that belong to the context and the categories outcome variables and moderators/mediators had no or a small contribution when discriminating peer assessment designs, whereas the instructional design elements played a major role.
“The first LCA revealed that variables that belong to the context and the categories outcome variables and moderators/mediators had no or a small contribution when discriminating peer assessment designs, whereas the instructional design elements played a major role.”
Quote in the paperSubstantial claim - Findingown data
Across all clusters, the combined use of scoring and comments is comparatively less important.
“Across all clusters, the combined use of scoring and comments is comparatively less important, which is striking given that Ashenafi (2017) reported it to be among the most common practices of peer assessment in higher education.”
Quote in the paperSubstantial claim - Findingnone given
The interpretations of classes 1, 2, and 3 of the 5-class model, in terms of the design elements, were analogous to the interpretations of the classes of the 3-class model.
“the interpretations of classes 1, 2, and 3 of the 5-class model, in terms of the design elements, were analogous to the interpretations of the classes of the 3-class model.”
Quote in the paperNot a substantial claim - Findingown data
The second LCA on the instructional design elements only revealed that the ten most important design elements in the first LCA were also the ten most important design elements in the second LCA.
“We then performed a second LCA on the instructional design elements only, which revealed that the ten most important design elements in the first LCA were also the ten most important design elements in the second LCA.”
Quote in the paperNot a substantial claim - Findingown data
Studies in class 4 were likely to use unit of assessment as individual for assessor but as group for assessee.
“In contrast, studies in class 4 were likely to use unit of assessment as individual for assessor but as group for assessee”
Quote in the paperSubstantial claim - Findingcitation
The overrepresentation of higher education is in line with several recent reviews that included multiple levels of education.
“and this overrepresentation is in line with several recent reviews that included multiple levels of education, i.e. 94% of all included studies by Li et al. (2016) were in higher education, 74% in Panadero and Alqassab (2019), and 54% in Double et al. (2020).”
Quote in the paperNot a substantial claim0 of 3 sources checked - Findingown data
In most studies, the purpose of peer assessment was formative.
“Our review revealed that (a) the subject domain that most frequently implemented peer assessment was arts and humanities, (b) in most studies, the purpose of assessment was formative, and (c) almost half of the studies used a written assignment as an object of peer assessment.”
Quote in the paperSubstantial claim - Findingown data
Almost half of the studies used a written assignment as an object of peer assessment.
“Our review revealed that (a) the subject domain that most frequently implemented peer assessment was arts and humanities, (b) in most studies, the purpose of assessment was formative, and (c) almost half of the studies used a written assignment as an object of peer assessment.”
- Also stated in: § Conclusion · plain
“Written assignment | 205 | 45.76”
Quote in the paperSubstantial claim - Also stated in: § Conclusion · plain
- Findingown data
Instructional design elements were the most important contributors to the five latent design patterns.
“We found five latent and prototypical peer assessment design patterns to which the instructional design elements were the most important contributors”
Quote in the paperSubstantial claim - Findingown data
Our results confirm an overrepresentation of research in higher education.
“our results confirm an overrepresentation of research in higher education.”
Quote in the paperSubstantial claim - Findingcitation
The most prominent design elements belong to two core themes of peer assessment design: decisions concerning the implementation of peer assessment and assessment groups and peer interactions within them.
“We showed that the most prominent design elements belong to two core themes of peer assessment design: decisions concerning the implementation of peer assessment and assessment groups and peer interactions within them (Gielen et al., 2011).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Gielen et al., 2011 source
Not checked in the free run.
- Findingown data
Variables belonging to context, outcomes, and moderators/mediators did not contribute to the five latent design patterns.
“whereas variables that belong to context, the outcomes variables, and moderators/mediators did not contribute.”
Quote in the paperSubstantial claim - Findingown data
Five latent and prototypical peer assessment design patterns were found.
“We found five latent and prototypical peer assessment design patterns to which the instructional design elements were the most important contributors”
Quote in the paperSubstantial claim - Findingown data
The subject domain that most frequently implemented peer assessment was arts and humanities.
“Our review revealed that (a) the subject domain that most frequently implemented peer assessment was arts and humanities”
Quote in the paperSubstantial claim - Findingown data
Almost 84% of the studies included were conducted in higher education (including postgraduate education).
“Furthermore, almost 84% of the studies we included were conducted in higher education (including postgraduate education)”
Quote in the paperNot a substantial claim - Findingown data
The information on study details and peer assessment design elements in many studies is either too limited or needs to be inferred.
“We are, regrettably, not alone in observing that the information on study details and peer assessment design elements in many studies is either too limited or needs to be inferred”
Quote in the paperSubstantial claim - Findingown data
The most commonly studied outcome of peer assessment was students' beliefs and perceptions.
“Further, the most commonly studied outcome of peer assessment was students' beliefs and perceptions”
- Also stated in: § Conclusion · plain
“Beliefs and perceptions | 275 | 38.46”
Quote in the paperSubstantial claim - Also stated in: § Conclusion · plain
- Interpretationown data
Peer assessment designs in research studies are not as diverse as they theoretically can be.
“indicating that peer assessment designs in research studies are not as diverse as they theoretically can be.”
Quote in the paperSubstantial claim - Premisecitation
Topping's consecutive refinements expanded the list of elements (55 in 2013, 43 in 2017, 44 in 2021a, 79 in 2021b).
“First, the list of elements was ever expanded (55 important factors in 2013, 43 variations in 2017, 44 variations in 2021a, and 79 pointers in 2021b).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Topping, 2013
Not checked in the free run.
- cites Topping, 2017 source
Not checked in the free run.
- cites Topping, 2021a
Not checked in the free run.
- Interpretationreasoning
Some design elements are more likely to co-occur in peer assessment designs.
“Some design elements are, therefore, more likely to co-occur in peer assessment designs.”
Quote in the paperSubstantial claim- Rests on an assumption: Peer assessment involving revisions has a formative purpose.
- Premisecitation
Lower values of BIC, CAIC, and classification error indicate a better model fit.
“Lower values of BIC, CAIC, and classification error indicate a better model fit (Aho et al., 2014).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Aho et al., 2014 source
Not checked in the free run.
- Study design and conductown data
Coding-pair 1 reached the minimum threshold of agreement on 100% of the study details categories.
“As shown in Table 3, coding-pair 1 reached the minimum threshold of agreement on 100% of the study details categories”
Quote in the paperSubstantial claim - Interpretationown data
Most peer assessment designs do not necessarily focus on outcomes in terms of reliability and validity.
“Thus, most peer assessment designs do not necessarily focus on outcomes in terms of reliability and validity.”
Quote in the paperSubstantial claim - Study design and conductown data
For coding-pair 2, agreement on study details was reached for all categories.
“For coding-pair 2, the agreement on study details was also reached for all the categories.”
Quote in the paperNot a substantial claim - Premisenone given
All four refinements of Topping's typology gradually drifted away from enabling comparison of studies on peer assessment in general and constitutive design elements in particular.
“However, all four refinements of Topping's (1998) typology gradually drifted away from enabling precisely such a comparison of studies on peer assessment in general, as well as the constitutive design elements in particular.”
Quote in the paperSubstantial claim - Study design and conductown data
For peer assessment design elements, an acceptable level of agreement was reached for 26 out of 28 elements by coding-pair 1.
“For the peer assessment design elements, an acceptable level of agreement was reached for 26 out of the 28 elements.”
Quote in the paperSubstantial claim - Premisecitation
According to Cohen (1988), R2 values of 0.13 and 0.26 correspond to medium and large effect sizes, respectively.
“According to Cohen (1988, p. 413-414), R 2 values of 0.13 and 0.26 correspond to medium and large effect sizes, respectively.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Cohen (1988, p. 413-414)
Not checked in the free run.
- Interpretationown data
The statistical significance of most peer assessment design elements is in part due to the large sample size (N=891).
“This is in part due to the large sample size (N = 891).”
Quote in the paperNot a substantial claim - Study design and conductown data
The only two elements below the 70% threshold for coding-pair 1 were place/time and degree of interactivity.
“The only two elements that were below the 70% threshold were place/time and degree of interactivity.”
Quote in the paperNot a substantial claim - Interpretationown data
The 5-class model has the optimal number of classes according to the BIC and CAIC statistics.
“Thus, the 5-class model has the optimal number of classes according to the BIC and CAIC statistics.”
Quote in the paperSubstantial claim - Study design and conductown data
Agreements on peer assessment instructional design were reached for 25 out of 28 categories by coding-pair 2.
“Agreements on the peer assessment instructional design were reached for 25 out of the 28 categories.”
Quote in the paperSubstantial claim - Interpretationreasoning
Context variables, outcome variables, and moderators and mediators were less important for discriminating between studies than the instructional design elements.
“It appears that the context variables, outcome variables, and moderators and mediators were less important for discriminating between studies than the instructional design elements.”
Quote in the paperSubstantial claim - Interpretationown data
This finding clearly contrasts the claim by Ashenafi (2017) that 'Gender effects are the least studied factors in peer assessment in higher education'.
“This clearly contrasts the claim by Ashenafi (2017) that 'Gender effects are the least studied factors in peer assessment in higher education… ' (p. 232).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Ashenafi (2017) source
Not checked in the free run.
- Interpretationown data
Design elements for which inter-rater reliability was difficult to assess--due to extreme unbalanced marginal distributions--were identified by the LCA as less relevant.
“Furthermore, note that the design elements for which inter-rater reliability was difficult to assess--due to extreme unbalanced marginal distributions--were identified by the LCA as less relevant.”
Quote in the paperNot a substantial claim - Interpretationown data
There was no evidence that alignment, rewards, and setting discriminate between the five classes, given the other design elements in the model.
“There was no evidence that these elements discriminate between the five classes, given the other design elements in the model.”
Quote in the paperSubstantial claim - Premisecitation
Among recent meta-analyses, only Li et al. (2016) explicitly investigated the object of assessment and concluded that assessment tasks in science/engineering might be more clear-cut and thus it might be easier for students to assess their peers.
“Furthermore, among recent meta-analyses, only Li et al. (2016) explicitly investigated the object of assessment (referred to as 'task being rated') and concluded that assessment tasks in science/ engineering might be more clear-cut and thus it might be easier for students to assess their peers.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Li et al. (2016)
Not checked in the free run.
- Interpretationreasoning
Peer assessment designs in research studies are clearly not as diverse as they theoretically can be or as diverse as assumed and implied by Topping's initial typology and subsequent refinements.
“Hence, peer assessment designs in research studies are clearly not as diverse as they theoretically can be or as diverse as assumed and implied by Topping's initial typology and subsequent refinements”
Quote in the paperSubstantial claim - Interpretationown data
The ten most prominent design elements that contributed to the five-class solution appear to belong to two core themes of peer assessment design.
“Importantly, the ten most prominent design elements that contributed to the five-class solution appear to belong to two core themes of peer assessment design (see Table 11).”
Quote in the paperSubstantial claim - Interpretationown data
The number of studies (18% and 3%, respectively) that included peer feedback content and peer feedback processing as outcome measures can be deemed sizeable.
“the number of studies (18% and 3%, respectively) that included them as outcome measures can be deemed sizeable”
Quote in the paperNot a substantial claim - Interpretationreasoning
Latent patterns of peer assessment designs at primary, secondary, or vocational education can be expected to look different from those in higher education.
“Latent patterns of peer assessment designs at these levels of education can, thus, be expected to look different-especially as the majority of studies we included were conducted in higher education (84%).”
Quote in the paperNot a substantial claim - Interpretationreasoning
Peer assessment designs in primary, secondary, or vocational education are likely to differ from designs implemented in higher education.
“Additionally, peer assessment designs in primary, secondary, or vocational education are likely to differ from designs implemented in higher education, for example, in terms of the composition of assessment constellations, complexity of the learning tasks, and degree to which peer assessment will be taking place online.”
Quote in the paperSubstantial claim - Premisecitation
Some decisions regarding the use of certain peer assessment design elements appear to be theoretically motivated rather than empirically supported.
“This is especially important given that, currently, some decisions regarding the use of certain peer assessment design elements appear to be theoretically motivated rather than empirically supported (Double et al., 2020).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Double et al., 2020 source
Not checked in the free run.
- Premisecitation
The complexity of peer assessment makes it difficult to disentangle the effects of different design elements without conducting controlled experiments.
“The complexity of peer assessment makes it difficult to disentangle the effects of different design elements without conducting controlled experiments (Double et al., 2020;Panadero & Alqassab, 2019)”
Quote in the paperSubstantial claim0 of 2 sources checked
- Classified as probably not a substantial claim. 18
- May combine two claims. 45
- The quote only partly states this claim. 2
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataAfter deduplication, 1089 records from March 2015 and 546 from October 2018 were screened.
“This process rendered a total of 1089 in March 2015 and 546 in October 2018 that were screened against the inclusion and exclusion criteria.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataThe literature search identified 1535 citations in March 2015 and 930 in October 2018.
“A total of 1535 citations were identified from the seven searched databases in the first database search (in March 2015) and a total of 930 in the updated database search (in October 2018; see Fig. 1).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe review included 424 articles and 449 studies.
“This process resulted in a total of 424 articles and 449 studies”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataA total of 907 articles were identified as eligible.
“A total of 907 articles have been identified as eligible”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataArts and humanities occurred most frequently among subject domains (n=118; 25.71%), followed by natural sciences, mathematics and statistics (n=87; 18.95%), education (n=81; 17.65%), and health and welfare (n=54; 11.76%).
“Arts and humanities occurred most frequently (n = 118; 25.71%), followed by natural sciences, mathematics and statistics (n = 87; 18.95%), education (n = 81; 17.65%), and health and welfare (n = 54; 11.76%).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataReliability of peer assessment (n=55; 7.69%), validity of peer assessment (n=82; 11.47%), and processing of peer feedback (n=23; 3.22%) were studied less often.
“Reliability of peer assessment (n = 55; 7.69%), validity of peer assessment (n = 82; 11.47%), and processing of peer feedback (n = 23; 3.22%) were studied less often.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOnly a few studies researched services (n=3; 0.65%) and generic programmes and qualifications (n=3; 0.65%), and no study covered agriculture, forestry, fisheries, and veterinary.
“Furthermore, only a few studies researched services (n = 3; 0.65%) and generic programmes and qualifications (n = 3; 0.65%), and no study covered agriculture, forestry, fisheries, and veterinary.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn a substantial number of studies, the purpose of assessment was summative (n=157; 36.09%), and only in a small number of studies, the purpose was both summative and formative (n=21; 4.83).
“In a substantial number of studies, the purpose of assessment was summative (n = 157; 36.09%), and only in a small number of studies, the purpose was both summative and formative (n = 21; 4.83).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe sample size for the LCA consisted of 891 cases.
“The sample size for the LCA consisted of 891 cases.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataOutcome variables most commonly studied were beliefs and perceptions (n=275; 38.46%), performance and skills (n=154; 21.54%), and content of peer feedback (n=126; 17.62%).
“Outcome variables that were most commonly studied were beliefs and perceptions (n = 275; 38.46%), performance and skills (n = 154; 21.54%), and content of peer feedback (n = 126; 17.62%).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataBoth the entropy R2 (0.938) and the standard R2 (0.930) of the 5-class model were quite high.
“Both the entropy R 2 (0.938) and the standard R 2 (0.930) of the 5-class model were quite high.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe 5-class model had both the lowest BIC value (31,462.1) and the lowest CAIC value (31,864.1).
“The 5-class model had both the lowest BIC value (31,462.1)and the lowest CAIC value (31,864.1).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataSummative assessment was used relatively often in peer assessment studies in the subject domains of 'information and communication technologies' (54.29%), 'engineering, manufacturing, and construction' (52.63%), and 'health and welfare' (54.72%).
“In contrast, Table 5 shows that summative assessment was used relatively often in peer assessment studies in the subject domains of 'information and communication technologies' (54.29%), 'engineering, manufacturing, and construction' (52.63%), and 'health and welfare' (54.72%).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEach of the ten design elements with the highest effect sizes had an associated effect size R2 that exceeds 0.20 and discriminated well across the five classes.
“Each of these design elements had an associated effect size R 2 that exceeds 0.20 and discriminated well across the five classes.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe ten design elements with the highest effect sizes are unit of assessment assessee (R2=0.880), group constellation (R2=0.681), revision (R2=0.476), unit of assessment assessor (R2=0.444), purpose (R2=0.370), scope of involvement (R2=0.363), contact (R2=0.315), product/output of peer assessment (R2=0.249), matching (R2=0.217), and privacy (R2=0.204).
“The ten design elements are unit of assessment assessee (R 2 = 0.880), group constellation (R 2 = 0.681), revision (R 2 = 0.476), unit of assessment assessor (R 2 = 0.444), purpose (R 2 = 0.370), scope of involvement (R 2 = 0.363), contact (R 2 = 0.315), product/output of peer assessment (R 2 = 0.249), matching (R 2 = 0.217), and privacy (R 2 = 0.204).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEven beliefs and perceptions of peer assessment were treated more often as an outcome variable compared to performance and skills-again, more so than one would expect from existing review studies.
“Furthermore, even beliefs and perceptions of peer assessment were treated more often as an outcome variable compared to performance and skills-again, more so than one would expect from existing review studies”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataClass 5 is described as online asynchronous summative non-anonymous domain-matched bidirectional intra-and intergroup (individual and group) peer assessment via scoring [without revision].
“• Class 5: Online asynchronous summative non-anonymous domain-matchedfoot_0 bidirectional intra-and intergroup (individual and group) peer assessment via scoring [without revision]”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe three peer assessment design elements that were not statistically significant at the 0.05 level and had very small effect sizes are alignment (p=1.000; R2=0.003), rewards (p=0.130; R2=0.007), and setting (p=0.210; R2=0.027).
“The three peer assessment design elements that were not statistically significant at the 0.05 level and had very small effect sizes are alignment (p = 1.000;R 2 = 0.003), rewards (p = 0.130; R 2 = 0.007), and setting (p = 0.210; R 2 = 0.027).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataStudies in classes 1, 2, and 4 were more likely to use anonymous peer assessment, whereas studies in classes 3 and 5 were more likely to use public peer assessment.
“Furthermore, studies in classes 1, 2, and 4 were more likely to use anonymous peer assessment, whereas studies in classes 3 and 5 were more likely to use public peer assessment.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataClasses 1, 2, and 3 were relatively large classes that contained 32.8%, 25.7%, and 21.6% of the sample, respectively.
“Classes 1, 2, and 3 were relatively large classes that contained 32.8%, 25.7%, and 21.6% of the sample, respectively.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenStudies in classes 1, 3, and 4 were quite likely to involve students in revising their work after peer assessment (i.e. revision=yes) with giving and receiving peer feedback as scope of involvement, whereas studies in classes 2 and 5 were quite likely to have no revision after peer assessment with giving and receiving peer assessment as scope of involvement.
“Finally, studies in classes 1, 3, and 4 were quite likely to involve students in revising their work after peer assessment (i.e.revision = yes) with giving and receiving peer feedback as scope of involvement, whereas studies in classes 2 and 5 were quite likely to have no revision after peer assessment with giving and receiving peer assessment as scope of involvement.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe purpose of peer assessment in studies in classes 1, 3, and 4 was more likely to be formative, whereas the purpose of peer assessment was more likely to be summative in studies in classes 2 and 5.
“The purpose of peer assessment in studies in classes 1, 3, and 4 was more likely to be formative, whereas the purpose of peer assessment was more likely to be summative in studies in classes 2 and 5.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataClose to two-third of all studies had a formative purpose, a little over one-third had a summative purpose, and close to 5% had both a summative and formative purpose.
“With respect to the assessment purpose, we found that close to two-third of all studies had a formative purpose, a little over one-third had a summative purpose, and close to 5% had both a summative and formative purpose.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe matching in peer assessment was more likely to be random in studies in classes 1, 2, and 4, self-select in studies in class 3, and based on domain in studies in class 5.
“The matching in peer assessment was more likely to be random in studies in classes 1, 2, and 4, self-select in studies in class 3, and based on domain (e.g.subject, specialty, academic major, degree) in studies in class 5.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenClass 2 is described as online asynchronous summative anonymous random-matched bi-directional intragroup (individually assessed) peer assessment via scoring [without revision].
“• Class 2: Online asynchronous summative anonymous random-matched bi-directional intragroup (individually assessed) peer assessment via scoring [without revision]”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataClose to half of the studies used a written assignment as the object of assessment and test performance was the least used object of assessment.
“Close to half of the studies used a written assignment as the object of assessment and test performance was the least used object of assessment.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataClasses 4 and 5 were relatively small classes that contained, respectively, 13.1% and 6.8% of the sample.
“Classes 4 and 5 were relatively small classes that contained, respectively, 13.1% and 6.8% of the sample.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWhen split by subject domains, a formative purpose was most often used in the majority of the domains, except in the domains 'information and communication technologies', 'engineering, manufacturing, and construction', and 'health and welfare' where a summative purpose was used more often compared to a formative purpose.
“When split by subject domains, cross-classification revealed that a formative purpose was most often used in the majority of the domains, except in the domains 'information and communication technologies', 'engineering, manufacturing, and construction', and 'health and welfare' where a summative purpose was used more often compared to a formative purpose.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataClass 1 is described as online asynchronous formative anonymous random-matched bi-directional intragroup (individually assessed) peer assessment via written comments [with revision].
“• Class 1: Online asynchronous formative anonymous random-matched bi-directional intragroup (individually assessed) peer assessment via written comments [with revision]”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe output of peer assessment was more likely to be written comments in studies in class 1, scoring in studies in classes 2 and 5, written and oral comments in studies in class 3, and scoring and written comments in studies in class 4.
“The output of peer assessment was more likely to be written comments in studies in class 1, scoring in studies in classes 2 and 5, written and oral comments in studies in class 3, and scoring and written comments in studies in class 4.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenClass 3 is described as face-to-face synchronous formative non-anonymous self-selection matched bi-directional intragroup (individually assessed) peer assessment via oral and written comments [with revision].
“• Class 3: Face-to-face synchronous formative non-anonymous self-selection matched bi-directional intragroup (individually assessed) peer assessment via oral and written comments [with revision]”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWhen cross classified the clusters with the subject domains, 'arts and humanities' was most prominent in classes 1 and 3 and 'natural sciences, mathematics, and statistics' in classes 2, 4, and 5.
“When we cross classified the clusters with the subject domains, we found that 'arts and humanities' was most prominent in classes 1 and 3 and 'natural sciences, mathematics, and statistics' in classes 2, 4, and 5.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn our study, written comments contributed to the largest identified class and thus was more common in peer assessment research.
“In our study, written comments contributed to the largest identified class and thus was more common in peer assessment research (see Table 11).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenClass 4 is described as online asynchronous formative anonymous random-matched bi-directional intergroup (individuals assessing groups) peer assessment via scoring and written comments [with revision].
“• Class 4: Online asynchronous formative anonymous random-matched bi-directional intergroup (individuals assessing groups) peer assessment via scoring and written comments [with revision]”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone givenThe contact between assessors and assessees in studies in classes 1, 2, 4, and 5 was likely to be online asynchronous, whereas in studies in class 3 was likely to be face-to-face synchronous.
“Moreover, the contact between assessors and assessees in studies in classes 1, 2, 4, and 5 was likely to be online asynchronous, whereas in studies in class 3 was likely to be face-to-face synchronous.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataBoth quasi-experimental and experimental designs, combined, only accounted for 24% of the included studies, whereas about 76% was coded as using pre-experimental research designs.
“both types, combined, only accounted for 24% of the included studies, whereas about 76% was coded as using pre-experimental research designs (i.e.studies conducted without any experimental manipulations).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataPeer assessment designs in research studies are not as diverse as they theoretically can be and are often assumed or implied to be.
“Overall, our findings have shown that peer assessment designs in research studies are not as diverse as they theoretically can be and are often assumed or implied to be.”
- Also stated in: § Conclusion · plain
“Although the diversity that we found is narrower than assumed in the literature”
Quote in the paperSubstantial claim- Also stated in: § Conclusion · plain
The quote only partly states this claim.
Findingown dataGender was the most investigated moderator/mediator of peer assessment.
“and 'ability and skills' and gender were the most investigated moderators/mediators of peer assessment.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataAbility and skills was the most investigated moderator/mediator of peer assessment.
“and 'ability and skills' and gender were the most investigated moderators/mediators of peer assessment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataTwo pairs of coders independently coded 10-14% of the total number of articles over several rounds.
“Over several rounds, two pairs of coders independently coded 10-14% of the total number of the articles.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataKrippendorff's alpha relies on list-wise deletion of cases with a missing value by one rater, which explains the observed inflation.
“An explanation to this observation is that Krippendorff's α relies on list-wise deletion of cases with a missing value by one rater (in the case of two raters).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataIn each coding round, coders first coded independently, and disagreements were identified and discussed until resolved.
“In each round, the coders first coded the articles independently, and subsequently disagreements were identified and discussed until resolved.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenThe coding scheme was used to extract data from the 424 articles after establishing sufficient inter-rater reliability.
“The coding scheme was used to extract the relevant data from the 424 articles included in this review after establishing a sufficient inter-rater reliability.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningMissing data were not a problem for the maximum likelihood estimation procedures in Latent GOLD because the procedures used all information available for each case.
“Missing data were not a problem for the maximum likelihood estimation procedures in Latent GOLD: the procedures used all information available for each case.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationTopping avoids dealing with clustering and is inconsistent in categories (six sections in 2013; seven categories in 2021b).
“Second, Topping not only avoids dealing with the issue of clustering (e.g.'The difficulty is that different researchers propose different clusters, so I have left the list un-clustered.',Topping, 2017, p. 6;Topping, 2021a), but is also inconsistent (six sections in 2013; seven categories in the 2021b).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Topping, 2017 source
Not checked in the free run.
- cites Topping, 2021a
Not checked in the free run.
- cites Topping, 2021b
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisereasoningSome peer assessment design elements are closely interconnected.
“Despite the continuous increase in peer assessment design elements, some of them are closely interconnected.”
Quote in the paperNot a substantial claim- Rests on an assumption: Peer assessment involving revisions has a formative purpose.
May combine two claims.
InterpretationcitationThe effect sizes associated with unit of assessment assessee and group constellation were very large, and the other effect sizes in this list can be considered medium to large.
“The effect sizes associated with unit of assessment assessee and group constellation were very large, and the other effect sizes in this list can be considered medium to large (Cohen, 1988).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Cohen, 1988
Not checked in the free run.
May combine two claims.
PremisecitationThe four refinements of Topping's typology distinguish between interaction between students and composition of assessment groups as if they were unrelated, which is not the case.
“these four refinements distinguish between two separate clusters describing on the one hand the interaction between students and on the other hand composition of assessment groups, as if they were unrelated, which is clearly not the case (Strijbos et al., 2009).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Strijbos et al., 2009 source
Not checked in the free run.
May combine two claims.
PremisecitationTopping (1998) concluded that peer assessment practices varied widely and first proposed a typology of 17 unordered variables to describe peer assessment designs in higher education.
“Topping (1998) concluded that peer assessment practices varied widely and first proposed a typology of 17 unordered variables to describe peer assessment designs in higher education.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Topping (1998)
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductreasoningA minimum of 70% agreement was considered adequate for each category of study details and peer assessment design elements.
“A minimum of 70% agreement was considered adequate for each category of the study details and the peer assessment design elements.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataThe minimum threshold of 70% was not obtained for relation to staff assessment, individual constellation assessor, and individual constellation assessee for coding-pair 2.
“The minimum threshold of 70% was not obtained for the following categories: relation to staff assessment, individual constellation assessor, and individual constellation assessee.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe (proto)typical peer assessment design described by Ashenafi (2017) clearly does not match with any of the five latent design patterns identified, despite around 84% of included studies being conducted in higher education.
“Yet, this (proto)typical design clearly does not match with any of the five latent design patterns we identified, despite the fact that around 84% of the studies we included have been conducted in higher education (including postgraduate).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Ashenafi (2017) source
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationcitationThe second theme is concerned with assessment groups and peer interaction within them, including group constellation, unit of assessment assessor, unit of assessment assessee, matching, contact, and privacy.
“The second theme is concerned with assessment groups and peer interaction within them (i.e.'composition of assessment groups' and 'interaction between peers'; Gielen et al., 2011).This theme includes the design elements: group constellation (intragroup vs. intergroup vs. both), unit of assessment assessor (group vs. individual vs. both), unit of assessment assessee (group vs. individual vs. both),”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Gielen et al., 2011 source
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationcitationThe first theme is decisions concerning the implementation of peer assessment, including purpose, scope of involvement, product/output of peer assessment, and whether the peer assessment involves revision.
“The first theme is decisions concerning the implementation of peer assessment (cf.Gielen et al., 2011).It includes the design elements: purpose (formative vs. summative), scope of involvement (e.g.provision and/or reception of peer assessment), the product/output of peer assessment (scoring and/or [written] comment), and whether the peer assessment involves revision.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Gielen et al., 2011 source
Not checked in the free run.
May combine two claims.
Interpretationown dataAlignment, rewards, and setting could not discriminate between the studies because in the sample typically a single peer assessment practice was used (e.g. setting=formal, and alignment to curriculum=yes).
“In fact, these design elements could not discriminate between the studies since (in our sample) typically a single peer assessment practice was used (e.g.setting = formal, and alignment to curriculum = yes).”
Quote in the paperSubstantial claimMay combine two claims.
Premisecitation'Ability and skills' is an important matching principle given there is accumulating evidence that it can affect the processes and outcomes of peer assessment.
“we stress that 'ability and skills' is an important matching principle given there is accumulating evidence that it can affect the processes and outcomes of peer assessment (e.g.Alqassab et al., 2018;Huisman et al., 2018;Patchan & Schunn, 2016;Patchan et al., 2013).”
Quote in the paperSubstantial claim0 of 3 sources checkedMay combine two claims.
Study design and conductown dataIn most cases we achieved the minimum of 70% for inter-rater reliability, except for three of the twenty-eight categories (relation to staff assessment, individual constellation assessor, and individual constellation assessee).
“In most cases we achieved the minimum of 70% for inter-rater reliability, except for three of the twenty-eight categories (relation to staff assessment, individual constellation assessor, and individual constellation assessee).”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationDouble et al. (2020) found in their meta-analysis that peer grading was beneficial for higher education students' performance, but not for students in secondary or primary education.
“For example, Double et al. (2020) found in their meta-analysis that peer grading was beneficial for higher education students' performance, but not for students in secondary or primary education.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Double et al. (2020) source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenThe five identified classes can be critiqued in that they consist of more basic or traditional design elements and they are far simpler than how peer assessment practices might be in reality.
“Finally, the five identified classes-or in other words, latent patterns that reflect five 'prototypical peer assessment designs'-can be critiqued in that they consist of more basic or traditional design elements and they are far simpler than how peer assessment practices might be in reality.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataOur findings predominantly show that diversity in research studies in higher education is limited; and for sure more limited than it can be in theory.
“Thus, our findings predominantly show that diversity in research studies in higher education is limited; and for sure more limited than it can be in theory.”
Quote in the paperSubstantial claim
The paper relies on this without support.
Sufficient time has passed for the field to notice Gielen et al. (2011) typology and assume a potential impact on reporting of peer assessment designs.
“sufficient time has passed for the field to notice this typology and assume a potential impact on the reporting of peer assessment designs.”
The paper relies on this without support.
Peer assessment involving revisions has a formative purpose.
“For instance, peer assessment involving revisions has a formative purpose.”
The paper relies on this without support.
It is still unknown if there are more commonly used peer assessment formats and design elements that could be considered when designing peer assessment activities in educational contexts.
“However, it is still unknown if there are more commonly used peer assessment formats and design elements that could be considered when designing peer assessment activities in educational contexts.”
The paper relies on this without support.
The theoretically assumed diversity of peer assessment design elements needs to be empirically tested.
“The theoretically assumed diversity of peer assessment design elements needs to be empirically tested.”
The paper relies on this without support.
There is no theoretically expected number of classes for the LCA.
“Since there was no theoretically expected number of classes, LCA models with one to eight classes were estimated”
The paper relies on this without support.
No other studies on peer assessment have employed LCA.
“because to our knowledge no other studies on peer assessment have employed LCA.”
The paper relies on this without support.
Reliability and validity of peer assessment as compared to teacher assessment has been established in general.
“Although reliability and validity of peer assessment as compared to teacher assessment has been established in general-thus supporting quantitative approaches to peer assessment in educational settings (i.e.peer scores and marks)”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The findings of the present study may not be applicable to all educational environments.
“As the pre sentation modes of peer feedback might vary across different educational settings (Alqassab et al., 2023), it is important to note that the findings of the present study may not be applicable to all educational environments.”
From A systematic review of the key components of online peer feedback practices in higher education (2024) · cites this paper as Alqassab et al., 2023
Other review studies also noted this limitation.
“This limitation was also noted by other review studies (e.g., Alqassab et al., 2023; Ashenafi, 2017).”
From A systematic review of the key components of online peer feedback practices in higher education (2024) · cites this paper as Alqassab et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Analysis plan refers to a nonexistent third research question
MinorThe paper states that latent class analysis addressed the third research question, but its numbered list contains only two research questions; latent design patterns are the second.
“The third research question was approached with latent class analysis (LCA; Vermunt & Magidson, 2005)”
“1. To what extent do peer assessment research studies differ according to subject domain, assessment purpose, objects, outcomes, and moderators/mediators? 2. Which peer assessment design elements, within all peer assessment elements combined, co-occur more often and thereby reflect latent peer assessment design patterns?”
Reliability limitations omit two categories below the threshold
MinorThe limitations name three categories as exceptions to the 70% agreement threshold, but the coding results also identify place/time and degree of interactivity as below that threshold for coding-pair 1.
“In most cases we achieved the minimum of 70% for inter-rater reliability, except for three of the twenty-eight categories (relation to staff assessment, individual constellation assessor, and individual constellation assessee).”
“The only two elements that were below the 70% threshold were place/time and degree of interactivity.”
Gender is incorrectly named the leading moderator/mediator
MinorThe abstract says gender topped the list, but the results report more occurrences for ability and skills (35) than for gender (28).
“Gender topped the list of the investigated moderators/mediators of peer assessment.”
“Moderators and mediators that were most commonly studied in relation to peer assessment designs were gender (n = 28; 27.45%) and ability and skills (n = 35; 34.31%).”
The citation lists haven’t been fetched yet
References 59 · Cited by 65
Research works printed but not linked: 5
- Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Sage Publications.
- Sluijsmans, D., Dochy, F., & Moerkerke, G. (1999). Creating a learning environment by using self-, peer- and co-assessment. Learning Environments Research, 1(3), 293–319. https://doi.org/10.1023/A:1009932704458 · DOI
- Strijbos, J. W. (2016). Assessment of collaborative learning. In G. T. L. Brown & L. R. Harris (Eds.), Handbook of social and human conditions in assessment (pp. 302–318). Routledge.
- Strijbos, J. W., Sluijsmans, D., Stegmann, K. (2017). Inferring individual scores from group scores via peer assessment: Part 1 – Methodological review [Conference presentation]. In Assessment and Evaluation SIG invited symposium EARLI 2017 conference, Tampere, Finland. EARLI.
- Wiliam, D., & Thompson, M. (2007). Integrating assessment with learning: What will it take to make it work? In C. A. Dwyer (Ed.), The future of assessment: Shaping teaching and learning (pp. 53–82). Erlbaum.
- Authors
- Alqassab, M., Strijbos, J.-W., Panadero, E., Ruiz, J. F., Warrens, M., & To, J.
- Affiliations
- Universidad de Las Palmas de Gran Canaria (ES)
- University of Groningen (NL)
- Ikerbasque (ES)
- Universidad Autónoma de Madrid (ES)
- Nanyang Technological University (SG)
- Corresponding author
- Alqassab, Maryam
- ORCID
- Venue
- Educational Psychology Review
- Published
- 2023-02-09
- Volume
- 35
- Issue
- 1
- ISSN
- 1040-726X (ISSN-L)
- 1573-336X
- OpenAlex
- W4319733082
- Semantic Scholar
- f7b2e50049609d1d20cd72b72e950d9e9b69435d
- Cited by
- 65
- References
- 59
- Citations by year
- 2026: 16 · 2025: 21 · 2024: 22 · 2023: 6
- FWCI
- 3.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Educational Psychology Review (publishedVersion, cc-by)
- NIE Digital Repository (National Institute of Education, Nanyang Technological University) (acceptedVersion)
- Acceda (Universidad de Las Palmas de Gran Canaria) (submittedVersion, cc-by)
- The HKU Scholars Hub (University of Hong Kong) (submittedVersion, cc-by-nc-nd)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- Biblos-e Archivo (Universidad Autónoma de Madrid) (submittedVersion, public-domain)
- Funders
- Universidad de las Palmas de Gran Canaria
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- peer assessment, design elements, formative assessment, written assessment, latent class analysis, purposes of assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
