Applying Criteria to Examples or Learning by Comparison: Effects on Students' Evaluative Judgment and Performance in Writing
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
In higher education, writing tasks are often accompanied by criteria indicating key aspects of writing quality. Sometimes, these criteria are also illustrated with examples of varying quality. It is, however, not yet clear how students learn from shared criteria and examples. This research aims to investigate the learning effects of two different instructional approaches: applying criteria to examples and comparative judgment. International business students were instructed to write a five-paragraph essay, preceded by a 30-minutes peer assessment in which they evaluated the quality of a range of example essays. Half of the students evaluated the quality of the example essays using a list of teacher-designed criteria (criteria condition; n = 20), the other group evaluated by pairwise comparisons (comparative judgment condition; n = 20). Students were also requested to provide peer feedback. Results show that the instructional approach influenced the kind of aspects students commented on when giving feedback. Students in the comparative judgment condition provided relatively more feedback on higher order aspects such as the content and structure of the text than students in the criteria condition. This was only the case for improvement feedback; for feedback on strengths there were no significant differences. Positive effects of comparative judgment on students’ own writing performance were only moderate and non-significant in this small sample. Although the transfer effects were inconclusive, this study nevertheless shows that comparative judgment can be as powerful as applying criteria to examples. Comparative judgement inherently activates students to engage with exemplars at a higher textual level and enables students to evaluate more example essays by comparison than by criteria. Further research is needed on the long-term and indirect effects of comparative judgment, as it might influence students’ conceptualization of writing, without directly improving their writing performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
Nicol, D. J., & Macfarlane‐Dick, D. (2006) · Studies in Higher Education · doi:10.1080/03075070600572090 · Cited by 5,846
Formative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · cited 4×
“According to Sadler (1989, 2002), criteria may well explain how the work will be graded, but they do so in rather discrete and abstract terms (e.g., is this text coherent or not), without revealing how the criteria are visualized in a text and how they interactively contribute to the overall…”
BackgroundInfluentialAn Introduction to Logistic Regression Analysis and Reporting
Peng, C.-Y. J., Lee, K. L., & Ingersoll, G. M. (2002) · The Journal of Educational Research · doi:10.1080/00220670209598786 · Cited by 2,671
The development of student feedback literacy: Enabling uptake of feedback
Carless, D., & Boud, D. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1463354 · Cited by 2,428
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
First principles of instruction
Merrill, M. D. (2002) · Educational Technology Research and Development · doi:10.1007/bf02505024 · Cited by 2,092 · cited 1×
“Key aspects of pedagogical interventions that successfully promote student’s learning include a combination of direct instruction, modeling, scaffolding and guided practice (Merrill, 2002).”
BackgroundScientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536 · cited 1×
“As the number of observations in each condition are rather low, we supplement the p-values with estimations of effect sizes and confidence to get insight in the magnitude and relative importance of the effects of condition (cf. Nuzzo, 2014; Wasserstein and Lazar, 2016).”
MethodRANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS
BRADLEY, R. A., & TERRY, M. E. (1952) · Biometrika · doi:10.1093/biomet/39.3-4.324 · Cited by 1,434
Beyond feedback: developing student capability in complex appraisal
Sadler, D. R. (2010) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930903541015 · Cited by 1,138
Formative Assessment: Revisiting the Territory
Sadler, D. R. (1998) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969595980050104 · Cited by 1,107 · cited 2×
“According to Sadler (1998), any feedback-enhanced intervention in which students are engaged in the process of assessing quality must be carried out long enough for it will be viewed by learners as normal and natural (p. 78).”
BackgroundRethinking feedback practices in higher education: a peer review perspective
Nicol, D., Thomson, A., & Breslin, C. (2014) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2013.795518 · Cited by 966
Developing evaluative judgement: enabling students to make decisions about the quality of work
Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018) · Higher Education · doi:10.1007/s10734-017-0220-3 · Cited by 810
Becoming a self-regulated writer: a social cognitive perspective
Zimmerman, B. J., & Risemberg, R. (1997) · Contemporary Educational Psychology · doi:10.1006/ceps.1997.0919 · Cited by 779
Examining dimensions of self-efficacy for writing
Bruning, R., Dempsey, M., Kauffman, D. F., McKim, C., & Zumbrunn, S. (2013) · Journal of Educational Psychology · doi:10.1037/a0029692 · Cited by 470
Huisman, B., Saab, N., van Driel, J., & van den Broek, P. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1424318 · Cited by 318
Orsmond, P., Merry, S., & Reiling, K. (2002) · Assessment & Evaluation in Higher Education · doi:10.1080/0260293022000001337 · Cited by 292
Appropriate criteria: key to effective rubrics
Brookhart, S. M. (2018) · Frontiers in Education · doi:10.3389/feduc.2018.00022 · Cited by 243
Transforming holistic assessment and grading into a vehicle for complex learning
Sadler, D. R. (2009) · Assessment, Learning and Judgement in Higher Education · doi:10.1007/978-1-4020-8905-3_4 · Cited by 162 · cited 6×
“This is especially relevant in the context of learning to write, as text quality is more than the sum of its constituent parts (Sadler, 2009).”
BackgroundInfluentialGenerating dialogue in assessment feedback: exploring the use of interactive cover sheets
Bloxham, S., & Campbell, L. (2010) · Assessment & Evaluation in Higher Education · doi:10.1080/02602931003650045 · Cited by 160
Managing dialogic use of exemplars
Carless, D., & Chan, K. K. H. (2017) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2016.1211246 · Cited by 151
To, J., & Carless, D. (2016) · Journal of Further and Higher Education · doi:10.1080/0309877x.2015.1014317 · Cited by 101
Yucel, R., Bird, F. L., Young, J., & Blanksby, T. (2014) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2014.880400 · Cited by 66
Cited by
The power of internal feedback: Exploiting natural comparison processes
Nicol, D. (2021) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2020.1823314 · Cited by 402
A critical review of the arguments against the use of rubrics
Panadero, E., & Jonsson, A. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100329 · Cited by 191
Comparative judgement for advancing research in applied linguistics
Thwaites, P., & Paquot, M. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100142 · Cited by 3
Metadata
- Authors
- Bouwer, R., Lesterhuis, M., Bonne, P., & De Maeyer, S.
- Venue
- Frontiers in Education
- Publisher
- Frontiers Media SA
- Published
- 2018-10-11
- Volume
- 3
- ISSN
- 2504-284X (ISSN-L)
- OpenAlex
- W2897965084
- Cited by
- 37
- References
- 23
- FWCI
- 8.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.