Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Unreliability in marking is well documented, yet we lack studies that have investigated assessors’ detailed use of assessment criteria. This project used a form of Kelly’s repertory grid method to examine the characteristics that 24 experienced UK assessors notice in distinguishing between students’ performance in four contrasting subject disciplines: that is their implicit assessment criteria. Variation in the choice, ranking and scoring of criteria was evident. Inspection of the individual construct scores in a sub-sample of academic historians revealed five factors in the use of criteria that contribute to marking inconsistency. The results imply that, whilst more effective and social marking processes that encourage sharing of standards in institutions and disciplinary communities may help align standards, assessment decisions at this level are so complex, intuitive and tacit that variability is inevitable. We conclude that universities should be more honest with themselves and with students, and actively help students to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, even experts vary in their interpretation of standards and use of marking criteria, especially when they are not made explicit from the outset (Bloxham et al. 2016).”
From Developing evaluative judgement: enabling students to make decisions about the quality of work (2018) · cites this paper as Bloxham et al. 2016
Citations
Cites
Formative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · cited 2×
“Similar notions have been theorised such as ‘teachers conceptions of quality’ (Sadler 1989, 127), assessors’ ‘evaluative frameworks’ (Broad 2003), ‘assessors’ interpretive frameworks’ (Shay 2005, 665), ‘pre-formed knowledge structures’ (Crisp 2008, 250) and ‘ways of understanding the world’ (Read,…”
BackgroundA manual for repertory grid technique
Fransella, F., Bell, R., & Bannister, D. (1977) · Thammasat University Digital Collections · Cited by 1,574
Indeterminacy in the use of preset criteria for assessment and grading
Sadler, D. R. (2009) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930801956059 · Cited by 489 · cited 6×
“This tends to reinforce arguments about the holistic nature of judgement (Sadler 2009), where assessors will balance different aspects of a piece of work.”
BackgroundInfluentialWolf, A. (2001) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 389
Assessment in Higher Education
Mathews, J. C., & Heywood, J. (1978) · British Journal of Educational Studies · doi:10.2307/3120746 · Cited by 299
What We Really Value: Beyond Rubrics in Teaching and Assessing Writing
Broad, B. (2003) · Digital Commons - USU (Utah State University) · Cited by 264
Assessment standards: the role of communities of practice and the scholarship of assessment
Price *, M. (2005) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930500063793 · Cited by 167 · cited 5×
“In addition, assessors use personal criteria beyond or different to those stated (Baume, Yorke and Coffey 2004; Broad 2003; Greatorex 2000; Price 2005; Read, Francis and Robson 2005; Webster, Pepper and Jenkins 2000) including the use of implicit assessment criteria not shared explicitly with students or other assesso…”
BackgroundInfluentialSelf-citationAssessing the Undergraduate Dissertation
Webster, F., Pepper, D., & Jenkins, A. (2000) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930050025042 · Cited by 143 · cited 2×
“…Education, 2016 Vol. 41, No. 3, 466–481, http://dx.doi.org/10.1080/02602938.2015.1024607 Greatorex 2000; Price 2005; Read, Francis, and Robson 2005; Webster, Pepper, and Jenkins 2000), including the use of implicit assessment criteria not shared explicitly with students or other assessors…”
BackgroundInfluentialMark my words: the role of assessment criteria in UK higher education grading practices
Bloxham, S., Boyd, P., & Orr, S. (2011) · Studies in Higher Education · doi:10.1080/03075071003777716 · Cited by 141 · cited 3×
“This study further clarifies the notion of personalised ‘standards frameworks’ (Bloxham, Boyd, and Orr 2011), which are conceived of as the lens through which individuals read student work.”
BackgroundSelf-citationSummative assessment: dealing with the ‘measurement fallacy’
Yorke, M. (2010) · Studies in Higher Education · doi:10.1080/03075070903545082 · Cited by 122 · cited 1×
“Short of turning our assessment methods into standardised tests, we have to live with a large element of unreliability and a recognition that grading is judgement and not measurement (Yorke 2011).”
MethodBalancing Dilemmas in Assessment and Learning in Contemporary Education
(2007) · Routledge eBooks · doi:10.4324/9780203942185 · Cited by 112
Educational Standards, Assessment, and the Search for Consensus
Moss, P., & Schutz, A. (2001) · American Educational Research Journal · doi:10.3102/00028312038001037 · Cited by 108 · cited 1×
“The language of criteria or standards can mask differences in interpretation (Moss and Schutz 2001).”
BackgroundEcclestone, K. (2001) · Journal of Further and Higher Education · doi:10.1080/03098770126527 · Cited by 106
The assessment of complex tasks: a double reading
Shay, S. (2005) · Studies in Higher Education · doi:10.1080/03075070500339988 · Cited by 101 · cited 5×
“In addition, assessors use personal criteria beyond or different to those stated (Baume, Yorke and Coffey 2004; Broad 2003; Greatorex 2000; Price 2005; Read, Francis and Robson 2005; Webster, Pepper and Jenkins 2000) including the use of implicit assessment criteria not shared explicitly with students or other assesso…”
BackgroundInfluentialStudents’ perceptions of the usefulness of marking guides, grade descriptors and annotated exemplars
Bell, A., Mladenovic, R., & Price, M. (2013) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2012.714738 · Cited by 95 · cited 1×
“Indeed, there is some evidence that students who have developed a more complex view of knowledge see criteria as guidance rather than prescription and are less dissatisfied (Bell, Mladenovic, and Price 2013).”
BackgroundSelf-citationWhat We Really Value: Beyond Rubrics in Teaching and Assessing Writing
BROAD, B. (2003) · Utah State University Press eBooks · doi:10.2307/j.ctt46nxvm · Cited by 89 · cited 3×
“Similar notions have been theorised such as ‘teachers conceptions of quality’ (Sadler 1989, 127), assessors’ ‘evaluative frameworks’ (Broad 2003), ‘assessors’ interpretive frameworks’ (Shay 2005, 665), ‘pre-formed knowledge structures’ (Crisp 2008, 250) and ‘ways of understanding the world’ (Read,…”
Quality and Qualms in the Marking of University Assignments by Sessional Staff: An Exploratory Study
Smith, E., & Coombe, K. (2006) · Higher Education · doi:10.1007/s10734-004-6376-7 · Cited by 79 · cited 3×
“A key source of variation is attributed to assessors’ different professional knowledge, experience, values and institutions (Read, Francis, and Robson 2005; Smith and Coombe 2006).”
BackgroundRead, B., Francis, B., & Robson, J. (2005) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930500063827 · Cited by 75 · cited 5×
“This replicates Read, Francis, and Robson (2005), who found that two undergraduate history essays received six different degree classifications when marked by 50 assessors.”
BackgroundInfluentialGrading Student Achievement in Higher Education
Yorke, M. (2007) · Routledge eBooks · doi:10.4324/9780203939413 · Cited by 73 · cited 1×
“This reinforces the argument that staff vary considerably both in the marks they give (Yorke 2008) and in the shape of their mark distributions (Heywood 2000), and provides some support for moderation of these distributions.”
BackgroundBaume, D., Yorke, M., & Coffey, M. (2004) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930310001689037 · Cited by 70
Hand, L., & Clewes, D. (2000) · Assessment & Evaluation in Higher Education · doi:10.1080/713611416 · Cited by 68 · cited 2×
“A final reason emerging in the literature is that assessors have different expectations of standards at the different grade levels (Grainger, Purnell, and Zipf 2008; Hand and Clewes 2000).”
BackgroundJudging quality through substantive conversations between markers
Grainger, P., Purnell, K., & Zipf, R. (2008) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930601125681 · Cited by 67
Social moderation, assessment and assuring standards for accounting graduates
Watty, K., Freeman, M. A., Howieson, B. A., Hancock, P., O’Connell, B., Lange, P. D., & Abraham, A. (2013) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2013.848336 · Cited by 67
Exploring the nature of examiner thinking during the process of examination marking
Crisp, V. (2008) · Cambridge Journal of Education · doi:10.1080/03057640802063486 · Cited by 65 · cited 2×
“…such as ‘teachers conceptions of quality’ (Sadler 1989, 127), assessors’ ‘evaluative frameworks’ (Broad 2003), ‘assessors’ interpretive frameworks’ (Shay 2005, 665), ‘pre-formed knowledge structures’ (Crisp 2008, 250) and ‘ways of understanding the world’ (Read, Francis, and Robson 2005, 242).”
BackgroundWhat makes marking reliable? Experiments with UK examinations
Baird, J., Greatorex, J., & Bell, J. F. (2004) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/0969594042000304627 · Cited by 61 · cited 2×
“23 that marking grids and assessment criteria are insufficient on their own because application of a marking scheme to a specific assignment is a ‘social construct’ negotiated between the members of that assessment community and influenced by their tacit knowledge (Baird et al. 2004).”
BackgroundReducing variation in the assessment of student writing
Hunter, K., & Docherty, P. (2009) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930903215842 · Cited by 47
Litigation and Learning: Tensions in improving university lecturers' assessment practice
Ecclestone, K., & Swann, J. (1999) · Assessment in Education: Principles, Policy & Practice · doi:10.1080/09695949992801 · Cited by 44 · cited 1×
“More importantly, this type of technical enhancement has been dismissed by other researchers (Price 2005; Swann and Ecclestone 1999), who suggest that marking grids and assessment criteria are insufficient on their own because application of a marking scheme to a specific assignment is a ‘social…”
Grading Student Achievement in Higher Education: Signals and Shortcomings
Palfreyman, D. M. (2010) · Perspectives Policy and Practice in Higher Education · doi:10.1080/13603100902808726 · Cited by 34
O'Hagan, S., & Wigglesworth, G. (2014) · Studies in Higher Education · doi:10.1080/03075079.2014.896890 · Cited by 33
External peer review of assessment: an effective approach to verifying standards?
Bloxham, S., Hudson, J., den Outer, B., & Price, M. (2015) · Higher Education Research & Development · doi:10.1080/07294360.2015.1024629 · Cited by 28
Self-citationHerbert, I. P., Joyce, J., & Hassall, T. (2014) · Accounting Education · doi:10.1080/09639284.2014.974195 · Cited by 20 · cited 1×
“It may prevent markers awarding good marks to unexpected responses (Herbert, Joyce, and Hassall 2014).”
BackgroundAssessment beyond belief: The cognitive process of grading
Orrell, J. (2007) · Routledge eBooks · doi:10.4324/9780203942185-28 · Cited by 19
Research works printed but not linked: 1
- Greatorex J. 2000. “Is the Glass Half Full or Half Empty? What Examiners Really Think of Candidates’ Achievement.” Paper presented at the British Educational Research Association Conference Cardiff September 7–10 2000.
Cited by
Developing evaluative judgement: enabling students to make decisions about the quality of work
Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018) · Higher Education · doi:10.1007/s10734-017-0220-3 · Cited by 810
“However, even experts vary in their interpretation of standards and use of marking criteria, especially when they are not made explicit from the outset (Bloxham et al. 2016).”
BackgroundAssessment for inclusion: rethinking contemporary strategies in assessment design
Tai, J., Ajjawi, R., Bearman, M., Boud, D. J., Dawson, P., & Jorre, T. J. D. S. (2022) · Higher Education Research & Development · doi:10.1080/07294360.2022.2057451 · Cited by 211
Does the use of summative peer assessment in collaborative group work inhibit good judgement?
Sridharan, B., Tai, J., & Boud, D. (2019) · Higher Education · doi:10.1007/s10734-018-0305-7 · Cited by 106
“However, it is well documented that reliability and validity of marking is a big problem even amongst experienced assessors (Bloxham et al. 2016).”
BackgroundInfluentialCan a rubric do more than be transparent? Invitation as a new metaphor for assessment criteria
Bearman, M., & Ajjawi, R. (2021) · Studies in Higher Education · doi:10.1080/03075079.2019.1637842 · Cited by 83
Good and bad practices in rubrics: the perspectives of students and educators
Chan, C. Y. Z., & Ho, S. S. M. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1522528 · Cited by 83
Metadata
- Authors
- Bloxham, S., den-Outer, B., Hudson, J., & Price, M.
- Affiliations
- University of Cumbria (GB)
- ,
- Oxford Brookes University (GB)
- Corresponding author
- Bloxham, Sue
- ORCID
- Price, Margaret
- Venue
- Assessment & Evaluation in Higher Education
- Publisher
- Informa UK Limited
- Published
- 2015-03-23
- Volume
- 41
- Issue
- 3
- Pages
- 466-481
- ISSN
- 0260-2938 (ISSN-L)
- 1469-297X
- OpenAlex
- W1983517965
- Cited by
- 135
- References
- 32
- Citations by year
- 2026: 16 · 2025: 12 · 2024: 17 · 2023: 11 · 2022: 10 · 2021: 14 · 2020: 8 · 2019: 12 · 2018: 20 · 2017: 10 · 2016: 2 · 2015: 3
- FWCI
- 58.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Assessment & Evaluation in Higher Education (publishedVersion)
- Insight (University of Cumbria) (acceptedVersion)
- Radar (Oxford Brookes University) (submittedVersion)
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Education › Educational Assessment and Pedagogy
- Social Sciences › Education › Evaluation of Teaching Practices
- Keywords
- evaluation criteria, academic historians, repertory grid technique, tacit knowledge, higher education assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.