The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling
Open accessAbstract
BACKGROUND: A potential problem of clinical examinations is known as the hawk-dove problem, some examiners being more stringent and requiring a higher performance than other examiners who are more lenient. Although the problem has been known qualitatively for at least a century, we know of no previous statistical estimation of the size of the effect in a large-scale, high-stakes examination. Here we use FACETS to carry out a multi-facet Rasch modelling of the paired judgements made by examiners in the clinical examination (PACES) of MRCP(UK), where identical candidates were assessed in identical situations, allowing calculation of examiner stringency. METHODS: Data were analysed from the first nine diets of PACES, which were taken between June 2001 and March 2004 by 10,145 candidates. Each candidate was assessed by two examiners on each of seven separate tasks. with the candidates assessed by a total of 1,259 examiners, resulting in a total of 142,030 marks. Examiner demographics were described in terms of age, sex, ethnicity, and total number of candidates examined. RESULTS: FACETS suggested that about 87% of main effect variance was due to candidate differences, 1% due to station differences, and 12% due to differences between examiners in leniency-stringency. Multiple regression suggested that greater examiner stringency was associated with greater examiner experience and being from an ethnic minority. Male and female examiners showed no overall difference in stringency. Examination scores were adjusted for examiner stringency and it was shown that for the present pass mark, the outcome for 95.9% of candidates would be unchanged using adjusted marks, whereas 2.6% of candidates would have passed, even though they had failed on the basis of raw marks, and 1.5% of candidates would have failed, despite passing on the basis of raw marks. CONCLUSION: Examiners do differ in their leniency or stringency, and the effect can be estimated using Rasch modelling. The reasons for differences are not clear, but there are some demographic correlates, and the effects appear to be reliable across time. Account can be taken of differences, either by adjusting marks or, perhaps more effectively and more justifiably, by pairing high and low stringency examiners, so that raw marks can be used in the determination of pass and fail.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
Bradley, R. A., & Terry, M. E. (1952) · Biometrika · doi:10.2307/2334029 · Cited by 2,412
Hambleton, R. K., & Swaminathan, H. (1985) · doi:10.1007/978-94-017-1988-9 · Cited by 1,777
Andrich, D. (1988) · doi:10.4135/9781412985598 · Cited by 1,345
Assessment of practical skills: The objective structured practical examination (OSPE)
Harden, R. M., & Cairncross, R. G. (1980) · Studies in Higher Education · doi:10.1080/03075078012331377216 · Cited by 51
Cited by
Khan, K. Z., Ramachandran, S., Gaunt, K., & Pushkar, P. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.818634 · Cited by 490
Wakeford, R., Denney, M., Ludka-Stempien, K., Dacre, J., & McManus, I. C. (2015) · BMC Medical Education · doi:10.1186/s12909-014-0281-2 · Cited by 281
Self-citationThe multiple mini-interview in the UK context: 3 years of experience at Dundee
Dowell, J., Lynch, B., Till, H., Kumwenda, B., & Husbands, A. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.652706 · Cited by 74
Roberts, C., Rothnie, I., Zoanetti, N., & Crossley, J. (2010) · Medical Education · doi:10.1111/j.1365-2923.2010.03689.x · Cited by 59
Roberts, C., Zoanetti, N., & Rothnie, I. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03292.x · Cited by 49
Metadata
- Authors
- McManus, I., Thompson, M., & Mollon, J.
- Venue
- BMC Medical Education
- Published
- 2006-08-18
- Volume
- 6
- Issue
- 1
- Pages
- 42-42
- ISSN
- 1472-6920 (ISSN-L)
- OpenAlex
- W1751360006
- Cited by
- 183
- References
- 4
- FWCI
- 1.86
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. https://app.cochat.ai/papers/10.1186/1472-6920-6-42
- The paper
- McManus, I., Thompson, M., & Mollon, J. (2006). Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Medical Education. https://doi.org/10.1186/1472-6920-6-42
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.