Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Raters represent a significant source of unexplained, and often undesired, variance in performance-based assessments. To better understand rater variance, this study investigated how various raters, observing the same performance, perceived relationships amongst different noncognitive attributes measured in performance assessments. METHOD: Medical admissions data from a Multiple Mini-Interview (MMI) used at one Canadian medical school were collected and subsequently analyzed using the Many Facet Rasch Model (MFRM) and hierarchical clustering. This particular MMI consisted of eight stations. At each station a faculty member and an upper-year medical student rated applicants on various noncognitive attributes including communication, critical thinking, effectiveness, empathy, integrity, maturity, professionalism, and resolution. RESULTS: The Rasch analyses revealed differences between faculty and student raters across the eight different MMI stations. These analyses also identified that, at times, raters were unable to distinguish between the various noncognitive attributes. Hierarchical clustering highlighted differences in how faculty and student raters observed the various noncognitive attributes. Differences in how individual raters associated the various attributes within a station were also observed. CONCLUSIONS: The MFRM and hierarchical clustering helped to explain some of the variability associated with raters in a way that other measurement models are unable to capture. These findings highlight that differences in ratings may result from raters possessing different interpretations of an observed performance. This study has implications for developing more purposeful rater selection and rater profiling in performance-based assessments.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings
Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003) · Teaching and Learning in Medicine · doi:10.1207/s15328015tlm1504_11 · Cited by 391
Holmboe, E. S., Hawkins, R. E., & Huot, S. J. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-11-200406010-00008 · Cited by 288
Govaerts, M. J. B., van der Vleuten, C. P. M., Schuwirth, L. W. T., & Muijtjens, A. M. M. (2007) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9043-1 · Cited by 282
Seeing the ‘black box’ differently: assessor cognition from three research perspectives
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014) · Medical Education · doi:10.1111/medu.12546 · Cited by 243
Cook, D. A., Dupras, D. M., Beckman, T. J., Thomas, K. G., & Pankratz, V. S. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0842-3 · Cited by 196
Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9372-1 · Cited by 183
Construct Validity of the MiniClinical Evaluation Exercise (MiniCEX)
Holmboe, E. S., Huot, S., Chung, J., Norcini, J., & Hawkins, R. E. (2003) · Academic Medicine · doi:10.1097/00001888-200308000-00018 · Cited by 167
Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2004) · Assessing Writing · doi:10.1016/j.asw.2004.11.001 · Cited by 142
Mini-clinical evaluation exercise in anaesthesia training
Weller, J. M., Jolly, B. C., Misur, M., Merry, A., Jones, A., Crossley, J., Pedersen, K., & Smith, K. (2009) · British Journal of Anaesthesia · doi:10.1093/bja/aep055 · Cited by 137
Expertise in performance assessment: assessors’ perspectives
Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. (2012) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9392-x · Cited by 129
Research works printed but not linked: 1
- Sebok, 2015, A national organizations investigation of score variation in large-scale performance assessments Adv Health Sci Educ Theory Pract, 20, 581
Cited by
Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review
Yusoff, M. S. B. (2019) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2019.03.006 · Cited by 57
“, (45) 2013 Observational Study design* University of Dundee Medical School N 1⁄4 452 candidate assessed in 2009 Explore whether the MMI as used at the University of Dundee Medical School is an effective instrument that can reliably separate the more able candidates form the less able candidates Explore whether the th…”
MethodEmotions and assessment: considerations for rater‐based judgements of entrustment
Gomez-Garibello, C. A., & Young, M. E. (2017) · Medical Education · doi:10.1111/medu.13476 · Cited by 28
Daly, M., Salamonson, Y., Glew, P. J., & Everett, B. (2016) · Collegian Journal of the Royal College of Nursing Australia · doi:10.1016/j.colegn.2016.09.009 · Cited by 22
Norman, E. J. (2017) · Veterinary Record · doi:10.1136/vr.104224 · Cited by 17
“However, other patterns of co-occurrence may reflect the application of independent skills and attributes as required in veterinary clinical situations, with supervisors considering different aspects of performance in different clinical situations (Govaerts and others 2013, Sebok and Syer 2015).”
BackgroundMultiple mini interviews: revealing similarities across institutions
Griffin, B., Auton, J. C., Duvivier, R. J., Shulruf, B., & Hu, W. C. Y. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1298-8 · Cited by 17
Metadata
- Authors
- Sebok, S. S., & Syer, M. D.
- Affiliations
- Association of American Medical Colleges (US)
- ORCID
- Syer, Mark D.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-11
- Volume
- 90
- Issue
- 11 Suppl
- Pages
- S50-S55
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W1852293207
- Cited by
- 20
- References
- 14
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 2 · 2021: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 4 · 2016: 3
- FWCI
- 3.21
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Academic Medicine (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Social Sciences and Humanities Research Council of Canada
- Topics
- Social Sciences › Education › Medical Education and Admissions
- Social Sciences › Education › Innovations in Medical Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- performance-based assessment, multiple mini-interviews, medical school admissions, many-facet Rasch model, professionalism, empathy, critical thinking
- MeSH terms
- Professionalism, Canada, Communication, Empathy, Faculty, Medical, Humans, Interviews as Topic, Models, Psychological, Psychometrics, School Admission Criteria, Schools, Medical, Students, Medical, Thinking, Reproducibility of Results, Observer Variation, Cluster Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes. https://app.cochat.ai/papers/10.1097/acm.0000000000000902
- The paper
- Sebok, S. S., & Syer, M. D. (2015). Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes. Academic Medicine. https://doi.org/10.1097/acm.0000000000000902
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.