CoChat

Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes

Open access
Sebok, Stefanie S., Syer, Mark D. · 2015 · Academic Medicine

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Raters represent a significant source of unexplained, and often undesired, variance in performance-based assessments. To better understand rater variance, this study investigated how various raters, observing the same performance, perceived relationships amongst different noncognitive attributes measured in performance assessments. METHOD: Medical admissions data from a Multiple Mini-Interview (MMI) used at one Canadian medical school were collected and subsequently analyzed using the Many Facet Rasch Model (MFRM) and hierarchical clustering. This particular MMI consisted of eight stations. At each station a faculty member and an upper-year medical student rated applicants on various noncognitive attributes including communication, critical thinking, effectiveness, empathy, integrity, maturity, professionalism, and resolution. RESULTS: The Rasch analyses revealed differences between faculty and student raters across the eight different MMI stations. These analyses also identified that, at times, raters were unable to distinguish between the various noncognitive attributes. Hierarchical clustering highlighted differences in how faculty and student raters observed the various noncognitive attributes. Differences in how individual raters associated the various attributes within a station were also observed. CONCLUSIONS: The MFRM and hierarchical clustering helped to explain some of the variability associated with raters in a way that other measurement models are unable to capture. These findings highlight that differences in ratings may result from raters possessing different interpretations of an observed performance. This study has implications for developing more purposeful rater selection and rater profiling in performance-based assessments.

Overview

Integrity signals

Registry

Crossref · 10.1097/acm.0000000000000902

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 14 references

Research works printed but not linked: 1

  1. Sebok, 2015, A national organizations investigation of score variation in large-scale performance assessments Adv Health Sci Educ Theory Pract, 20, 581

Cited by

See all 21

Metadata

Authors
Sebok, S. S., & Syer, M. D.
Affiliations
Association of American Medical Colleges (US)
Venue
Academic Medicine
Published
2015-11
Volume
90
Issue
11 Suppl
Pages
S50-S55
ISSN
1040-2446 (ISSN-L)
OpenAlex
W1852293207
Cited by
20
References
14
Citations by year
2025: 1 · 2024: 2 · 2023: 2 · 2021: 1 · 2020: 2 · 2019: 3 · 2018: 2 · 2017: 4 · 2016: 3
FWCI
3.21
Type
journal article
Language
en
Open access
Locations
  • Academic Medicine (publishedVersion)
  • PubMed (publishedVersion)
Funders
Social Sciences and Humanities Research Council of Canada
Topics
  • Social Sciences › Education › Medical Education and Admissions
  • Social Sciences › Education › Innovations in Medical Education
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
performance-based assessment, multiple mini-interviews, medical school admissions, many-facet Rasch model, professionalism, empathy, critical thinking
MeSH terms
Professionalism, Canada, Communication, Empathy, Faculty, Medical, Humans, Interviews as Topic, Models, Psychological, Psychometrics, School Admission Criteria, Schools, Medical, Students, Medical, Thinking, Reproducibility of Results, Observer Variation, Cluster Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes. https://app.cochat.ai/papers/10.1097/acm.0000000000000902
The paper
Sebok, S. S., & Syer, M. D. (2015). Seeing Things Differently or Seeing Different Things? Exploring Raters’ Associations of Noncognitive Attributes. Academic Medicine. https://doi.org/10.1097/acm.0000000000000902

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.