CoChat
Sign Up
Cited by 6Innovations in Medical EducationMedical Education and AdmissionsHealth Education and Validation

A Scoping Review of Assessments in Undergraduate Medical Education: Implications for Residency Programs and Medical Schools

Open access
Song, Xiaomei, Cleaves, Elle, Gluzman, Ellen, Kotlyar, Biana, Russo, Rachel A., Schilling, David C., Tsao, Carol Ping, West, James C.·2025·Academic Psychiatry
DOI 10.1007/s40596-025-02136-4PMC PMC12095347
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: Assessment at medical schools plays a crucial role by providing feedback, monitoring student promotion, and informing resident selection. Limited research has been conducted to synthesize key features of assessments, and even less is known about how these studies sought validity and reliability evidence. A scoping review was performed to explore key features of assessments and their validity and reliability evidence. Methods: Various databases were searched ranging from 2004 to April 2021 following PRISMA guidelines. In-depth reviews of the full text were performed on all selected empirical studies during the extraction phase. Results: The final analysis included 218 empirical studies. Various tools were identified, ranging from traditional multiple-choice questions to more contemporary tools incorporating technology and more contextualized workplace-based assessment. Patient care and medical knowledge were the most frequently assessed, primarily using the quantitative methodology. These studies often adopted traditional perspectives in collecting validity evidence based on relations to other variables and internal structure. Many of them used a narrow conceptualization of validity, with some failing to reference validity at all. Conclusions: There was no single assessment that could easily differentiate medical students in a standardized, meaningful way. Despite the existence of contemporary validity theories for over two decades, there remains a need for greater education regarding the pivotal role of validity in conducting assessment research. While psychiatry and other residency programs will continue to face challenges in differentiating applicants, these changes present opportunities for medical educators and schools to validate assessments that are highly contextualized to their specific educational environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s40596-025-02136-4

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional written exams fail to measure complex, context-dependent decision-making and communication.

    “traditional written exams fail to measure complex, context dependent decision‑making and communication [5, 6]”

    From Reforming hospital management education with the P-MASE pedagogy: a randomized controlled trial (2026) · cites this paper as [5]

    The Discussion (page unavailable) distinguishes traditional MCQs from simulated and workplace assessments that assess communication and decision-making in context. But it does not say written exams fail to measure those abilities; the Results, “Types of Assessments” (page unavailable), describes a written Script Concordance Test used to assess clinical reasoning and multiple judgments. The categorical claim overstates the review. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [10]Coalition for Physician Accountability. The coalition for physician accountability’s undergraduate medical education-graduate medical education review committee (UGRC): recommendations for comprehensive improvement of the UME-GME transition. 2021. https://physicianaccountability.org/wp-content/uploads/2021/08/UGRC-Coalition-Report-FINAL.pdf. Accessed 24 Nov 2023.

Cited by

Sorted by citations

Authors
Song, X., Cleaves, E., Gluzman, E., Kotlyar, B., Russo, R. A., Schilling, D. C., Tsao, C. P., & West, J. C.
Affiliations
  • Case Western Reserve University (US)
  • Uniformed Services University of the Health Sciences (US)
  • Temple University (US)
  • Rosalind Franklin University of Medicine and Science (US)
  • VA San Diego Healthcare System (US)
  • Loyola University Chicago (US)
Corresponding author
Song, Xiaomei
ORCID
Venue
Academic Psychiatry
Publisher
Springer Science and Business Media LLC
Published
2025-04-01
Volume
49
Issue
3
Pages
263-273
ISSN
  • 1042-9670 (ISSN-L)
  • 1545-7230
DOI
10.1007/s40596-025-02136-4
PMC
PMC12095347
OpenAlex
W4409119517
Cited by
6
References
56
Citations by year
2026: 6
FWCI
1.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Academic Psychiatry (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
  • Social Sciences › Education › Health Education and Validation
Keywords
medical education assessment, validity evidence, undergraduate medical education, residency selection, multiple-choice questions, workplace-based assessment, assessment instruments, medical knowledge assessment, validity theory, resident selection
MeSH terms
Education, Medical, Undergraduate, Educational Measurement, Humans, Internship and Residency, Schools, Medical, Students, Medical, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review