CoChat
Sign Up
Cited by 28Clinical Reasoning and Diagnostic SkillsInnovations in Medical EducationArtificial Intelligence in Healthcare and Education

Machine Scoring of Medical Students’ Written Clinical Reasoning: Initial Validity Evidence

Open access
Cianciolo, Anna T., LaVoie, Noelle, Parker, James·2021·Academic Medicine
DOI 10.1097/acm.0000000000004010PMC PMC8243833
Open this paper in CoChat — free account

Abstract

Purpose: Developing medical students’ clinical reasoning requires a structured longitudinal curriculum with frequent targeted assessment and feedback. Performance-based assessments, which have the strongest validity evidence, are currently not feasible for this purpose because they are time-intensive to score. This study explored the potential of using machine learning technologies to score one such assessment—the diagnostic justification essay. Method: From May to September 2018, machine scoring algorithms were trained to score a sample of 700 diagnostic justification essays written by 414 third-year medical students from the Southern Illinois University School of Medicine classes of 2012–2017. The algorithms applied semantically based natural language processing metrics (e.g., coherence, readability) to assess essay quality on 4 criteria (differential diagnosis, recognition and use of findings, workup, and thought process); the scores for these criteria were summed to create overall scores. Three sources of validity evidence (response process, internal structure, and association with other variables) were examined. Results: Machine scores correlated more strongly with faculty ratings than faculty ratings did with each other (machine: .28–.53, faculty: .13–.33) and were less case-specific. Machine scores and faculty ratings were similarly correlated with medical knowledge, clinical cognition, and prior diagnostic justification. Machine scores were more strongly associated with clinical communication than were faculty ratings (.43 vs .31). Conclusions: Machine learning technologies may be useful for assessing medical students’ long-form written clinical reasoning. Semantically based machine scoring may capture the communicative aspects of clinical reasoning better than faculty ratings, offering the potential for automated assessment that generalizes to the workplace. These results underscore the potential of machine scoring to capture an aspect of clinical reasoning performance that is difficult to assess with traditional analytic scoring methods. Additional research should investigate machine scoring generalizability and examine its acceptability to trainees and educators.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/acm.0000000000004010

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI was used to grade exams, particularly those involving short-answer, long-answer, and essay-based responses.

    “AI was also used to grade exams, particularly those that involve short-answer, long-answer, and essay-based responses [137–139].”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [138]

    The Method (Abstract) reports machine scoring of 700 diagnostic justification essays, and the Method section places those essays within a clinical competency exam. The Limitations section says the study was restricted to long-form essays; it does not establish that the system graded short-answer responses or other long-answer formats. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • ML effectively evaluated long-form written clinical reasoning in more than 400 third-year medical students; semantically based machine scoring captured the communicative aspects of clinical reasoning more accurately than faculty.

    “ML effectively evaluated long-form written clinical reasoning in more than 400 third-year medical students; remarkably, semantically based machine scoring was able to capture the communicative aspects of clinical reasoning more accurately than faculty [138].”

    From Artificial intelligence in undergraduate medical education: an updated scoping review (2025) · cites this paper as [138]

    The Abstract Method reports machine scoring of 700 essays from 414 third-year medical students. The Results found a stronger association with clinical communication for machine scores than for faculty ratings (.43 vs .31), but the Conclusions say machine scoring *may* capture communicative aspects better than analytic faculty ratings. The claim’s assertion that it did so “more accurately” presents this preliminary finding too definitively. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. [22]Landauer TK, Laham RD, Foltz PW. Automated scoring and annotation of essays with the Intelligent Essay Assessor. Assess Educ. 2003; 10(3): 295– 308.
  2. [23]Shermis MD, Burstein J, Higgins D, Zechner K. Automated essay scoring: Writing assessment and instruction. In: Peterson PL, Baker EL, McGaw B, eds. International Encyclopedia of Education, 3rd ed. Oxford, England: Elsevier; 2010: 75– 80.
  3. [26]Streeter L, Bernstein J, Foltz P, DeLand D. Pearson’s automated scoring of writing, speaking, and mathematics. San Antonio, TX: Pearson; 2011. https://images.pearsonassessments.com/images/tmrs/PearsonsAutomatedScoringofWritingSpeakingandMathematics.pdf. Accessed January 15, 2021.
  4. [33]Hellman S, Rosenstein M, Gorman A, Scaling up writing in the curriculum: Batch mode active learning for automated essay scoring. Paper presented at: 2019 Learning @ Scale conference; June 24–25, 2019; Chicago, IL.
  5. [36]Martin DI, Berry MW. Latent semantic indexing. In: Bates MJ, Maack MN, eds. Encyclopedia of Library and Information Sciences, 3rd ed. New York, NY: Taylor & Francis; 2010: 3195– 3204.

Cited by

Sorted by citations

Related

Authors
Cianciolo, A. T., LaVoie, N., & Parker, J.
Affiliations
  • SIU Medicine (US)
  • Parallel Consulting (United States) (US)
ORCID
Venue
Academic Medicine
Publisher
Oxford University Press (OUP)
Published
2021-02-23
Volume
96
Issue
7
Pages
1026-1035
ISSN
1040-2446 (ISSN-L)
DOI
10.1097/acm.0000000000004010
PMC
PMC8243833
OpenAlex
W3131346895
Cited by
28
References
33
Citations by year
2026: 8 · 2025: 9 · 2024: 4 · 2023: 5 · 2022: 2
FWCI
2.95
Type
journal article
Language
en
Open access
Locations
  • Academic Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
National Institute of General Medical Sciences (R42GM108104)
Topics
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
machine scoring, automated essay scoring, clinical reasoning, medical students, natural language processing, faculty ratings, validity evidence, clinical communication, medical knowledge, differential diagnosis
MeSH terms
Academic Medical Centers, Machine Learning, Clinical Reasoning, Algorithms, Communication, Curriculum, Educational Measurement, Faculty, Humans, Illinois, Research Design, Semantics, Students, Medical, Writing, Reproducibility of Results, Workplace

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review