Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 3 of the 16 sources its claims cite were checked for support.

AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation

Open access

Abstract

Background: Multiple mini interviews (MMIs) are widely used in medical school admissions to assess applicants' nonacademic attributes in a structured and reliable manner. However, the development of high-quality MMI stations is resource intensive and dependent on expert input. Objective: This study explored the utility of artificial intelligence (AI) in the generation of MMI stations for the Direct and Graduate Entry Medicine Program admissions process for domestic applicants at Monash Medical School. To our knowledge, this study represents the first empirical evaluation of AI-generated MMI stations deployed in a real-world medical school admissions context. Methods: A total of 56 MMI stations from the 2025 admissions cycle were evaluated, including 17 (30.4%) AI-generated and 39 (69.6%) traditionally developed stations, administered across 824 domestic applicants for a total of 4897 applicant-station interactions. We assessed station quality through both reliability (using Cronbach α to examine internal consistency) and discrimination capability (using SD and range of scores) at the station level. Results: AI-generated stations exhibited slightly higher reliability (α=0.82) compared with traditional stations (α=0.81), though this difference was not statistically significant (P=.91). Both AI-generated and traditionally developed stations demonstrated variable discrimination capability, with some stations from each development method showing excellent combinations of high reliability and strong discriminatory power, while others exhibited ceiling effects that limited their discriminatory power. Of note, a greater proportion of AI-generated stations were classified as optimal (α>0.85), and a smaller proportion were classified in the review category (α<0.75), compared with traditional stations. These results suggest that AI-generated stations can achieve psychometric performance comparable to traditionally developed stations. Conclusions: Our findings highlight the utility of AI as a useful tool for MMI station generation, offering a scalable approach that may reduce the resource burden on faculty while maintaining or enhancing psychometric quality for applicants. Ongoing quality assurance and evaluation remain essential to ensure fairness and validity across the admissions process.

Overview

CoChat Summary

TL;DR

In a real-world 2025 Monash medical school admissions cycle, 17 AI-generated MMI stations showed internal consistency comparable to 39 traditionally developed stations (mean Cronbach α=0.82 vs 0.81; P=.91). Both development methods produced stations across the full psychometric quality spectrum, supporting AI as a scalable complementary tool that still requires ongoing quality assurance.

Key findings

  • AI-generated and traditional stations did not differ significantly in reliability: mean α=0.82 (SD 0.06) for AI-generated vs 0.81 (SD 0.09) for traditional; Mann-Whitney U=325, z=−0.116, r=−0.015, P=.91.
  • A greater proportion of AI-generated stations were classified as optimal (α>0.85): 41.2% (7/17; 95% CI 22%-64%) vs 33.3% (13/39; 95% CI 21%-49%) for traditional stations; fewer AI-generated stations were in the review category (α<0.75): 5.9% (1/17; 95% CI 1%-27%) vs 12.8% (5/39; 95% CI 6%-27%).
  • Discrimination varied within both groups. High performers included AI-generated EMP_17 (α=0.90; mean 13.8, SD 3.9; range 2-20) and traditional EMP_6 (α=0.93; mean 14.2, SD 4.5; range 3-20). Poor performers included traditional COLL_14 (α=0.3925; mean 17.3, SD 1.6; range 15-20) and ceiling-effect stations ADV_21 (α=.75; mean 17.3, SD 2.3) and MOT_12 (α=.79; mean 17.6, SD 2.1).
  • Sensitivity analysis excluding stations with n<50 showed no significant reliability difference (P=.89), and category distributions became similar: optimal 41% (7/17) AI-generated vs 37% (33/39) traditional; acceptable 53% vs 54%; review 6% vs 13%.

Methods

  • Retrospective psychometric evaluation of 56 MMI stations from the 2025 admissions cycle for domestic applicants to Monash University’s Direct Entry and Graduate Entry Medicine Program: 17 (30.4%) AI-generated and 39 (69.6%) traditionally developed, mapped to 7 MMI domains.
  • Data came from 824 domestic applicants and 4897 applicant-station interactions; each station was scored on a 20-point scale (5 questions, each 0-4), with 252 trained interviewers and 2-17 interviewers per station across sessions. AI stations were generated using Claude 3.5 Sonnet with a structured prompt, then reviewed by faculty and senior academic leads.
  • Main analyses calculated station-level Cronbach α and discrimination metrics (mean, SD, range), compared station types using Mann-Whitney U tests, categorized stations into review/acceptable/optimal reliability groups, and ran sensitivity analyses excluding n<50 stations plus linear regression controlling for MMI domain and interaction models.

Limitations

  • The paper states that some stations had small sample sizes (n<50), especially some AI-generated stations, producing less stable station-level Cronbach α estimates; sensitivity analyses attenuated category-based differences.
  • The paper states that each candidate was assessed by a single interviewer per station, so interinterviewer reliability and interviewer variance were not examined.
  • The paper states that the proprietary large language model lacks transparency regarding training data and internal parameters, so potential biases are not eliminated; it also notes wide, overlapping CIs around category proportions and that station-level α reflects the station rubric rather than the overall MMI decision. Evident from the text: this is a single-institution, single-cycle evaluation without formal prospective noninferiority testing for validity or fairness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/86208

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Comparison of internal consistency reliability (Cronbach α) between artificial intelligence (AI)–generated and traditionally developed multiple mini interview stations.
Figure 1. Comparison of internal consistency reliability (Cronbach α) between artificial intelligence (AI)–generated and traditionally developed multiple mini interview stations.CC BY · PubMed Central
Figure 2. Distribution of reliability categories for artificial intelligence (AI)–generated and existing multiple mini interview stations.
Figure 2. Distribution of reliability categories for artificial intelligence (AI)–generated and existing multiple mini interview stations.CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 0 Not valid · 9 Not checked · 68 Own results and interpretations

38 more claims

Create a free account to see all 80 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 18
  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 18 references

Research works printed but not linked: 1

  1. Ayub, J Pak Med Assoc, 67, 1905

Metadata

Authors
Hamila, S., Birchill, K., Cao, K., Hossain, M. N., Bullock, S., Hodgson, W., Harrison, J., & Leech, M.
Affiliations
Monash University (AU)
Corresponding author
Hamila, Sabryn
Venue
JMIR Medical Education
Published
2026-06-11
Volume
12
Pages
e86208-e86208
ISSN
2369-3762 (ISSN-L)
OpenAlex
W7151288766
Cited by
0
References
18
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Social Sciences › Education › Medical Education and Admissions
  • Social Sciences › Education › Innovations in Medical Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
multiple mini-interviews, medical school admissions, psychometric assessment, discriminatory power, ceiling effect
MeSH terms
Artificial Intelligence, Humans, Interviews as Topic, Psychometrics, School Admission Criteria, Schools, Medical, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation. https://app.cochat.ai/papers/10.2196/86208
The paper
Hamila, S., Birchill, K., Cao, K., Hossain, M. N., Bullock, S., Hodgson, W., Harrison, J., & Leech, M. (2026). AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation. JMIR Medical Education. https://doi.org/10.2196/86208

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.