The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 3 of the 16 sources its claims cite were checked for support.
AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation
Open accessAbstract
Background: Multiple mini interviews (MMIs) are widely used in medical school admissions to assess applicants' nonacademic attributes in a structured and reliable manner. However, the development of high-quality MMI stations is resource intensive and dependent on expert input. Objective: This study explored the utility of artificial intelligence (AI) in the generation of MMI stations for the Direct and Graduate Entry Medicine Program admissions process for domestic applicants at Monash Medical School. To our knowledge, this study represents the first empirical evaluation of AI-generated MMI stations deployed in a real-world medical school admissions context. Methods: A total of 56 MMI stations from the 2025 admissions cycle were evaluated, including 17 (30.4%) AI-generated and 39 (69.6%) traditionally developed stations, administered across 824 domestic applicants for a total of 4897 applicant-station interactions. We assessed station quality through both reliability (using Cronbach α to examine internal consistency) and discrimination capability (using SD and range of scores) at the station level. Results: AI-generated stations exhibited slightly higher reliability (α=0.82) compared with traditional stations (α=0.81), though this difference was not statistically significant (P=.91). Both AI-generated and traditionally developed stations demonstrated variable discrimination capability, with some stations from each development method showing excellent combinations of high reliability and strong discriminatory power, while others exhibited ceiling effects that limited their discriminatory power. Of note, a greater proportion of AI-generated stations were classified as optimal (α>0.85), and a smaller proportion were classified in the review category (α<0.75), compared with traditional stations. These results suggest that AI-generated stations can achieve psychometric performance comparable to traditionally developed stations. Conclusions: Our findings highlight the utility of AI as a useful tool for MMI station generation, offering a scalable approach that may reduce the resource burden on faculty while maintaining or enhancing psychometric quality for applicants. Ongoing quality assurance and evaluation remain essential to ensure fairness and validity across the admissions process.
Overview
CoChat Summary
TL;DR
In a real-world 2025 Monash medical school admissions cycle, 17 AI-generated MMI stations showed internal consistency comparable to 39 traditionally developed stations (mean Cronbach α=0.82 vs 0.81; P=.91). Both development methods produced stations across the full psychometric quality spectrum, supporting AI as a scalable complementary tool that still requires ongoing quality assurance.
Key findings
- AI-generated and traditional stations did not differ significantly in reliability: mean α=0.82 (SD 0.06) for AI-generated vs 0.81 (SD 0.09) for traditional; Mann-Whitney U=325, z=−0.116, r=−0.015, P=.91.
- A greater proportion of AI-generated stations were classified as optimal (α>0.85): 41.2% (7/17; 95% CI 22%-64%) vs 33.3% (13/39; 95% CI 21%-49%) for traditional stations; fewer AI-generated stations were in the review category (α<0.75): 5.9% (1/17; 95% CI 1%-27%) vs 12.8% (5/39; 95% CI 6%-27%).
- Discrimination varied within both groups. High performers included AI-generated EMP_17 (α=0.90; mean 13.8, SD 3.9; range 2-20) and traditional EMP_6 (α=0.93; mean 14.2, SD 4.5; range 3-20). Poor performers included traditional COLL_14 (α=0.3925; mean 17.3, SD 1.6; range 15-20) and ceiling-effect stations ADV_21 (α=.75; mean 17.3, SD 2.3) and MOT_12 (α=.79; mean 17.6, SD 2.1).
- Sensitivity analysis excluding stations with n<50 showed no significant reliability difference (P=.89), and category distributions became similar: optimal 41% (7/17) AI-generated vs 37% (33/39) traditional; acceptable 53% vs 54%; review 6% vs 13%.
Methods
- Retrospective psychometric evaluation of 56 MMI stations from the 2025 admissions cycle for domestic applicants to Monash University’s Direct Entry and Graduate Entry Medicine Program: 17 (30.4%) AI-generated and 39 (69.6%) traditionally developed, mapped to 7 MMI domains.
- Data came from 824 domestic applicants and 4897 applicant-station interactions; each station was scored on a 20-point scale (5 questions, each 0-4), with 252 trained interviewers and 2-17 interviewers per station across sessions. AI stations were generated using Claude 3.5 Sonnet with a structured prompt, then reviewed by faculty and senior academic leads.
- Main analyses calculated station-level Cronbach α and discrimination metrics (mean, SD, range), compared station types using Mann-Whitney U tests, categorized stations into review/acceptable/optimal reliability groups, and ran sensitivity analyses excluding n<50 stations plus linear regression controlling for MMI domain and interaction models.
Limitations
- The paper states that some stations had small sample sizes (n<50), especially some AI-generated stations, producing less stable station-level Cronbach α estimates; sensitivity analyses attenuated category-based differences.
- The paper states that each candidate was assessed by a single interviewer per station, so interinterviewer reliability and interviewer variance were not examined.
- The paper states that the proprietary large language model lacks transparency regarding training data and internal parameters, so potential biases are not eliminated; it also notes wide, overlapping CIs around category proportions and that station-level α reflects the station rubric rather than the overall MMI decision. Evident from the text: this is a single-institution, single-cycle evaluation without formal prospective noninferiority testing for validity or fairness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
3 Valid · 0 Not valid · 9 Not checked · 68 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other stations showing this desirable combination of high reliability and strong discrimination included ADV_18 (α=0.8925; mean 12.6, SD 4.1) and ER_6.1 (α=0.88; mean 14.2, SD 4.0).”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Results > Station Quality Analysis: Reliability and Discrimination Ability
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though this difference was not statistically significant (P=.91)”
- Also stated in: § Overall Reliability of AI-Generated and Traditional Stations · plain
“However, this difference was neither statistically significant (U=325; z=–0.116; r=−0.015; P=.91) nor was the difference large enough to be practically relevant (Figure 1).”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Overall Reliability of AI-Generated and Traditional Stations · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, some stations with acceptable reliability showed limited discrimination capability.”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Results > Station Quality Analysis: Reliability and Discrimination Ability
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both AI-generated and traditionally developed stations demonstrated variable discrimination capability”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Abstract > Results
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results showed that AI-generated stations demonstrated comparable internal consistency to traditionally developed stations (mean α=0.82, SD 0.06 vs mean α=0.81, SD 0.09; P=.91).”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Discussion > Implications for the Use of AI in Medical School Admissions
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims
Create a free account to see all 80 claims and every source verdict.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 24
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Results > Overall Reliability of AI-Generated and Traditional Stations
“When restricted to stations with n≥50, there remained no significant difference in internal consistency between AI-generated and traditionally developed stations (P=.89). The distribution of reliability categories was similar between groups, with 41% (7/17) of AI-generated and 37% (33/39) of traditionally developed stations classified as optimal”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Results > Station Quality Analysis: Reliability and Discrimination Ability
“It should be noted that stations ADV_21, MOT_12, and COLL_14 had fewer than 50 candidate responses (Multimedia Appendix 2)”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“with 41% (7/17) of AI-generated and 37% (33/39) of traditionally developed stations classified as optimal”
§ AI-Generated Multiple Mini Interview (MMI) Stations for Medical School Admissions: Psychometric Evaluation > Results > Overall Reliability of AI-Generated and Traditional Stations
Reported 37 · recomputed 84.615385
Citations
Cites
Making sense of cronbach’s alpha.
Tavakol, M., & Dennick, R. (2011) · International Journal of Medical Education · doi:10.5116/ijme.4dfb.8dfd · Cited by 15,126
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
Reliability: on the reproducibility of assessment data
Downing, S. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01932.x · Cited by 721
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 679
Time-Limit Tests: Estimating their Reliability and Degree of Speeding
Cronbach, L. J., & Warrington, W. G. (1951) · Psychometrika · doi:10.1007/bf02289113 · Cited by 677
An admissions OSCE: the multiple mini-interview
Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01776.x · Cited by 617
The ability of the multiple mini-interview to predict preclerkship performance in medical school
Eva, K. W., Reiter, H. I., Rosenfeld, J., & Norman, G. R. (2004) · Academic Medicine · doi:10.1097/00001888-200410001-00012 · Cited by 225
Artificial Intelligence Revolutionizing the Field of Medical Education
Narayanan, S., Ramakrishnan, R., Durairaj, E., & Das, A. (2023) · Cureus · doi:10.7759/cureus.49604 · Cited by 164
Mapping the use of artificial intelligence in medical education: a scoping review
Rincón, E. H. H., Jimenez, D., Aguilar, L. A. C., Flórez, J. M. P., Tapia, Á. E. R., & Peñuela, C. L. J. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07089-8 · Cited by 145
Rees, E. L., Hawarden, A. W., Dent, G., Hays, R., Bates, J., & Hassell, A. B. (2016) · Medical Teacher · doi:10.3109/0142159x.2016.1158799 · Cited by 139
Research works printed but not linked: 1
- Ayub, J Pak Med Assoc, 67, 1905
Metadata
- Authors
- Hamila, S., Birchill, K., Cao, K., Hossain, M. N., Bullock, S., Hodgson, W., Harrison, J., & Leech, M.
- Affiliations
- Monash University (AU)
- Corresponding author
- Hamila, Sabryn
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2026-06-11
- Volume
- 12
- Pages
- e86208-e86208
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC13256173
- OpenAlex
- W7151288766
- Cited by
- 0
- References
- 18
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Education › Medical Education and Admissions
- Social Sciences › Education › Innovations in Medical Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- multiple mini-interviews, medical school admissions, psychometric assessment, discriminatory power, ceiling effect
- MeSH terms
- Artificial Intelligence, Humans, Interviews as Topic, Psychometrics, School Admission Criteria, Schools, Medical, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.