Reliability and acceptability of a five-station multiple mini-interview model for residency program recruitment
Open accessAbstract
BACKGROUND: Standard interviews are used by most residency programs in the United States for assessment of aptitude of the non-cognitive competencies, but variability of interviewer skill, interviewer bias, interviewer leniency or stringency, and context specificity limit reliability. AIM: To investigate reliability and acceptability of five-station multiple mini-interview (MMI) model for resident selection into an internal medicine residency program in the United States. SETTING: One independent academic medical center. PARTICIPANTS: Two hundred and thirty-seven applicants and 17 faculty interviewers. PROGRAM DESCRIPTION: Five, 10-min MMI stations with five different interviewers blinded to the candidate's records and one traditional 20-min interview with the program director. Candidates were rated on two items: interpersonal and communication skills, and overall performance. PROGRAM EVALUATION: Generalizability data showed that the reliability of our process was high (>0.9). The results of anonymous surveys demonstrated that both applicants and interviewers consider the MMI as a fair and more effective tool to evaluate non-cognitive traits, and prefer the MMI to standard interviews. DISCUSSION: The MMI process for residency interviews can generate reliable interview results using only five stations, and it is acceptable and preferred over standard interview modalities by the applicants and faculty members of one US residency program.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Fraga et al. (2013), G-coefficients for each station ranged from 0.97 for the Last Call station to 0.98 for the Overloaded Census station.
“G-coefficient are reported for each station and ranged from a minimum of 0.97 for the Last Call station to a high of 0.98 for Overloaded Census station”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
The Results identify Last Call as the lowest station and Overloaded Census as the highest, but report their G-coefficients as 0.9797 and 0.9848, respectively. The claimed 0.97 for Last Call truncates rather than rounds the reported value. (LLM, conf=0.88)
Experimental — accuracy not yet measuredIn Fraga et al. (2013), the G-coefficient for interviewer within candidate was 0.95.
“G-coefficient for interviewer within candidate = 0.95”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), there was no difference in perceived stressfulness of the MMI compared to the traditional interview (3.06 vs. 3.18, p=0.32).
“No difference in perceived stressfulness of the MMI compared to the traditional interview (3.06 vs. 3.18, p = 0.32)”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), applicants had higher agreement that 'the MMI is effective at evaluating non-cognitive skills' than that 'the traditional interview is effective evaluating non-cognitive skills' (5.05 vs. 3.41, p<0.001).
“Applicants had higher agreement that ‘the MMI is effective at evaluating non-cognitive skills’ than for ‘the traditional interview is effective evaluating non-cognitive skills’ (5.05 vs. 3.41, p < 0.001)”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), generalizability data showed that even with only five stations, the reliability of the process was high enough for high-stakes decisions such as admissions (>0.9 for candidate within station).
“Generalizability data showed that even with only five stations, the reliability of the process was high enough for high-stakes decisions such as admissions (>0.9 for candidate within station)”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), the G-coefficient for station within interviewer was 0.98.
“G-coefficient for station within interviewer = 0.98”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), applicants agreed more strongly that 'the MMI was fair' than that 'a traditional interview is fair' (5.12 vs. 4.07, p<0.001).
“Applicants indicated that they agreed with the statements that ‘the MMI was fair’ more strongly than with ‘a traditional interview is fair’ (5.12 vs. 4.07, p < 0.001)”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
In Fraga et al. (2013), the G-coefficient for combinations of candidate within station was 0.96.
“G-coefficient for combinations of candidate within station = 0.96”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Fraga J. D. et al., (38) 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
General competencies and accreditation in graduate medical education.
Batalden, P., Leach, D., Swing, S. R., Dreyfus, H., & Dreyfus, S. B. (2002) · Health Affairs · doi:10.1377/hlthaff.21.5.103 · Cited by 699 · see citing works
An admissions OSCE: the multiple mini-interview
Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01776.x · Cited by 617 · see citing works
Predictive validity of the multiple mini-interview for selecting medical trainees
Eva, K. W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G. R. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03407.x · Cited by 267 · see citing works
Multiple mini-interviews predict clerkship and licensing examination performance
Reiter, H. I., Eva, K. W., Rosenfeld, J., & Norman, G. R. (2007) · Medical Education · doi:10.1111/j.1365-2929.2007.02709.x · Cited by 235 · see citing works
The interview in the admission process
Edwards, J. C., Johnson, E. K., & Molidor, J. B. (1990) · Academic Medicine · doi:10.1097/00001888-199003000-00008 · Cited by 192 · see citing works
Engel, B. T. (1967) · Archives of General Psychiatry · doi:10.1001/archpsyc.1967.01730230058008 · Cited by 185 · see citing works
Assessment of non-cognitive traits through the admissions multiple mini-interview
Lemay, J., Lockyer, J. M., Collin, V. T., & Brownell, A. K. W. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02767.x · Cited by 156 · see citing works
Investigating the Reliability of the Medical School Admissions Interview
Kreiter, C. D., Yin, P., Solow, C. M., & Brennan, R. L. (2004) · Advances in Health Sciences Education · doi:10.1023/b:ahse.0000027464.22411.0f · Cited by 153 · see citing works
Roberts, C., Walton, M., Rothnie, I., Crossley, J., Lyon, P., Kumar, K., & Tiller, D. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03018.x · Cited by 121 · see citing works
Eva, K. W., Reiter, H., Rosenfeld, J. M., & Norman, G. R. (2004) · Academic Medicine · doi:10.1097/00001888-200406000-00021 · Cited by 112 · see citing works
Harris, S., & Owen, C. (2007) · Medical Education · doi:10.1111/j.1365-2929.2007.02682.x · Cited by 101 · see citing works
Dore, K. L., Kreuger, S., Ladhani, M., Rolfson, D., Kurtz, D., Kulasegaram, K., Cullimore, A. J., Norman, G. R., Eva, K. W., Bates, S., et al. (2010) · Academic Medicine · doi:10.1097/acm.0b013e3181ed442b · Cited by 98 · see citing works
Multiple mini-interviews versus traditional interviews: stakeholder acceptability comparison
Razack, S., Faremo, S., Drolet, F., Snell, L., Wiseman, J., & Pickering, J. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03447.x · Cited by 82 · see citing works
Quintero, A., Segal, L. S., King, T. S., & Black, K. P. (2009) · Academic Medicine · doi:10.1097/acm.0b013e3181b6a9af · Cited by 81 · see citing works
Multiple mini-interviews: opinions of candidates and interviewers
Humphrey, S., Dowson, S., Wall, D., Diwakar, V., & Goodyear, H. M. (2008) · Medical Education · doi:10.1111/j.1365-2923.2007.02972.x · Cited by 74 · see citing works
Hofmeister, M., Lockyer, J., & Crutcher, R. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03380.x · Cited by 73 · see citing works
Editorial – The Morality of Medical School Admissions
Norman, G. R. (2004) · Advances in Health Sciences Education · doi:10.1023/b:ahse.0000027553.28703.cf · Cited by 66 · see citing works
Reliability and validity of interviewersʼ judgments of medical school candidates
Harasym, P. H., Woloschuk, W., Mandin, H., & Brundin‐Mather, R. (1996) · Academic Medicine · doi:10.1097/00001888-199601000-00038 · Cited by 57 · see citing works
The acceptability of the multiple mini interview for resident selection.
(2008) · PubMed · Cited by 49 · see citing works
Elam, C. L., & Andrykowski, M. A. (1991) · PubMed · Cited by 35 · see citing works
Interviewer scoring differences in student selection interviews.
Wc, M. (1979) · PubMed · Cited by 24 · see citing works
An analysis of admission committee voting patterns
Elam, C. L., & Johnson, M. M. (1997) · Academic Medicine · doi:10.1097/00001888-199710001-00025 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review
Yusoff, M. S. B. (2019) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2019.03.006 · Cited by 57 · see citing works
“, (27) 2010 Historical Study design* University of Sydney, Australia N 1⁄4 207 interviewer; 686 candidates Medicine and Dentistry Admission Using the same candidates and interviewer in Questions Bank in MMI article (Roberts C.”
BackgroundResidency Placement Fever: Is It Time for a Reevaluation?
Gruppuso, P. A., & Adashi, E. Y. (2016) · Academic Medicine · doi:10.1097/acm.0000000000001468 · Cited by 42 · see citing works
<p>Multiple Mini-Interviews: Current Perspectives on Utility and Limitations</p>
Ali, S., Sadiq Hashmi, M. S., Umair, M., Beg, M. A., & Huda, N. (2019) · Advances in Medical Education and Practice · doi:10.2147/amep.s181332 · Cited by 32 · see citing works
Callwood, A., Jeevaratnam, K., Kotronoulas, G., Schneider, A., Lewis, L. S., & Nadarajah, V. D. (2018) · Nurse Education Today · doi:10.1016/j.nedt.2018.01.016 · Cited by 174 · see citing works
Glazer, G., Startsman, L. F., Bankston, K., Michaels, J., Danek, J., & Fair, M. A. (2016) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2016.13.12 · Cited by 24 · see citing works
Yoshimura, H., Kitazono, H., Fujitani, S., Machi, J., Saiki, T., Suzuki, Y., & Ponnamperuma, G. G. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0361-y · Cited by 24 · see citing works
Correlation between MMI performance and OSCE performance – a pilot study
Oluwasanjo, A., Wasser, T., & Alweis, R. L. (2015) · Journal of Community Hospital Internal Medicine Perspectives · doi:10.3402/jchimp.v5.27808 · Cited by 21 · see citing works
Self-citationMini Surgical Simulation, Role Play, and Group and Behavioral Interviews in Resident Selection
Ogunyemi, D. A., Alexander, C. J., Tangchitnob, E., & Kim, D. S. (2016) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-15-00203.1 · Cited by 18 · see citing works
Min, A., Leetch, A. N., Nuño, T., & Fiorello, A. B. (2016) · Medical Education Online · doi:10.3402/meo.v21.29587 · Cited by 11 · see citing works
Multiple mini-interviews as a selection tool for initial teacher education admissions
Metsäpelto, R., Utriainen, J., Poikkeus, A., Muotka, J. S., Tolvanen, A. J., & Warinowski, A. (2022) · Teaching and Teacher Education · doi:10.1016/j.tate.2022.103660 · Cited by 11 · see citing works
Abdul Rahim, A. F., & Yusoff, M. S. B. (2016) · Education in Medicine Journal · doi:10.5959/eimj.v8i2.437 · Cited by 10 · see citing works
Rater perceptions of bias using the multiple mini-interview format: a qualitative study
Alweis, R. L., Fitzpatrick, C., & Donato, A. A. (2015) · Journal of Education and Training Studies · doi:10.11114/jets.v3i5.818 · Cited by 9 · see citing works
Self-citationKulasegaram, K. M., Baxan, V., Giannone, E., Latter, D. A., & Hanson, M. D. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004331 · Cited by 7 · see citing works
Burgos, L. M., de Lima, A. A., Parodi, J. B., Costabel, J. P., Ganiele, M. D. L. N., Durante, E., Arceo, M. D., & Gelpi, R. J. (2020) · Europe PMC (PubMed Central) · doi:10.30476/jamp.2019.83903.1116 · Cited by 7 · see citing works
Dao, A. T., Garcia, M. M., Correa, R. R., Gay, L. J., Wininger, D. A., Sweet, M., Luther, V. P., Chow, T. M., Harper, W., & Lai, C. J. (2022) · The American Journal of Medicine · doi:10.1016/j.amjmed.2022.08.001 · Cited by 5 · see citing works
Developing and evaluating the multiple mini interview in student midwife selection
Callwood, A. (2015) · Surrey Research Insight Open Access (The University of Surrey) · Cited by 4 · see citing works
Ferguson, R. P. (2013) · Journal of Community Hospital Internal Medicine Perspectives · doi:10.3402/jchimp.v3i3-4.23202 · Cited by 2 · see citing works
Multiple Mini Interview as a Selection Tool for Initial Teacher Education AdmissionsPreprint
Metsäpelto, R., Utriainen, J., Poikkeus, A., Muotka, J. S., Tolvanen, A. J., & Warinowski, A. (2020) · doi:10.31234/osf.io/ph5t8 · Cited by 2 · see citing works
Utilizing Multiple Mini Interviews in Medical School Admissions
Apple, R. W. (2017) · Journal of Psychology & Clinical Psychiatry · doi:10.15406/jpcpy.2017.07.00462 · Cited by 1 · see citing works
Multiple mini-interview for undergraduate allied health programs in Singapore
Yeung, M. T. L. (2021) · Education in the Health Professions · doi:10.4103/ehp.ehp_15_21
Kolagari, S., Sabzi, Z., Modanloo, M., & Behnampour, N. (2022) · Bangladesh Journal of Medical Science · doi:10.3329/bjms.v21i4.60249
Fasan, M. A., Burgos, L. M., Sigal, A. R., Costabel, J. P., & de Lima, A. A. (2023) · Revista Argentina de Cardiología · doi:10.7775/rac.es.v91.i5.20674
Validation of a mini focus group interview for postgraduate year training program selection
Yen, J., Yang, C., Lin, Y., Tsai, M., Lu, P.-L., & Chen, C. (2026) · Tzu Chi Medical Journal · doi:10.4103/tcmj.tcmj-d-26-00061
- Authors
- Fraga, J. D., Oluwasanjo, A., Wasser, T., Donato, A., & Alweis, R.
- Affiliations
- Thomas Jefferson University Hospital (US)
- Reading Hospital (US)
- Consult-StatMacungie, PA, USA
- Corresponding author
- Alweis, Richard
- Venue
- Journal of Community Hospital Internal Medicine Perspectives
- Publisher
- Greater Baltimore Medical Center
- Published
- 2013-01
- Volume
- 3
- Issue
- 3-4
- Pages
- 21362
- ISSN
- 2000-9666 (ISSN-L)
- PMC
- PMC3879511
- OpenAlex
- W2020329253
- Cited by
- 23
- References
- 22
- Citations by year
- 2026: 1 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 2 · 2019: 2 · 2018: 1 · 2017: 1 · 2016: 5 · 2015: 4 · 2013: 1
- FWCI
- 3.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- Journal of Community Hospital Internal Medicine Perspectives (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Gender Studies › Diversity and Career in Medicine
- Keywords
- multiple mini-interviews, residency program recruitment, resident selection, internal medicine residency, non-cognitive skills, interview reliability, interviewer bias, generalizability, interpersonal and communication skills
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
