Cutting costs of multiple mini-interviews–changes in reliability and efficiency of the Hamburg medical school admission test between two applications
Open accessAbstract
BACKGROUND: Multiple mini-interviews (MMIs) are a valuable tool in medical school selection due to their broad acceptance and promising psychometric properties. With respect to the high expenses associated with this procedure, the discussion about its feasibility should be extended to cost-effectiveness issues. METHODS: Following a pilot test of MMIs for medical school admission at Hamburg University in 2009 (HAM-Int), we took several actions to improve reliability and to reduce costs of the subsequent procedure in 2010. For both years, we assessed overall and inter-rater reliabilities based on multilevel analyses. Moreover, we provide a detailed specification of costs, as well as an extrapolation of the interrelation of costs, reliability, and the setup of the procedure. RESULTS: The overall reliability of the initial 2009 HAM-Int procedure with twelve stations and an average of 2.33 raters per station was ICC=0.75. Following the improvement actions, in 2010 the ICC remained stable at 0.76, despite the reduction of the process to nine stations and 2.17 raters per station. Moreover, costs were cut down from $915 to $495 per candidate. With the 2010 modalities, we could have reached an ICC of 0.80 with 16 single rater stations ($570 per candidate). CONCLUSIONS: With respect to reliability and cost-efficiency, it is generally worthwhile to invest in scoring, rater training and scenario development. Moreover, it is more beneficial to increase the number of stations instead of raters within stations. However, if we want to achieve more than 80 % reliability, a minor improvement is paid with skyrocketing costs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Hissbach et al. (2014), with the 2010 modalities, an ICC of 0.80 could have been reached with 16 single rater stations ($570 per candidate).
“With the 2010 modalities, we could have reached an ICC of 0.80 with 16 single rater stations ($570 per candidate).”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Hissbach J. C. et al., (52) 2014
In Hissbach et al. (2014), costs were cut from $915 to $495 per candidate.
“Costs were cut down from $915 to $495 per candidate.”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Hissbach J. C. et al., (52) 2014
In Hissbach et al. (2014), in 2010 the ICC remained stable at 0.76, despite reduction to nine stations and 2.17 raters per station.
“In 2010 the ICC remained stable at 0.76, despite the reduction of the process to nine stations and 2.17 raters per station”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Hissbach J. C. et al., (52) 2014
In Hissbach et al. (2014), the mean inter-station correlation was 0.20 in 2009 (min −0.11; max 0.44) and 0.19 in 2010 (min 0.07; max 0.32).
“The mean inter-station correlation in 2009 was 0.20 (min: −0.11; max: 0.44) and 0.19 in 2010 (min: 0.07; max: 0.32)”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Hissbach J. C. et al., (52) 2014
In Hissbach et al. (2014), the overall reliability of the initial 2009 HAM-Int procedure with 12 stations and an average of 2.33 raters per station was ICC=0.75.
“The overall reliability of the initial 2009 HAM-Int procedure with 12 station and an average of 2.33 raters per station was ICC = 0.75”
From Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review (2019) · cites this paper as Hissbach J. C. et al., (52) 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Multilevel and Longitudinal Modeling Using Stata
Rabe-Hesketh, S., & Skrondal, A. (2005) · Biometrics · doi:10.1111/j.1541-0420.2006.00588_16.x · Cited by 4,626 · see citing works
Brennan, R. L. (2001) · doi:10.1007/978-1-4757-3456-0 · Cited by 1,121 · see citing works
An admissions OSCE: the multiple mini-interview
Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004) · Medical Education · doi:10.1046/j.1365-2923.2004.01776.x · Cited by 617 · see citing works
Donnon, T., Paolucci, E. O., & Violato, C. (2006) · Academic Medicine · doi:10.1097/01.acm.0000249878.25186.b7 · Cited by 294 · see citing works
Predictive validity of the multiple mini-interview for selecting medical trainees
Eva, K. W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G. R. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03407.x · Cited by 267 · see citing works
Trapmann, S., Hell, B., Weigand, S., & Schuler, H. (2007) · Zeitschrift für Pädagogische Psychologie · doi:10.1024/1010-0652.21.1.11 · Cited by 213 · see citing works
Generalizability Theory and Classical Test Theory
Brennan, R. L. (2010) · Applied Measurement in Education · doi:10.1080/08957347.2011.532417 · Cited by 204 · see citing works
Multilevel and longitudinal modeling using Stata. 2nd edition
Rabe‐Hesketh, S., & Skrondal, A. (2008) · Cited by 199 · see citing works
Pau, A., Jeevaratnam, K., Chen, Y. S., Fall, A. A., Khoo, C., & Nadarajah, V. D. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.829912 · Cited by 195 · see citing works
Eva, K. W., Reiter, H., Rosenfeld, J. M., & Norman, G. R. (2004) · Academic Medicine · doi:10.1097/00001888-200406000-00021 · Cited by 112 · see citing works
Admissions processes for five year medical courses at English schools: review
Parry, J. M., Mathers, J. M., Stevens, A. J., Parsons, A., Lilford, R. J., Spurgeon, P. C., & Thomas, H. (2006) · BMJ · doi:10.1136/bmj.38768.590174.55 · Cited by 101 · see citing works
Rosenfeld, J. M., Reiter, H. I., Trinh, K., & Eva, K. W. (2008) · Advances in Health Sciences Education · doi:10.1007/s10459-006-9029-z · Cited by 88 · see citing works
Ziv, A., Rubin, O., Moshinsky, A., Gafni, N., Kotler, M., Dagan, Y., Lichtenberg, D. A., Mekori, Y. A., & Mittelman, M. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03161.x · Cited by 73 · see citing works
Brownell, K., Lockyer, J., Collin, T., & Lemay, J.-F. (2007) · Medical Teacher · doi:10.1080/01421590701311713 · Cited by 72 · see citing works
Internet-based multiple mini-interviews for candidate selection for graduate entry programmes
Tiller, D., O'Mara, D., Rothnie, I., Dunn, S., Lee, L., & Roberts, C. (2013) · Medical Education · doi:10.1111/medu.12224 · Cited by 61 · see citing works
Kreiter, C. D., & Kreiter, Y. (2007) · Teaching and Learning in Medicine · doi:10.1080/10401330701332094 · Cited by 61 · see citing works
Roberts, C., Rothnie, I., Zoanetti, N., & Crossley, J. (2010) · Medical Education · doi:10.1111/j.1365-2923.2010.03689.x · Cited by 59 · see citing works
Uijtdehaage, S., Doyle, L. “., & Parker, N. (2011) · Academic Medicine · doi:10.1097/acm.0b013e3182223ab7 · Cited by 57 · see citing works
A comparison of multiple mini-interviews and structured interviews in a UK setting
O’Brien, A., Harvey, J., Shannon, M., Lewis, K., & Valencia, O. (2011) · Medical Teacher · doi:10.3109/0142159x.2010.541532 · Cited by 55 · see citing works
Hissbach, J. C., Klusmann, D., & Hampe, W. (2011) · BMC Medical Education · doi:10.1186/1472-6920-11-83 · Cited by 51 · see citing works
Self-citationThe multiple mini‐interview: how long is long enough?
Michael, D., Crotty, B. J., Prideaux, D. J., Carne, R. P., Ward, A. C., & de Leeuw, E. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03260.x · Cited by 45 · see citing works
Rater and occasion impacts on the reliability of pre‐admission assessments
Axelson, R. D., & Kreiter, C. D. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03537.x · Cited by 35 · see citing works
Resident Selection for a Physical Medicine and Rehabilitation Program
Finlayson, H. C., & Townson, A. F. (2011) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e31820f9677 · Cited by 34 · see citing works
Reliability estimates: behavioural stations and questionnaires in medical school admissions
Gafni, N., Moshinsky, A., Eisenberg, O., Zeigler, D., & Ziv, A. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04155.x · Cited by 33 · see citing works
Hissbach, J. C., Klusmann, D., & Hampe, W. (2011) · Europe PMC (PubMed Central) · doi:10.3205/zma000756 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
1 of the 26 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Hissbach J, Klusmann D, Hampe W: Reliability of a science admission test (HAM-Nat) at Hamburg medical school. GMS Z Med Ausbild. 2011, 28: Doc44.
Cited by
Sorted by citations
How effective are selection methods in medical education? A systematic review
Patterson, F., Knight, A., Dowell, J. S., Nicholson, S., Cousans, F., & Cleland, J. (2015) · Medical Education · doi:10.1111/medu.12817 · Cited by 489 · see citing works
Medizinstudierendenauswahl in Deutschland
Schwibbe, A., Lackamp, J., Knorr, M., Hissbach, J. C., Kadmon, M., & Hampe, W. (2018) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-017-2670-2 · Cited by 183 · see citing works
Self-citationCallwood, A., Jeevaratnam, K., Kotronoulas, G., Schneider, A., Lewis, L. S., & Nadarajah, V. D. (2018) · Nurse Education Today · doi:10.1016/j.nedt.2018.01.016 · Cited by 174 · see citing works
Reliability and validity of new online selection tests for midwifery students
Groene, O. R., Knorr, M., Vogel, D., Hild, C., & Hampe, W. (2021) · Midwifery · doi:10.1016/j.midw.2021.103245 · Cited by 153 · see citing works
Self-citationMultiple mini-interviews: same concept, different approaches
Knorr, M., & Hissbach, J. (2014) · Medical Education · doi:10.1111/medu.12535 · Cited by 110 · see citing works
Self-citationRoberts, C., Khanna, P., Rigby, L., Bartle, E. K., Llewellyn, A., Gustavs, J., Newton, L., Newcombe, J. P., Davies, M. W., Thistlethwaite, J. E., et al. (2017) · Medical Teacher · doi:10.1080/0142159x.2017.1367375 · Cited by 64 · see citing works
Is perfect good? – Dimensions of perfectionism in newly admitted medical students
Seeliger, H., & Harendza, S. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1034-9 · Cited by 58 · see citing works
Self-citationMultiple Mini Interview as an admission tool in higher education: Insights from a systematic review
Yusoff, M. S. B. (2019) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2019.03.006 · Cited by 57 · see citing works
“, (36) 2012 Observational Study design University of California, Davis (UCD), California N 1⁄4 444 candidates School of Medicine Examine relationship among applicant personality, MMI performance and medical school acceptance offers Those with extraversion scores (among 444 candidates) in the top (versus bottom) quarti…”
BackgroundValidity evidence for the Hamburg multiple mini-interview
Knorr, M., Schwibbe, A., Ehrhardt, M., Lackamp, J., Zimmermann, S., & Hampe, W. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1208-0 · Cited by 37 · see citing works
Self-citationCOVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape
Cleland, J., Chu, J., Lim, S. Y. E., Low, J., Low‐Beer, N., & Kwek, T. K. (2020) · Medical Teacher · doi:10.1080/0142159x.2020.1762851 · Cited by 36 · see citing works
<p>Multiple Mini-Interviews: Current Perspectives on Utility and Limitations</p>
Ali, S., Sadiq Hashmi, M. S., Umair, M., Beg, M. A., & Huda, N. (2019) · Advances in Medical Education and Practice · doi:10.2147/amep.s181332 · Cited by 32 · see citing works
The relationship between perfectionism and symptoms of depression in medical school applicants
Bußenius, L., & Harendza, S. (2019) · BMC Medical Education · doi:10.1186/s12909-019-1823-4 · Cited by 31 · see citing works
Self-citationSelection and academic success of medical students in Hamburg, Germany
Meyer, H., Zimmermann, S., Hissbach, J. C., Klusmann, D., & Hampe, W. (2019) · BMC Medical Education · doi:10.1186/s12909-018-1443-4 · Cited by 31 · see citing works
Self-citationCorrecting the predictive validity of a selection test for the effect of indirect range restriction
Zimmermann, S., Klusmann, D., & Hampe, W. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1070-5 · Cited by 30 · see citing works
Self-citationJerant, A. F., Henderson, M. C., Griffin, E., Rainwater, J. A., Hall, T. R., Kelly, C., Peterson, E. M., Wofsy, D., & Franks, P. J. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1030-0 · Cited by 30 · see citing works
Does selection pay off? A cost–benefit comparison of medical school selection and lottery systems
Schreurs, S., Cleland, J., Muijtjens, A. M. M., Egbrink, M. G. A. O., & Cleutjens, K. B. J. M. (2018) · Medical Education · doi:10.1111/medu.13698 · Cited by 28 · see citing works
Best practices for interviewing applicants for medical school admissions: a systematic review
Lin, J. C. H., Lokhande, A., Margo, C. E., & Greenberg, P. B. (2022) · Perspectives on Medical Education · doi:10.1007/s40037-022-00726-8 · Cited by 27 · see citing works
Need for cognitive closure, tolerance for ambiguity, and perfectionism in medical school applicants
Gärtner, J., Bußenius, L., Prediger, S., Vogel, D., & Harendza, S. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02043-2 · Cited by 27 · see citing works
Self-citationLee, H. J., Park, S. B., Park, S. C., Park, W. S., Ryu, S.-W., Yang, J. H., Na, S., Won, J. Y., & Chae, G. (2016) · BMC Research Notes · doi:10.1186/s13104-016-1866-0 · Cited by 26 · see citing works
Cleland, J., Foo, J., Ilić, D., Maloney, S. R., & You, Y. (2020) · Advances in Health Sciences Education · doi:10.1007/s10459-020-10007-w · Cited by 19 · see citing works
Knorr, M., Meyer, H., Sehner, S., Hampe, W., & Zimmermann, S. (2019) · BMC Medical Education · doi:10.1186/s12909-019-1674-z · Cited by 18 · see citing works
Self-citationBußenius, L., & Harendza, S. (2021) · BMC Medical Education · doi:10.1186/s12909-021-02703-x · Cited by 16 · see citing works
Self-citationAssociations between communication OSCE performance and admission interviews in medical education
Groene, O. R., Mielke, I., Knorr, M., Ehrhardt, M., & Bergelt, C. (2021) · Patient Education and Counseling · doi:10.1016/j.pec.2021.11.005 · Cited by 14 · see citing works
Modellstudiengänge in der Medizin
Guse, A. H., & Kuhlmey, A. (2017) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-017-2678-7 · Cited by 13 · see citing works
Comparison of open and closed book test for admission in medical school
Cade, C., Riou, J., Richard, I., Passirani, C., Letertre, E., & Fayolle, A.-V. (2018) · MedEdPublish · doi:10.15694/mep.2018.0000025.1 · Cited by 7 · see citing works
Vogel, D., Seeliger, H., & Harendza, S. (2019) · Europe PMC (PubMed Central) · doi:10.3205/zma001252 · Cited by 7 · see citing works
Self-citationExploring the relationships between first impressions and MMI ratings: a pilot study
Klusmann, D., Knorr, M., & Hampe, W. (2022) · Advances in Health Sciences Education · doi:10.1007/s10459-022-10151-5 · Cited by 4 · see citing works
Self-citationInterviews, Multiple Mini-Interviews, and Selection Centers
Knorr, M., Hissbach, J. C., & Hampe, W. (2018) · Springer eBooks · doi:10.1007/978-3-319-94971-0_5 · Cited by 4 · see citing works
Self-citationDeveloping and evaluating the multiple mini interview in student midwife selection
Callwood, A. (2015) · Surrey Research Insight Open Access (The University of Surrey) · Cited by 4 · see citing works
Angleichung von Schulnoten für die Studierendenauswahl
Zimmermann, R., Klusmann, & Hampe (2018) · Zeitschrift für Hochschulentwicklung · doi:10.3217/zfhe-13-04/11 · Cited by 3 · see citing works
Non-technical attributes in paramedicine: Is situational judgement testing the solution?
Bennett, R., & Williams, B. A. (2019) · Irish Journal of Paramedicine · doi:10.32378/ijp.v4i1.171 · Cited by 3 · see citing works
Interactional competencies in medical student admission– what makes a “good medical doctor”?
Fleck, L., Amelung, D., Fuchs, A., Mayer, B., Escher, M., Listunova, L., Schultz, J.-H., Möltner, A., Schütte, C., Wittenberg, T., et al. (2024) · Advances in Health Sciences Education · doi:10.1007/s10459-024-10348-w · Cited by 2 · see citing works
Knorr, M., Mielke, I., Amelung, D., Safari, M., Groene, O. R., Breil, S. M., & MacIntosh, A. (2024) · PubMed Central · doi:10.3205/zma001685 · Cited by 2 · see citing works
Zaccagnini, M. (2019) · Canadian Journal of Respiratory Therapy · doi:10.29390/cjrt-2019-002 · Cited by 1 · see citing works
Waheed, G., Zia, S., Bhatti, Y. A., Zafar, M., & Aziz, M. S. (2021) · Pakistan Journal of Medical and Health Sciences · doi:10.53350/pjmhs2115123294 · Cited by 1 · see citing works
Alam, F., Az-Zahraa, F., Idris, F. I., & Rahman, H. A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0314778 · Cited by 1 · see citing works
Utilizing Multiple Mini Interviews in Medical School Admissions
Apple, R. W. (2017) · Journal of Psychology & Clinical Psychiatry · doi:10.15406/jpcpy.2017.07.00462 · Cited by 1 · see citing works
- Authors
- Hissbach, J. C., Sehner, S., Harendza, S., & Hampe, W.
- Affiliations
- University Medical Center Hamburg-Eppendorf (DE)
- Corresponding author
- Hampe, Wolfgang
- Venue
- BMC Medical Education
- Published
- 2014-03-19
- Volume
- 14
- Issue
- 1
- Pages
- 54-54
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC3995077
- OpenAlex
- W2038227228
- Semantic Scholar
- 5e2bfc24aaa716e82b85c75ea9bf21e507d9836e
- Cited by
- 37
- References
- 26
- Citations by year
- 2024: 3 · 2022: 2 · 2021: 4 · 2020: 3 · 2019: 8 · 2018: 7 · 2017: 6 · 2016: 1 · 2015: 2 · 2014: 1
- FWCI
- 3.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Gender Studies › Diversity and Career in Medicine
- Keywords
- multiple mini-interviews, medical school admissions, cost-effectiveness, inter-rater reliability, rater training, scenario development
- MeSH terms
- Cost-Benefit Analysis, Germany, Interviews as Topic, School Admission Criteria, Schools, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
