A laboratory study on the reliability estimations of the mini-CEX
Open accessAbstract
Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent. This is difficult to achieve. Furthermore, the variance caused by the case/patient or by assessor is completely confounded. We have no idea how much each of these factors contribute to the noise in the measurement. The aim of this study was to use a controlled setup that overcomes these difficulties and to estimate the reproducibility of the mini-CEX. Three encounters were videotaped from 21 residents. The patients were the same for all residents. Each encounter was assessed by 3 assessors who assessed all encounters for all residents. This delivered a fully crossed (all random) two-facet generalizability design. A quarter of the total variance was associated with universe score variance (28%). The largest source of variance was the general error term (34%) followed by the main effect of assessors (18%). Generalizability coefficients indicated that an approximate sample of 9 encounters was needed assuming a single different assessor per encounter and assuming different cases per encounter (the usual situation in real practice), 4 encounters when 2 raters were used and 3 encounters when 3 raters are used. Unexplained general error and the leniency/stringency of assessors are the major causes for unreliability in mini-CEX. To optimize reliability rater training might have an effect.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Supervisors differ considerably in the feedback they give after reviewing the same videotaped consultation.
“Supervisors differ considerably in the feedback they give after reviewing the same videotaped consultation [80].”
From Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education. (2015) · cites this paper as [80]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Mini-CEX: A Method for Assessing Clinical Skills
Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-6-200303180-00012 · Cited by 730 · see citing works
Tools for Direct Observation and Assessment of Clinical Skills of Medical Trainees
Kogan, J. R., Holmboe, E. S., & Hauer, K. E. (2009) · JAMA · doi:10.1001/jama.2009.1365 · Cited by 602 · see citing works
The assessment of professional competence: building blocks for theory development
van der Vleuten, C. P. M., Schuwirth, L., Scheele, F., Driessen, E. W., & Hodges, B. D. (2010) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2010.04.001 · Cited by 341 · see citing works
Cook, K., Kallen, M., & Amtmann, D. (2009) · Quality of Life Research · doi:10.1007/s11136-009-9464-4 · Cited by 300 · see citing works
Opening the black box of clinical skills assessment via observation: a conceptual model
Kogan, J. R., Conforti, L. N., Bernabeo, E. C., Iobst, W. F., & Holmboe, E. S. (2011) · Medical Education · doi:10.1111/j.1365-2923.2011.04025.x · Cited by 235 · see citing works
Durning, S. J., Cation, L. J., Markert, R. J., & Pangaro, L. N. (2002) · Academic Medicine · doi:10.1097/00001888-200209000-00020 · Cited by 185 · see citing works
Generalisability: a key to unlock professional assessment
Crossley, J., Davies, H., Humphris, G., & Jolly, B. (2002) · Medical Education · doi:10.1046/j.1365-2923.2002.01320.x · Cited by 169 · see citing works
In-training assessment using direct observation of single-patient encounters: a literature review
Pelgrim, E. A. M., Kramer, A. W. M., Mokkink, H. G. A., van den Elsen, L., Grol, R. P. T. M., & van der Vleuten, C. P. M. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9235-6 · Cited by 141 · see citing works
Mini-clinical evaluation exercise in anaesthesia training
Weller, J. M., Jolly, B. C., Misur, M., Merry, A., Jones, A., Crossley, J., Pedersen, K., & Smith, K. (2009) · British Journal of Anaesthesia · doi:10.1093/bja/aep055 · Cited by 124 · see citing works
de Lima, A. A., Barrero, C., Baratta, S., Costa, Y. C., Bortman, G., Carabajales, J., Conde, D. G., Galli, A., Degrange, G., & van der Vleuten, C. P. M. (2007) · Medical Teacher · doi:10.1080/01421590701352261 · Cited by 98 · see citing works
Self-citationCook, D. A., Beckman, T. J., Mandrekar, J. N., & Pankratz, V. S. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9224-9 · Cited by 80 · see citing works
Implementing the undergraduate mini‐CEX: a tailored approach at Southampton University
Hill, F. J., Kendall, K., Galbraith, K., & Crossley, J. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03275.x · Cited by 79 · see citing works
Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., & Hawkins, R. E. (2006) · Academic Medicine · doi:10.1097/01.acm.0000236514.53194.f4 · Cited by 70 · see citing works
Flagged first, then by citations
1 of the 14 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Guidelines: The do’s, don’ts and don’t knows of feedback for clinical education.
Lefroy, J., Watling, C., Teunissen, P. W., & Brand, P. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0231-7 · Cited by 347 · see citing works
Seeing the same thing differently
Yeates, P., O'Neill, P. A., Mann, K. V., & Eva, K. W. (2012) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9372-1 · Cited by 185 · see citing works
Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem
Kogan, J. R., Conforti, L. N., Iobst, W. F., & Holmboe, E. S. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000221 · Cited by 148 · see citing works
Hejri, S. M., Jalili, M., Masoumi, R., Shirazi, M., Nedjat, S., & Norcini, J. J. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1652732 · Cited by 69 · see citing works
Barrett, A., Galvin, R., Steinert, Y., Scherpbier, A. J. J. A., O’Shaughnessy, A., Horgan, M. M., & Horsley, T. (2016) · Medical Teacher · doi:10.1080/0142159x.2016.1215413 · Cited by 50 · see citing works
Relatively speaking: contrast effects influence assessors’ scores and narrative feedback
Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015) · Medical Education · doi:10.1111/medu.12777 · Cited by 34 · see citing works
A situated cognition model for clinical reasoning performance assessment: a narrative review
Rencic, J. J., Schuwirth, L., Gruppen, L. D., & Durning, S. J. (2020) · Diagnosis · doi:10.1515/dx-2019-0106 · Cited by 26 · see citing works
A mobile application to facilitate implementation of programmatic assessment in anaesthesia training
Marty, A., Braun, J., Schick, C., Zalunardo, M. P., Spahn, D. R., & Breckwoldt, J. (2022) · British Journal of Anaesthesia · doi:10.1016/j.bja.2022.02.038 · Cited by 27 · see citing works
Love, J., Doty, C., Smith, J., Deiorio, N. M., Jordan, J., Meter, M. V., Edens, M. A., & Hegarty, C. (2020) · Western Journal of Emergency Medicine · doi:10.5811/westjem.2020.3.45077 · Cited by 24 · see citing works
Rogausch, A., Beyeler, C., Montagne, S., Jucker-Kupper, P., Berendonk, C., Huwendiek, S., Gemperli, A., & Himmel, W. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0490-3 · Cited by 23 · see citing works
The reliability and sources of error of using rubrics-based assessment for student projects
Varela, J. L. M., & Giralt, E. G. (2017) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2017.1360838 · Cited by 23 · see citing works
Duijn, C. C. M. A., van Dijk, E. J., Mándoki, M., Bok, H. G. J., & ten Cate, O. (2019) · Journal of Veterinary Medical Education · doi:10.3138/jvme.0917-123r · Cited by 21 · see citing works
Assessment toolbox for Indian medical graduate competencies
Singh, T., Saiyad, S. N., Virk, A., Kalra, J., & Mahajan, R. (2021) · Journal of Postgraduate Medicine · doi:10.4103/jpgm.jpgm_1260_20 · Cited by 20 · see citing works
Chang, Y., Lee, C.-H., Chen, C.-K., Liao, C., Ng, C., Chen, J.-C., & Chaou, C. (2016) · Advances in Health Sciences Education · doi:10.1007/s10459-016-9682-9 · Cited by 16 · see citing works
Al‐Jewair, T., & Kumar, S. R. (2019) · Journal of Dental Education · doi:10.21815/jde.019.131 · Cited by 15 · see citing works
Clinical assessment of transthoracic echocardiography skills: a generalizability study
Nielsen, D. G., Jensen, S. L., & O’Neill, L. D. (2015) · BMC Medical Education · doi:10.1186/s12909-015-0294-5 · Cited by 15 · see citing works
Yeates, P., Moult, A., Cope, N. A., McCray, G., Goddard‐Fuller, R., & McKinley, R. K. (2021) · Medical Education · doi:10.1111/medu.14713 · Cited by 13 · see citing works
Vaughan, B., & Moore, K. M. (2015) · International journal of osteopathic medicine · doi:10.1016/j.ijosm.2015.07.002 · Cited by 13 · see citing works
Schlair, S., Dyche, L., & Milan, F. B. (2017) · MedEdPORTAL · doi:10.15766/mep_2374-8265.10648 · Cited by 12 · see citing works
Lertsakulbunlue, S., & Kantiwong, A. (2025) · BMC Medical Education · doi:10.1186/s12909-025-06772-0 · Cited by 16 · see citing works
Making the horse drink: use of mini-CEX in an assessment for learning view
Schuwirth, L. (2012) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9423-7 · Cited by 11 · see citing works
Assessing the Validity of a Multidisciplinary Mini-Clinical Evaluation Exercise
Humphrey-Murto, S. M., Côté, M., Pugh, D., & Wood, T. (2017) · Teaching and Learning in Medicine · doi:10.1080/10401334.2017.1387553 · Cited by 9 · see citing works
Current Trends in Mini-Clinical Evaluation Exercise in Medical Education: A Bibliometric Analysis
Sharma, R., Gupta, T., Haidery, T. H., Sinha, S., & Kumar, A. (2022) · Cureus · doi:10.7759/cureus.33121 · Cited by 7 · see citing works
de Lima, A. A. (2013) · doi:10.26481/dis.20131023aa · Cited by 6 · see citing works
Guo, H., Li, H., Yao, Y., Li, Y., & Huang, Y. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1561504 · Cited by 5 · see citing works
Framework for an effective assessment: From rocky roads to silk route
Siddiqui, Z. S. (2017) · Pakistan Journal of Medical Sciences · doi:10.12669/pjms.332.12334 · Cited by 6 · see citing works
Dewhirst, S., Cheung, W. J., Wood, T. J., Szabo, N. D., & Frank, J. R. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005802 · Cited by 4 · see citing works
Hemming, P., Levine, R. B., & Gallo, J. J. (2017) · Patient Education and Counseling · doi:10.1016/j.pec.2017.07.014 · Cited by 2 · see citing works
Filho, L. E. C. D. S., Cunha, K. D. C., Santos, M. C. D. S., Falcão, L. F. M., & Normando, V. M. F. (2021) · Revista Eletrônica Acervo Saúde · doi:10.25248/reas.e7428.2021
Residencias médicas del Hospital Británico de Buenos Aires: pasado, presente y futuro
Young, P. (2022) · Fronteras en Medicina · doi:10.31954/rfem/2022001/0053-0069
Rajashree, R., ., S. S. K., Jambale, T., & Channashetti, G. (2022) · BLDE University Journal of Health Sciences · doi:10.4103/bjhs.bjhs_9_22
Residencias médicas del Hospital Británico de Buenos Aires: pasado, presente y futuro
Young, P. (2022) · Fronteras en Medicina · doi:10.31954/rfem/202201/0053-0069
Evaluation par le miniclinical évaluation exercise (Mini-CEX) en réanimation
Ghezala, H. B., Benzarti, A., Jazia, A. B., & Brahmi, N. (2024) · La Tunisie Médicale · doi:10.62438/tunismed.v102i5.4773
- Authors
- Alves de Lima, A., Conde, D., Costabel, J., Corso, J., & Van der Vleuten, C.
- Affiliations
- Instituto Cardiovascular de Buenos Aires (AR)
- Venue
- Advances in Health Sciences Education
- Published
- 2011-12-23
- Volume
- 18
- Issue
- 1
- Pages
- 5-13
- ISSN
- 1382-4996 (ISSN-L)
- 1573-1677
- PMC
- PMC3569586
- OpenAlex
- W1994615095
- Cited by
- 37
- References
- 14
- Citations by year
- 2025: 3 · 2024: 2 · 2022: 7 · 2021: 3 · 2020: 2 · 2019: 3 · 2017: 5 · 2016: 3 · 2015: 5 · 2014: 1 · 2013: 1 · 2012: 2
- FWCI
- 1.26
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Advances in Health Sciences Education (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, cc-by-nc)
- Radboud Repository (Radboud University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Mini-CEX, workplace-based assessment, reliability statistics, generalizability theory, rater variability, local independence, sample size determination, rater training, medical residents
- MeSH terms
- Argentina, Cardiology, Clinical Competence, Education, Medical, Graduate, Educational Measurement, Female, Humans, Male, Medical Staff, Hospital, Videotape Recording, Reproducibility of Results, Patient Simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
