Exploring examinee behaviours as validity evidence for multiple‐choice question examinations
Abstract
Context: Clinical‐vignette multiple choice question ( MCQ ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical‐vignette MCQ items are like the processes thought to occur with ‘real‐world’ clinical reasoning as theorised by dual‐process theory. Methods: Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self‐Assessment Program ( MKSAP ). Upon answering a set of MCQ s, each participant completed a retrospective think aloud ( TA ) protocol. Using constant comparative analysis ( CCA ) methods sensitised by dual‐process theory, we performed a qualitative thematic analysis. Results: Examinee behaviours fell into three categories: clinical reasoning behaviours, test‐taking behaviours and reactions to the MCQ . Consistent with dual‐process theory, statements about clinical reasoning behaviours were divided into two sub‐categories: analytical reasoning and non‐analytical reasoning. Each of these categories included several themes. Conclusions: Our study provides some validity evidence that test‐takers’ descriptions of their cognitive processes during completion of high‐quality clinical‐vignette MCQ s align with processes expected in real‐world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQ s elicit other cognitive processes, including certain test‐taking behaviours, that seem ‘inauthentic’ to real‐world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ ‐based assessments.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When responding to high-quality clinical vignette-based MCQs, examinees engage cognitive processes equivalent to those involved in authentic clinical reasoning.
“Prior research indicates that when responding to high-quality clinical vignette–based MCQs, examinees engage cognitive processes equivalent to those involved in authentic clinical reasoning36,37.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [37]
Cites
Lincoln, Y. S., Guba, E. G., & Pilotta, J. J. (1985) · International Journal of Intercultural Relations · doi:10.1016/0147-1767(85)90062-8 · Cited by 45,787 · see citing works
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449 · see citing works
Basics of Qualitative Research: Grounded Theory Procedures and Techniques
Long, D. R., Strauss, A., & Corbin, J. M. (1993) · Modern Language Journal · doi:10.2307/328955 · Cited by 17,612 · see citing works
Dual-Process Theories of Higher Cognition: Advancing the Debate
Evans, J. S. B. T., & Stanovich, K. E. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691612460685 · Cited by 4,271 · see citing works
Validating the interpretations and uses of test scores
Kane, M. T. (2013) · Journal of Educational Measurement · doi:10.1111/jedm.12000 · Cited by 2,110 · see citing works
A Universal Model of Diagnostic Reasoning
Croskerry, P. (2009) · Academic Medicine · doi:10.1097/acm.0b013e3181ace703 · Cited by 1,016 · see citing works
Stanovich, K. E. (1999) · Psychology Press eBooks · doi:10.4324/9781410603432 · Cited by 956 · see citing works
The Constant Comparative Analysis Method Outside of Grounded Theory
Fram, S. M. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2013.1569 · Cited by 673 · see citing works
A contemporary approach to validity arguments: a practical guide to K ane's framework
Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. A. (2015) · Medical Education · doi:10.1111/medu.12678 · Cited by 644 · see citing works
Grounded theory in medical education research: AMEE Guide No. 70
Watling, C., & Lingard, L. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.704439 · Cited by 573 · see citing works
Clinical cognition and diagnostic error: applications of a dual process model of reasoning
Croskerry, P. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9182-2 · Cited by 494 · see citing works
Constant Comparison Method: A Kaleidoscope of Data
Dye, J., Schatz, I. M., Rosenberg, B. C., & Coleman, S. (2000) · The Qualitative Report · doi:10.46743/2160-3715/2000.2090 · Cited by 453 · see citing works
Ericsson, K. A. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816796.013 · Cited by 446 · see citing works
Improving the fairness of multiple-choice questions: a literature review
McCoubrie, P. (2004) · Medical Teacher · doi:10.1080/01421590400013495 · Cited by 318 · see citing works
Gut Feelings as a Third Track in General Practitioners’ Diagnostic Reasoning
Stolper, E. C. F., van de Wiel, M. W. J., Royen, P. V., van Bokhoven, M. A., van der Weijden, T., & Dinant, G. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1524-5 · Cited by 266 · see citing works
Papadakis, M. A., Arnold, G. K., Blank, L. L., Holmboe, E. S., & Lipner, R. S. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-11-200806030-00009 · Cited by 261 · see citing works
Context and clinical reasoning: understanding the perspective of the expert’s voice
Durning, S. J., Artino, A. R., Pangaro, L. N., van der Vleuten, C. P. M., & Schuwirth, L. (2011) · Medical Education · doi:10.1111/j.1365-2923.2011.04053.x · Cited by 206 · see citing works
Self-citationDurning, S. J., Artino, A. R., Boulet, J. R., et al. (2011) · Advances in Health Sciences Education · doi:10.1007/s10459-011-9294-3 · Cited by 147 · see citing works
Self-citationAlthouse, L. A., & McGuinness, G. A. (2008) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2008.03.012 · Cited by 74 · see citing works
Waxman, H. S., Braunstein, G. D., Dantzker, D. R., et al. (1994) · Journal of General Internal Medicine · doi:10.1007/bf02599012 · Cited by 67 · see citing works
Perez, J. A., & Greer, S. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9158-2 · Cited by 67 · see citing works
The Predictive Validity of the Internal Medicine In-Training Examination
Babbott, S. F., Beasley, B. W., Hinchey, K. T., Blotzer, J. W., & Holmboe, E. S. (2007) · The American Journal of Medicine · doi:10.1016/j.amjmed.2007.05.003 · Cited by 64 · see citing works
Durning, S. J., Artino, A. R., Beckman, T. J., Graner, J. L., van der Vleuten, C. P. M., Holmboe, E. S., & Schuwirth, L. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.801938 · Cited by 63 · see citing works
Self-citationPredicting pass rates on the american board of internal medicine certifying examination
Rollins, L. K., Martindale, J. R., Edmond, M. B., et al. (1998) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1998.00122.x · Cited by 58 · see citing works
Consequences of contextual factors on clinical reasoning in resident physicians
McBee, E. C., Ratcliffe, T. A., Picho, K., Artino, A. R., Schuwirth, L., Kelly, W. F., Masel, J., van der Vleuten, C. P. M., & Durning, S. J. (2015) · Advances in Health Sciences Education · doi:10.1007/s10459-015-9597-x · Cited by 58 · see citing works
Self-citationDurning, S. J., Dong, T., Artino, A. R., van der Vleuten, C. P. M., Holmboe, E. S., & Schuwirth, L. (2015) · Perspectives on Medical Education · doi:10.1007/s40037-015-0196-6 · Cited by 46 · see citing works
Self-citationA nomogram to predict the probability of passing the American Board of Internal Medicine examination
Brateanu, A., Yu, C., Kattan, M. W., Olender, J., & Nielsen, C. D. (2012) · Medical Education Online · doi:10.3402/meo.v17i0.18810 · Cited by 42 · see citing works
Heist, B. S., Gonzalo, J. D., Durning, S., Torre, D., & Elnicki, D. M. (2014) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-14-00176.1 · Cited by 29 · see citing works
Self-citationLohr, K. M., Clauser, A. L., Hess, B., Gelber, A. C., Valeriano‐Marcet, J., Lipner, R. S., Haist, S. A., Hawley, J. L., Zirkle, S., & Bolster, M. B. (2015) · Arthritis & Rheumatology · doi:10.1002/art.39281 · Cited by 22 · see citing works
Are Commonly Used Premedical School or Medical School Measures Associated With Board Certification?
Durning, S. J., Dong, T., Hemmer, P. A., et al. (2015) · Military Medicine · doi:10.7205/milmed-d-14-00569 · Cited by 20 · see citing works
Self-citationMKSAP 14—Medical Knowledge Self-Assessment Program
Winter, F. D. (2007) · Baylor University Medical Center Proceedings · doi:10.1080/08998280.2007.11928313 · Cited by 13 · see citing works
Predicting performance on the American Board of Internal Medicine Certifying Examination
Grossman, R., Murata, G. H., Fincher, R. M. E., et al. (1996) · Academic Medicine · doi:10.1097/00001888-199610000-00049 · Cited by 12 · see citing works
Brill-Edwards, P. A., Evans, D. G., Hamilton, P., et al. (2001) · PubMed · Cited by 9 · see citing works
Sisson, S. D., Bertram, A. K., & Yeh, H. (2014) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-14-00054.1 · Cited by 8 · see citing works
MKSAP for students 2 : medical knowledge self-assessment program
Medcine, C. D. D. I. I. (2003) · American College of Physicians eBooks · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Song, X., Cleaves, E., Gluzman, E., Kotlyar, B., Russo, R. A., Schilling, D. C., Tsao, C. P., & West, J. C. (2025) · Academic Psychiatry · doi:10.1007/s40596-025-02136-4 · Cited by 6 · see citing works
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
“Our CAT system implemented a cyclic process with four core components demonstrated in Fig.”
- Authors
- Surry, L. T., Torre, D., & Durning, S. J.
- Affiliations
- Uniformed Services University of the Health Sciences (US)
- Corresponding author
- Surry, Luke T
- Venue
- Medical Education
- Publisher
- Wiley
- Published
- 2017-07-31
- Volume
- 51
- Issue
- 10
- Pages
- 1075-1085
- ISSN
- 0308-0110 (ISSN-L)
- 1365-2923
- OpenAlex
- W2740575825
- Cited by
- 40
- References
- 35
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 3 · 2023: 5 · 2022: 3 · 2021: 7 · 2020: 2 · 2019: 6 · 2018: 6
- FWCI
- 3.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Medical Education (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
- Keywords
- clinical vignettes, clinical reasoning, dual-process theory, think-aloud, validity evidence, medical licensing examinations, analytical thinking
- MeSH terms
- Behavior, Cognition, Education, Medical, Educational Measurement, Humans, Physical Examination, Students, Medical, Thinking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
