The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts
Open accessAbstract
Background: Pencil-and-paper examination formats, and specifically the standard, five-option multiple-choice question, have often been questioned as a means for assessing higher-order clinical reasoning or problem solving. This study firstly investigated whether two paper formats with differing number of alternatives (standard five-option and extended-matching questions) can test problem-solving abilities. Secondly, the impact of the alternatives number on psychometrics and problem-solving strategies was examined. Methods: Think-aloud protocols were collected to determine the problem-solving strategy used by experts and non-experts in answering Gastroenterology questions, across the two pencil-and-paper formats. Results: The two formats demonstrated equal ability in testing problem-solving abilities, while the number of alternatives did not significantly impact psychometrics or problem-solving strategies utilized. Conclusions: These results support the notion that well-constructed multiple-choice questions can in fact test higher order clinical reasoning. Furthermore, it can be concluded that in testing clinical reasoning, the question stem, or content, remains more important than the number of alternatives.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Well-designed MCQs can validly assess higher-order clinical reasoning.
“Well-designed MCQs can validly assess higher-order clinical reasoning35.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Medical Problem Solving: An Analysis of Clinical Reasoning
Elstein, A. S., Shulman, L. S., & Sprafka, S. A. (1978) · doi:10.4159/harvard.9780674189089 · Cited by 1,377 · see citing works
A cognitive perspective on medical expertise
Schmidt, H. G., Norman, G. R., & Boshuizen, H. P. A. (1990) · Academic Medicine · doi:10.1097/00001888-199010000-00001 · Cited by 1,135 · see citing works
Issues in cognitive psychology
Regehr, G., & Norman, G. R. (1996) · Academic Medicine · doi:10.1097/00001888-199609000-00015 · Cited by 396 · see citing works
Diagnostic reasoning strategies and diagnostic success
Coderre, S., Mandin, H., Harasym, P. H., & Fick, G. H. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01577.x · Cited by 359 · see citing works
Self-citationEffects of conventional and problem-based medical curricula on problem solving
Patel, V. L., Groen, G. J., & Norman, G. R. (1991) · Academic Medicine · doi:10.1097/00001888-199107000-00002 · Cited by 210 · see citing works
Helping students learn to think like experts when solving clinical problems
Mandin, H., Jones, A., Woloschuk, W., & Harasym, P. H. (1997) · Academic Medicine · doi:10.1097/00001888-199703000-00009 · Cited by 209 · see citing works
Self-citationDomain Knowledge and Hypothesis Genenation in Diagnostic Reasoning
Joseph, G.-M., & Patel, V. L. (1990) · Medical Decision Making · doi:10.1177/0272989x9001000107 · Cited by 182 · see citing works
Extended‐matching items: A practical alternative to free‐response questions
Case, S. M., & Swanson, D. B. (1993) · Teaching and Learning in Medicine · doi:10.1080/10401339309539601 · Cited by 171 · see citing works
Davison, G. C., Vogel, R. S., & Coffman, S. G. (1997) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.65.6.950 · Cited by 159 · see citing works
Cognitive differences in clinical reasoning related to postgraduate training
Norman, G. R., Trott, A. J., Brooks, L. R., Kinsey, E. E., & Smith, M. (1994) · Teaching and Learning in Medicine · doi:10.1080/10401339409539657 · Cited by 127 · see citing works
How can we test clinical reasoning?
van der Vleuten, C., & Newble, D. I. (1995) · The Lancet · doi:10.1016/s0140-6736(95)90763-7 · Cited by 111 · see citing works
A comparison of multiple‐choice tests and free‐response tests in examinations of clinical competence
Newble, D. I., BAXTER, A., & Elmslie, R. G. (1979) · Medical Education · doi:10.1111/j.1365-2923.1979.tb01511.x · Cited by 109 · see citing works
Differences between medical students and doctors in memory for clinical cases
Patel, V. L., Groen, G. J., & Frederiksen, C. H. (1986) · Medical Education · doi:10.1111/j.1365-2923.1986.tb01033.x · Cited by 104 · see citing works
Guidelines for assessing clinical competence
Newble, D. I., Dawson, B. K., Dauphinee, D. W., Page, G., Macdonald, M. M., Swanson, D. B., Mulholland, H., Thomson, A. N., & van der Vleuten, C. P. M. (1994) · Teaching and Learning in Medicine · doi:10.1080/10401339409539680 · Cited by 98 · see citing works
Beyond multiple-choice questions and essays
Elstein, A. S. (1993) · Academic Medicine · doi:10.1097/00001888-199304000-00002 · Cited by 83 · see citing works
The virtues of extended matching and uncued tests as alternatives to multiple choice questions
Fenderson, B. A., Damjanov, I., Robeson, M., et al. (1997) · Human Pathology · doi:10.1016/s0046-8177(97)90073-3 · Cited by 73 · see citing works
Cognitive psychology and medical diagnosis
Gilhooly, K. J. (1990) · Applied Cognitive Psychology · doi:10.1002/acp.2350040404 · Cited by 72 · see citing works
Measuring knowledge and clinical reasoning skills in a problem‐based curriculum
Boshuizen, H. P. A., van der Vleuten, C., Schmidt, H. G., et al. (1997) · Medical Education · doi:10.1111/j.1365-2923.1997.tb02469.x · Cited by 53 · see citing works
Comparison of items in five-option and extended-matching formats for assessment of diagnostic skills
Case, S. M., Swanson, D. B., & RIPKEY, D. R. (1994) · Academic Medicine · doi:10.1097/00001888-199410000-00023 · Cited by 47 · see citing works
Small worlds and medical expertise: implications for medical cognition and knowledge engineering
Kushniruk, A. W., Patel, V. L., & Marley, A. A. J. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00044-6 · Cited by 42 · see citing works
Strategy choices in multiple-choice items
Skakun, E. N., Maguire, T. O., & Cook, D. (1994) · Academic Medicine · doi:10.1097/00001888-199410000-00025 · Cited by 22 · see citing works
Setting Standards for Multiple-Choice Items in Clinical Reasoning
Maguire, T. O., Skakun, E. N., & Harley, C. D. (1992) · Evaluation & the Health Professions · doi:10.1177/016327879201500406 · Cited by 13 · see citing works
Assessment of a pattern-recognition examination in a clinical clerkship
Dunn, M. M., & Woolliscroft, J. O. (1994) · Academic Medicine · doi:10.1097/00001888-199408000-00019 · Cited by 12 · see citing works
Solomon, D., Speer, A. J., Perkowski, L. C., et al. (1994) · Academic Medicine · doi:10.1097/00001888-199409000-00022 · Cited by 11 · see citing works
Blackwell, T. A., Ainsworth, M. A., DORSEY, N. K., et al. (1991) · Academic Medicine · doi:10.1097/00001888-199109000-00035 · Cited by 7 · see citing works
Gruppen, L. D., Grum, C. M., Fincher, R. M. E., et al. (1996) · Academic Medicine · doi:10.1097/00001888-199610000-00046 · Cited by 7 · see citing works
Cope, D., Blackwell, T. A., AINSWORTH, M. A., et al. (1991) · Academic Medicine · doi:10.1097/00001888-199109001-00015 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Case S Swanson D Constructing written test questions for the basic and clinical sciences. Third edition 1994
Cited by
Sorted by citations
Qureshi, A., Rizvi, F., Syed, A., Shahid, A., & Manzoor, H. (2014) · Advances in Physiology Education · doi:10.1152/advan.00092.2013 · Cited by 55 · see citing works
Heist, B. S., Gonzalo, J. D., Durning, S., Torre, D., & Elnicki, D. M. (2014) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-14-00176.1 · Cited by 29 · see citing works
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
“Capability to assess higher-order cognition when well-designed Well-designed MCQs can validly assess higher-order clinical reasoning 35 .”
- Authors
- Coderre, S. P., Harasym, P., Mandin, H., & Fick, G.
- Affiliations
- University of Calgary (CA)
- Foothills Medical Centre (CA)
- Corresponding author
- Coderre, Sylvain P
- Venue
- BMC Medical Education
- Published
- 2004-11-05
- Volume
- 4
- Issue
- 1
- Pages
- 23-23
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC533882
- OpenAlex
- W2169966878
- Cited by
- 70
- References
- 27
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 4 · 2023: 5 · 2022: 2 · 2021: 2 · 2020: 2 · 2019: 5 · 2018: 5 · 2017: 2 · 2016: 1 · 2015: 5 · 2014: 3 · 2013: 5 · 2012: 3
- FWCI
- 2.16
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Medical Education and Admissions
- Keywords
- multiple-choice questions, extended matching questions, number of alternatives, problem-solving strategies, clinical reasoning assessment, novices and experts, psychometrics, think-aloud, test format
- MeSH terms
- Adult, Canada, Clinical Competence, Decision Making, Deglutition Disorders, Diarrhea, Educational Measurement, Gastroenterology, Gastrointestinal Diseases, Humans, Liver, Liver Function Tests, Logic, Middle Aged, Nausea, Pattern Recognition, Visual, Problem Solving, Psychometrics, Surveys and Questionnaires, Students, Medical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
