Pattern recognition as a concept for multiple-choice questions in a national licensing exam
Open accessAbstract
BACKGROUND: Multiple-choice questions (MCQ) are still widely used in high stakes medical exams. We wanted to examine whether and to what extent a national licensing exam uses the concept of pattern recognition to test applied clinical knowledge. METHODS: We categorized all 4,134 German National medical licensing exam questions between October 2006 and October 2012 by discipline, year, and type. We analyzed questions from the four largest disciplines: internal medicine (n = 931), neurology (n = 305), pediatrics (n = 281), and surgery (n = 233), with respect to the following question types: knowledge questions (KQ), pattern recognition questions (PRQ), inverse PRQ (IPRQ), and pseudo PRQ (PPRQ). RESULTS: A total 51.1% of all questions were of a higher taxonomical order (PRQ and IPRQ) with a significant decrease in the percentage of these questions (p <0.001) from 2006 (61.5%) to 2012 (41.6%). The proportion of PRQs and IPRQs was significantly lower (p <0.001) in internal medicine and surgery, compared to neurology and pediatrics. PRQs were mostly used in questions about diagnoses (71.7%). A significantly higher (p <0.05) percentage of PR/therapy questions was found for internal medicine compared with neurology and pediatrics. CONCLUSION: The concept of pattern recognition is used with different priorities and to various extents by the different disciplines in a high stakes exam to test applied clinical knowledge. Being aware of this concept may aid in the design and balance of MCQs in an exam with respect to testing clinical reasoning as a desired skill at the threshold of postgraduate medical education.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing multiple-choice evaluations are primarily limited to evaluating pattern recognition and information recall.
“While these methods allow for the swift assessment of model performance, they are primarily limited to evaluating pattern recognition and information recall2.”
From Large Language Models lack essential metacognition for reliable medical reasoning (2025) · cites this paper as [2]
The Methods classify questions as pattern-recognition, knowledge-recall, or related types, and the Results report that 42.2% were pattern-recognition questions and 44.3% were knowledge questions. But the study examined four disciplines in one German medical licensing exam, not multiple-choice evaluations generally or assessments of model performance. The Background also says well-constructed MCQs can test application of knowledge, so the claim's broad limitation goes beyond the paper. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2011) · Cited by 14,317 · see citing works
The assessment of clinical skills/competence/performance
Miller, G. E. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00045 · Cited by 4,883 · see citing works
Education and thinking: The role of knowledge.
Glaser, R. (1984) · American Psychologist · doi:10.1037/0003-066x.39.2.93 · Cited by 1,272 · cited 1× · see citing works
“However, Glaser has already demonstrated in a developmental study in 1984, that knowledge is the single best determinant of expertise [5].”
BackgroundA cognitive perspective on medical expertise
Schmidt, H. G., Norman, G. R., & Boshuizen, H. P. A. (1990) · Academic Medicine · doi:10.1097/00001888-199010000-00001 · Cited by 1,135 · see citing works
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973 · cited 1× · see citing works
“This could be very helpful in designing PRQs at a taxonomically higher order and, therefore, we suggest that pattern recognition should be added as a specific medical concept to MCQ item writing guidelines [24].”
BackgroundWhat every teacher needs to know about clinical reasoning
Eva, K. W. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01972.x · Cited by 939 · see citing works
The real test bias: Influences of testing on teaching and learning.
Frederiksen, N. (1984) · American Psychologist · doi:10.1037/0003-066x.39.3.193 · Cited by 536 · see citing works
Diagnostic error and clinical reasoning
Norman, G. R., & Eva, K. W. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03507.x · Cited by 500 · see citing works
Judd, D. M. (1995) · American Journal of Occupational Therapy · doi:10.5014/ajot.49.3.273a · Cited by 416 · see citing works
Non‐analytical models of clinical reasoning: the role of experience
Norman, G. R., Young, M. E., & Brooks, L. R. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02914.x · Cited by 404 · see citing works
Issues in cognitive psychology
Regehr, G., & Norman, G. R. (1996) · Academic Medicine · doi:10.1097/00001888-199609000-00015 · Cited by 396 · see citing works
Pelaccia, T., Tardif, J., Triby, E., & Charlin, B. (2011) · Medical Education Online · doi:10.3402/meo.v16i0.5890 · Cited by 395 · see citing works
Diagnostic reasoning strategies and diagnostic success
Coderre, S., Mandin, H., Harasym, P. H., & Fick, G. H. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01577.x · Cited by 359 · see citing works
Clinical Reasoning: Forms of Inquiry in a Therapeutic Practice
Mattingly, C., & Fleming, M. H. (1993) · Medical Entomology and Zoology · Cited by 350 · see citing works
Palmer, E. J., & Devitt, P. G. (2007) · BMC Medical Education · doi:10.1186/1472-6920-7-49 · Cited by 294 · see citing works
Varying conceptions of competence: an analysis of how health sciences educators define competence
Fernández, N., Dory, V., Ste‐Marie, L., Chaput, M., Charlin, B., & Boucher, A. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04183.x · Cited by 222 · see citing works
Building on Experience — The Development of Clinical Reasoning
Norman, G. R. (2006) · New England Journal of Medicine · doi:10.1056/nejme068134 · Cited by 144 · see citing works
Using automatic item generation to create multiple‐choice test items
Gierl, M. J., Lai, H., & Turner, S. R. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04289.x · Cited by 139 · see citing works
Context is everything or how could I have been that stupid?
Croskerry, P. (2009) · Healthcare Quarterly · doi:10.12927/hcq.2009.20945 · Cited by 127 · see citing works
Groves, M., O'Rourke, P. K., & Alexander, H. (2003) · Medical Teacher · doi:10.1080/01421590310001605688 · Cited by 93 · see citing works
The modified essay question: Its exit from the exit examination?
Palmer, E., Duggan, P., Devitt, P. G., & Russell, R. (2010) · Medical Teacher · doi:10.3109/0142159x.2010.488705 · Cited by 58 · see citing works
Tracking Development of Clinical Reasoning Ability Across Five Medical Schools Using a Progress Test
Williams, R. G., Klamen, D. L., White, C. B., PETRUSA, E. R., Fincher, R. E., Whitfield, C. F., Shatzer, J. H., McCarty, T. A., & Miller, B. M. (2011) · Academic Medicine · doi:10.1097/acm.0b013e31822631b3 · Cited by 58 · see citing works
Ware, J. S., & Vik, T. (2008) · Medical Teacher · doi:10.1080/01421590802155597 · Cited by 52 · see citing works
Medical Student Acquisition of Clinical Working Knowledge
Williams, R. G., Klamen, D. L., & Hoffman, R. S. (2008) · Teaching and Learning in Medicine · doi:10.1080/10401330701542552 · Cited by 30 · see citing works
Verbosity, window dressing, and red herrings
Case, S. M., Swanson, D. B., & Becker, D. F. (1996) · Academic Medicine · doi:10.1097/00001888-199610000-00035 · Cited by 28 · see citing works
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00044 · Cited by 26 · see citing works
How we created virtual patient cases for primary care-based learning
Adams, E., Rodgers, C. J., Harrington, R., Young, M. D., & Sieber, V. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.544796 · Cited by 26 · see citing works
Pattern Recognition in Pediatric and Adolescent Gynecology—A Case for Formal Education
Muram, D., & Simmons, K. J. (2008) · Journal of Pediatric and Adolescent Gynecology · doi:10.1016/j.jpag.2007.10.009 · Cited by 16 · see citing works
A practical discussion to avoid common pitfalls when constructing multiple choice questions items
Al-Rukban, M., Al-Faris, E., Alorainy, I., & Abdel-Hameed, A. A. (2010) · Journal of Family and Community Medicine · doi:10.4103/1319-1683.71992 · Cited by 16 · see citing works
Twelve tips on setting up a new medical school
Cookson, J. B. (2013) · Medical Teacher · doi:10.3109/0142159x.2013.799638 · Cited by 6 · see citing works
Rutkove, S. B. (2003) · Neurology · doi:10.1212/01.wnl.0000078930.98769.11 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6920/14/232/prepub
Cited by
Sorted by citations
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
Pushing Critical Thinking Skills With Multiple-Choice Questions: Does Bloom’s Taxonomy Work?
Zaidi, N. L. B., Grob, K. L., Monrad, S. U., Kurtz, J. B., Tai, A. W., Ahmed, A. Z., Gruppen, L. D., & Santen, S. A. (2017) · Academic Medicine · doi:10.1097/acm.0000000000002087 · Cited by 96 · see citing works
Five decades of research and theorization on clinical reasoning: a critical review
Yazdani, S., & Abardeh, M. H. (2019) · Advances in Medical Education and Practice · doi:10.2147/amep.s213492 · Cited by 78 · see citing works
Ontology-Based Generation of Medical, Multi-term MCQs
Leo, J., Kurdi, G. R., Matentzoglu, N., Parsia, B., Sattler, U., Forge, S., Donato, G., & Dowling, W. (2019) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-018-00172-w · Cited by 73 · see citing works
Kompetenzorientiert prüfen im Staatsexamen Medizin
Jünger, J. (2018) · Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz · doi:10.1007/s00103-017-2668-9 · Cited by 47 · see citing works
The Evolving Importance of Artificial Intelligence and Radiology in Medical Trainee Education
Fischetti, C., Bhatter, P., Frisch, E., Sidhu, A., Helmy, M., Lungren, M., & Duhaime, E. (2022) · Academic Radiology · doi:10.1016/j.acra.2021.03.023 · Cited by 47 · see citing works
Harendza, S., Krenz, I., Klinge, A., Wendt, U., & Janneck, M. (2017) · Europe PMC (PubMed Central) · doi:10.3205/zma001143 · Cited by 30 · see citing works
Self-citationMonrad, S. U., Zaidi, N. L. B., Grob, K. L., Kurtz, J. B., Tai, A. W., Hortsch, M., Gruppen, L. D., & Santen, S. A. (2021) · Medical Teacher · doi:10.1080/0142159x.2021.1879376 · Cited by 31 · see citing works
Shuriaqi, S. A., Abdulsalam, A. A., Masters, K., Zidoum, H. M., & Alzaabi, A. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8100139 · Cited by 27 · see citing works
Rodríguez-Díez, M. C., Alegre, M., Díez, N., Arbea, L., & Ferrer, M. (2016) · BMC Medical Education · doi:10.1186/s12909-016-0559-7 · Cited by 24 · see citing works
Berberat, P. O., Rotthoff, T., Baerwald, C., Ehrhardt, M., Huenges, B., Johannink, J., Narciß, E., Obertacke, U., Peters, H., & Kadmon, M. (2019) · Europe PMC (PubMed Central) · doi:10.3205/zma001278 · Cited by 21 · see citing works
Mastering Electrocardiogram Interpretation Skills Through a Perceptual and Adaptive Learning Module
Krasne, S., Stevens, C. D., Kellman, P. J., & Niemann, J. T. (2020) · AEM Education and Training · doi:10.1002/aet2.10454 · Cited by 20 · see citing works
Harendza, S., Berberat, P. O., & Kadmon, M. (2017) · Journal of Community Medicine & Health Education · doi:10.4172/2161-0711.1000550 · Cited by 19 · see citing works
Self-citationEffectiveness of mnemonics based teaching in medical education
Meenu, S., Zachariah, A. M., & Balakrishnan, S. (2022) · International Journal of Health Sciences · doi:10.53730/ijhs.v6ns2.7599 · Cited by 10 · see citing works
A comparative study of methods for a priori prediction of MCQ difficulty
Kurdi, G. R., Leo, J., Matentzoglu, N., Parsia, B., Sattler, U., Forge, S., Donato, G., & Dowling, W. (2020) · Semantic Web · doi:10.3233/sw-200390 · Cited by 10 · see citing works
Building an Exam Through Rigorous Exam Quality Improvement
Zaidi, N. L. B., Monrad, S. U., Grob, K. L., Gruppen, L. D., Cherry‐Bukowiec, J. R., & Santen, S. A. (2017) · Medical Science Educator · doi:10.1007/s40670-017-0469-2 · Cited by 10 · see citing works
Assessing core competences of medical students with a test for flight school applicants
Harendza, S., Soll, H., Prediger, S., Kadmon, M., Berberat, P. O., & Oubaid, V. (2019) · BMC Medical Education · doi:10.1186/s12909-018-1438-1 · Cited by 7 · see citing works
Self-citationPotter, H. G., & McLachlan, J. C. (2025) · Medical Teacher · doi:10.1080/0142159x.2025.2496382 · Cited by 6 · see citing works
Raciocínio clínico: percepções e práticas de estudantes de medicina
Landim, D. M. P., Neto, J. L. M., & Soares, J. F. D. S. (2021) · Revista Brasileira de Educação Médica · doi:10.1590/1981-5271v45.1-20200062 · Cited by 5 · see citing works
Huang, S.-M., Yang, H. J., Wang, F., Guo, J., & Liu, S. (2021) · Journal of Curriculum and Teaching · doi:10.5430/jct.v10n1p36 · Cited by 4 · see citing works
When passing fails: Designing multiple choice assessments to control for false positives
Dubins, D. N., Poon, G. M. K., & Raman‐Wilms, L. (2016) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2016.05.005 · Cited by 3 · see citing works
Harendza, S., Bußenius, L., Gärtner, J., Heuser, M., Ahles, J., & Prediger, S. (2023) · PubMed Central · doi:10.3205/zma001599 · Cited by 2 · see citing works
Self-citationSchänzler, V., Riedel, M., Riedel, F., Walter, A., Strizek, B., Weber, E. C., Gembruch, U., Weiß, M., & Recker, F. (2022) · Die Gynäkologie · doi:10.1007/s00129-022-04944-3 · Cited by 2 · see citing works
Can multiple choice questions assess application of knowledge in pharmaceutical science teaching
Ngo, S. N. T. (2018) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 1 · see citing works
Concerns that Linger: Medical School Perspectives on the UKMLA and Shared Assessment
MacDougall, M. (2018) · Journal of Education and Development · doi:10.20849/jed.v2i1.368
FLEX: A BYOD Approach to Electronic Examinations
Küppers, B., & Schroeder, U. (2020) · Intelligent systems reference library · doi:10.1007/978-981-15-9908-8_6
Open-Ended Testing Stays Relevant for Critical Knowledge Assessment
Zrinyi, M. (2019) · COJ Nursing & Healthcare · doi:10.31031/cojnh.2019.05.000619
Wu, R., & Straub, J. (2025) · The Open Journal of Occupational Therapy · doi:10.15453/2168-6408.2304
CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging
Singh, P., Ujjain, S., Gandhi, T. K., & Kumar, S. (2025) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi67747.2025.11269541
Kimura, A., & Fumimoto, S. (2026) · AJP Advances in Physiology Education · doi:10.1152/advan.00042.2026
- Authors
- Freiwald, T., Salimi, M., Khaljani, E., & Harendza, S.
- Affiliations
- University Hospital Frankfurt (DE)
- Vivantes Klinikum (DE)
- University Medical Center Hamburg-Eppendorf (DE)
- Corresponding author
- Harendza, Sigrid
- Venue
- BMC Medical Education
- Published
- 2014-11-14
- Volume
- 14
- Issue
- 1
- Pages
- 232-232
- ISSN
- 1472-6920 (ISSN-L)
- PMC
- PMC4289202
- OpenAlex
- W2120351503
- Cited by
- 31
- References
- 32
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 4 · 2020: 3 · 2019: 5 · 2018: 2 · 2017: 5 · 2016: 2 · 2015: 1
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Education (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Social Sciences › Education › Educational Assessment and Pedagogy
- Keywords
- pattern recognition, multiple-choice questions, medical licensing examinations, high-stakes testing, clinical reasoning, question design, internal medicine, neurology, graduate medical education
- MeSH terms
- Adult, Choice Behavior, Clinical Competence, Education, Medical, Undergraduate, Educational Measurement, Female, Germany, Humans, Licensure, Medical, Male, Problem Solving, Surveys and Questionnaires, Thinking, Pattern Recognition, Physiological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
