Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine?
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Written assessments fall into two classes: constructed-response or open-ended questions, such as the essay and a number of variants of the short-answer question, and selected-response or closed-ended questions; typically in the form of multiple-choice. It is widely believed that constructed response written questions test higher order cognitive processes in a manner that multiple-choice questions cannot, and consequently have higher validity. DISCUSSION: An extensive review of the literature suggests that in summative assessment neither premise is evidence-based. Well-structured open-ended and multiple-choice questions appear equivalent in their ability to assess higher cognitive functions, and performance in multiple-choice assessments may correlate more highly than the open-ended format with competence demonstrated in clinical practice following graduation. Studies of construct validity suggest that both formats measure essentially the same dimension, at least in mathematics, the physical sciences, biology and medicine. The persistence of the open-ended format in summative assessment may be due to the intuitive appeal of the belief that synthesising an answer to an open-ended question must be both more cognitively taxing and similar to actual experience than is selecting a correct response. I suggest that cognitive-constructivist learning theory would predict that a well-constructed context-rich multiple-choice item represents a complex problem-solving exercise which activates a sequence of cognitive processes which closely parallel those required in clinical practice, hence explaining the high validity of the multiple-choice format. SUMMARY: The evidence does not support the proposition that the open-ended assessment format is superior to the multiple-choice format, at least in exit-level summative assessment, in terms of either its ability to test higher-order cognitive functioning or its validity. This is explicable using a theory of mental models, which might predict that the multiple-choice format will have higher validity, a statement for which some empiric support exists. Given the superior reliability and cost-effectiveness of the multiple-choice format consideration should be given to phasing out open-ended format questions in summative assessment. Whether the same applies to non-exit-level assessment and formative assessment is a question which remains to be answered; particularly in terms of the educational effect of testing, an area which deserves intensive study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The assessment of clinical skills/competence/performance
Miller, G. E. (1990) · Academic Medicine · doi:10.1097/00001888-199009000-00045 · Cited by 4,883
Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer
Dreyfus, H. L., Drey-fus, S. E., & Zadeh, L. A. (1987) · IEEE Expert · doi:10.1109/mex.1987.4307079 · Cited by 3,172
Schema induction and analogical transfer
Gick, M. L., & Holyoak, K. J. (1983) · Cognitive Psychology · doi:10.1016/0010-0285(83)90002-6 · Cited by 3,010 · cited 1×
“Where transfer in other words, the successful application of knowledge to new situations has been shown, the common factor appears to be deep learning, and the abstraction of general principles [6-8].”
BackgroundThe Role of Assessment in a Learning Culture
Shepard, L. A. (2000) · Educational Researcher · doi:10.3102/0013189x029007004 · Cited by 2,161 · cited 1×
“A critical concept is that assessment is not only of learning, but also for learning [27,31].”
BackgroundWhen and where do we apply what we learn?: A taxonomy for far transfer
Barnett, S. M., & Ceci, S. J. (2002) · Psychological Bulletin · doi:10.1037/0033-2909.128.4.612 · Cited by 2,076 · cited 1×
“Where transfer in other words, the successful application of knowledge to new situations has been shown, the common factor appears to be deep learning, and the abstraction of general principles [6-8].”
BackgroundValidity: on the meaningful interpretation of assessment data
Downing, S. M. (2003) · Medical Education · doi:10.1046/j.1365-2923.2003.01594.x · Cited by 1,434
The interplay of evidence and consequences in the validation of performance assessments
Messick, S. (1994) · Educational Researcher · doi:10.3102/0013189x023002013 · Cited by 1,315 · cited 1×
“For example, five types of evidence have been proposed which may provide support for such an association [60,61], namely content, the response process, internal structure, relationship to other variables and consequences.”
BackgroundAssessing professional competence: from methods to programmes
Van Der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02094.x · Cited by 1,268
Representations in Distributed Cognitive Tasks
Zhang, J., & Norman, D. A. (1994) · Cognitive Science · doi:10.1207/s15516709cog1801_3 · Cited by 1,203 · cited 1×
“In effect a relationship has been defined between environmental information, activation of higher level cognition and externalisation of internal representations [114].”
BackgroundProgrammatic assessment: From assessment of learning to assessment for learning
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2011) · Medical Teacher · doi:10.3109/0142159x.2011.565828 · Cited by 902 · cited 1×
“Such considerations are particularly useful in using assessment to guide learning rather than merely as a judgment of competence [77].”
BackgroundScouller, K. (1998) · Higher Education · doi:10.1023/a:1003196224280 · Cited by 724 · cited 2×
“Though some studies have suggested that students tended to make more use of surface learning strategies in preparation for MCQ and deeper learning strategies in preparation for open-ended questions [81,82], other studies have failed to show such an association [80,83].”
BackgroundResultHow expertise develops in medicine: knowledge encapsulation and illness script formation
Schmidt, H. G., & Rikers, R. M. J. P. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02915.x · Cited by 619
Different written assessment methods: what can be said about their strengths and weaknesses?
Schuwirth, L. W. T., & Van Der Vleuten, C. P. M. (2004) · Medical Education · doi:10.1111/j.1365-2929.2004.01916.x · Cited by 309
Palmer, E. J., & Devitt, P. G. (2007) · BMC Medical Education · doi:10.1186/1472-6920-7-49 · Cited by 294
Recent research on human learning challenges conventional instructional strategies
Rohrer, D., & Pashler, H. (2010) · Educational Researcher · doi:10.3102/0013189x10374770 · Cited by 269 · cited 4×
“Recent observations that retrieval of knowledge improves retention, and may be enhanced in the learning process by frequent testing [20,21], and in particular a recent publication summarising four studies performed in an authentic learning environment which demonstrates that that testing using MCQ format is as effecti…”
BackgroundResultInfluentialWainer, H., & Thissen, D. (1993) · Applied Measurement in Education · doi:10.1207/s15324818ame0602_1 · Cited by 216
Factors underlying performance on written tests of knowledge
NORMAN, G. R., SMITH, E. K. M., POWLES, A. C. P., ROONEY, P. J., HENRY, N. L., & DODD, P. E. (1987) · Medical Education · doi:10.1111/j.1365-2923.1987.tb00367.x · Cited by 58
Cited by
Teaching, Learning and Assessing Anatomy with Artificial Intelligence: The Road to a Better Future
Abdellatif, H., Al Mushaiqri, M., Albalushi, H., Al-Zaabi, A. A., Roychoudhury, S., & Das, S. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192114209 · Cited by 120
“…in the methods for delivering Anatomy including AI, the assessment practices have adapted accordingly with modern strategies such as competency based assessment, portfolio based and programmatic assessment, which are now implemented in many modern and health-related medical curricula [51,70].”
BackgroundMethodCherif, H., Moussa, C., Missaoui, A. M., Salouage, I., Mokaddem, S., & Dhahri, B. (2024) · JMIR Medical Education · doi:10.2196/52818 · Cited by 18
“The way questions are presented greatly affects how well both medical students and AI tools like ChatGPT perform [21,22].”
BackgroundHollis-Sando, L., Pugh, C., Franke, K., Zerner, T., Tan, Y., Carneiro, G., Van den Hengel, A., Symonds, I., Duggan, P., & Bacchi, S. (2023) · Focus on Health Professional Education: A Multi-Professional Journal · doi:10.11157/fohpe.v24i1.531 · Cited by 7
“, 2019), although this point may be debated (Hift, 2014).”
Background
Metadata
- Authors
- Hift, R. J.
- Venue
- BMC Medical Education
- Published
- 2014-11-28
- Volume
- 14
- Issue
- 1
- Pages
- 249-249
- ISSN
- 1472-6920 (ISSN-L)
- OpenAlex
- W2036851253
- Cited by
- 133
- References
- 17
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.