Comparing comprehension measured by multiple-choice and open-ended questions.
The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
This study compared the nature of text comprehension as measured by multiple-choice format and open-ended format questions. Participants read a short text while explaining preselected sentences. After reading the text, participants answered open-ended and multiple-choice versions of the same questions based on their memory of the text content. The results indicated that performance on open-ended questions was correlated with the quality of self-explanations, but performance on multiple-choice questions was correlated with the level of prior knowledge related to the text. These results suggest that open-ended and multiple-choice format questions measure different aspects of comprehension processes. The results are discussed in terms of dual process theories of text comprehension.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, verifying and falsifying suggestions is a different cognitive task than independently gathering findings and deducing a diagnosis15,16.”
From Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays (2023) · cites this paper as [15]
Citations
Cites
(1998) · Cited by 4,875
The role of knowledge in discourse comprehension: A construction-integration model.
Kintsch, W. (1988) · Psychological Review · doi:10.1037/0033-295x.95.2.163 · Cited by 4,363
The Nature of Recollection and Familiarity: A Review of 30 Years of Research
Yonelinas, A. P. (2002) · Journal of Memory and Language · doi:10.1006/jmla.2002.2864 · Cited by 3,840 · cited 1×
“This is because the target information or its variant is present as a part of the answer options (see Yonelinas, 2002 for discussions on recollection and familiarity).”
BackgroundTheoretical Models and Processes of Reading
Barnett, M. A., Singer, H., & Ruddell, R. B. (1987) · The Modern Language Journal · doi:10.2307/326762 · Cited by 3,086
Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention
Roediger, H. L., & Karpicke, J. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01693.x · Cited by 2,867
Levels of processing versus transfer appropriate processing
Morris, C. D., Bransford, J. D., & Franks, J. J. (1977) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(77)80016-9 · Cited by 2,296 · cited 1×
“Further, note that the transfer appropriate processing perspective (Morris et al., 1977) states that memory retrieval is best when the nature of the encoding process corresponds to the nature of the retrieval process.”
BackgroundMcNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996) · Cognition and Instruction · doi:10.1207/s1532690xci1401_1 · Cited by 1,341
Self-citationEliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
Learning from texts: Effects of prior knowledge and text coherence
McNamara, D. S., & Kintsch, W. (1996) · Discourse Processes · doi:10.1080/01638539609544975 · Cited by 745 · cited 3×
“We presented the prior knowledge questions at the end of the experiment to avoid exposing participants to topic-related questions that could influence participants’ reading behaviour (see McNamara & Kintsch, 1996).”
BackgroundSelf-citationToward an Interactive Model of Reading
Rumelhart, D. (2013) · Theoretical Models and Processes of Reading · doi:10.1598/0710.29 · Cited by 628
Cited by
Hamilton, D., McKechnie, J., Edgerton, E., & Wilson, C. (2021) · Journal of Computers in Education · doi:10.1007/s40692-020-00169-2 · Cited by 1,081
Elgort, I., & Warren, P. (2014) · Language Learning · doi:10.1111/lang.12052 · Cited by 240
Connor Desai, S., & Reimers, S. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-1066-z · Cited by 127
“This finding is particularly interesting in light of the fact that responses to open and closed questions might be supported by different underlying retrieval processes (Fisher, Brewer, & Mitchell, 2009; Ozuru et al., 2013; Shapiro, 2006).”
BackgroundStuijfzand, S., Creswell, C., Field, A. P., Pearcey, S., & Dodd, H. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12822 · Cited by 151
“These response formats require active and passive information generation, respectively, which may influence responses and subsequent conclusions regarding bias (Ozuru, Best, Bell, Witherspoon, & McNamara, 2007; Ozuru, Briner, Kurby, & McNamara, 2013).”
A systematic review of pedagogical agent research: Similarities, differences and unexplored aspects
Dai, L., Jung, M. M., Postma, M., & Louwerse, M. M. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104607 · Cited by 128
Yeari, M., van den Broek, P., & Oudega, M. (2015) · Reading and Writing · doi:10.1007/s11145-015-9561-4 · Cited by 104
Very‐short‐answer questions: reliability, discrimination and acceptability
Sam, A. H., Field, S. M., Collares, C. F., van der Vleuten, C. P. M., Wass, V. J., Melville, C., Harris, J., & Meeran, K. (2018) · Medical Education · doi:10.1111/medu.13504 · Cited by 91
Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays
Gaube, S., Suresh, H., Raue, M., Lermer, E., Koch, T. K., Hudecek, M. F. C., Ackery, A. D., Grover, S. C., Coughlin, J. F., Frey, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-28633-w · Cited by 92
Balepur, N., Rudinger, R., & Boyd-Graber, J. L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.169 · Cited by 84
Golding, S., & Verrier, D. B. (2020) · Journal of Graphic Novels & Comics · doi:10.1080/21504857.2020.1786419 · Cited by 84
“Open ended questions were used, as these have been shown to better relate to inferential understanding (Ozuru et al., 2013).”
Actual drawing of histological images improves knowledge retention
Balemans, M. C. M., Kooloos, J. G. M., Donders, A. R. T., & der Zee, C. E. E. M. V. (2015) · Anatomical Sciences Education · doi:10.1002/ase.1545 · Cited by 83
“During this study, students were exposed to post-tests with variable levels of difficulty, and presented to tests in a particular order (see also Van Meter, 2001, Van Meter et al., 2006; Ozuru et al., 2013).”
BackgroundMethodvan der Graaf, J., van de Sande, E., Gijsel, M. A. R., & Segers, E. (2019) · International Journal of Science Education · doi:10.1080/09500693.2019.1594442 · Cited by 75
“On the other hand, multiple choice relies more on topic-specific knowledge (Ozuru et al., 2013).”
BackgroundPersonalized learning in iSTART: Past modifications and future design
McCarthy, K. S., Watanabe, M., Dai, J., & McNamara, D. S. (2020) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1716201 · Cited by 75
Self-citationMultiple-choice questions: Tips for optimizing assessment in-seat and online.
Xu, X., Kauer, S. D., & Tupy, S. (2016) · Scholarship of Teaching and Learning in Psychology · doi:10.1037/stl0000062 · Cited by 70
McCarthy, K. S., Likens, A. D., Johnson, A. M., Guerrero, T. A., & McNamara, D. S. (2018) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-018-0164-5 · Cited by 49
“These rubrics award partial credit (0.0, 0.25, 0.50, 0.75, or 1.0) for incomplete answers (McNamara et al. 2006; Ozuru et al. 2013).”
BackgroundInfluentialSelf-citationVasan, C., DeFouw, D. O., Holland, B. K., & Vasan, N. S. (2017) · Anatomical Sciences Education · doi:10.1002/ase.1739 · Cited by 62
Learn with surprize from a robot professor
Velentza, A.-M., Fachantidis, N., & Lefkos, I. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104272 · Cited by 50
Difficulties and innovations in teaching anatomy and physiology in nursing
Satoh, M., Fujimura, A., & Miyagawa, S. (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2023.103551 · Cited by 43
Impact of text availability and question format on reading comprehension processes
Ferrer, A., Vidal‐Abarca, E., Serrano-Mendizábal, M., & Gilabert, R. (2017) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2017.10.002 · Cited by 41
Steiner, M., van Loon, M. H., Bayard, N. S., & Roebers, C. M. (2020) · Metacognition and Learning · doi:10.1007/s11409-019-09208-5 · Cited by 38
Evaluating Persona Prompting for Question Answering Tasks
Olea, C., Tucker, H., Phelan, J., Pattison, C., Zhang, S., Lieb, M., Schmidt, D., & White, J. (2024) · International Conference on Security, Privacy and Trust Management (SPTM) · doi:10.5121/csit.2024.141106 · Cited by 38
“As found in [14], it is possible that multiple choice and short answer/open-ended questions measure different aspects of comprehension processes.”
BackgroundAutomated essay scoring in applied games: Reducing the teacher bandwidth problem in online training
Westera, W., Dascălu, M., Kurvers, H., Ruşeţi, Ş., & Trăușan-Matu, Ș. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.05.010 · Cited by 37
Liu, H. (2020) · Journal of Research in Reading · doi:10.1111/1467-9817.12339 · Cited by 34
Kim, K., Clarianay, R. B., & Kim, Y. A. (2018) · Educational Technology Research and Development · doi:10.1007/s11423-018-9626-6 · Cited by 33
Wongwatkit, C., Panjaburee, P., & Srisawasdi, N. (2016) · International Journal of Mobile Learning and Organisation · doi:10.1504/ijmlo.2017.080898 · Cited by 31
Öksüz, Y., & DEMİR, E. G. (2018) · Hacettepe University Journal of Education · doi:10.16986/huje.2018040550 · Cited by 31
Effectual networking capability and SME performance in international B2B markets
Karami, M., Wooliscroft, B., & McNeill, L. S. (2023) · Journal of Business and Industrial Marketing · doi:10.1108/jbim-01-2022-0020 · Cited by 24
Tan, S., Thibault, G., Chew, A. C. Y., & Rajalingam, P. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12680 · Cited by 23
Inoue, M., Fukahori, H., Matsubara, M., Yoshinaga, N., & Tohira, H. (2022) · Japan Journal of Nursing Science · doi:10.1111/jjns.12520 · Cited by 22
“Yet, open-ended questions often include diverse answers incorporating respondents ’ unique perspectives and experience, unlike pre-defined questions (He & Schonlau, 2021; Ozuru et al., 2013).”
Human-robot interaction methodology: Robot teaching activity
Velentza, A.-M., Fachantidis, N., & Lefkos, I. (2022) · MethodsX · doi:10.1016/j.mex.2022.101866 · Cited by 22
“Type of the questions : We used both types of questions since multiple-choice and open-ended questions correspond to different aspects of comprehension processes [28] .”
MethodDésiron, J. C., Bétrancourt, M., & de Vries, E. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.592509 · Cited by 21
Teachers’ Use of Open-ended Questions: A Mixed-methods Study
Karaman, P., & BÜYÜKKIDIK, S. (2023) · The Clearing House A Journal of Educational Strategies Issues and Ideas · doi:10.1080/00098655.2023.2166891 · Cited by 19
Morris, W., Crossley, S. A., Holmes, L., & Trumbore, A. (2023) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3576050.3576098 · Cited by 20
“research indicates essay questions measure different competencies frommultiple choice items [31].”
BackgroundDevelopments in language education : a memorial volume in honour of Sauli Takala
Huhta, A., Erickson, G., & Figueras, N. (2019) · Jyväskylä University Digital Archive (University of Jyväskylä) · Cited by 19
Schiller, I., Aspöck, L., & Schlittmeier, S. J. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1243249 · Cited by 19
“Multiple-choice questions are generally easier, because they require less active memory recall and prior knowledge (Ozuru et al., 2013; Polat, 2020).”
BackgroundChoosing the right question: the effect of different question types on multiple text integration
Primor, L., Yeari, M., & Katzir, T. (2021) · Reading and Writing · doi:10.1007/s11145-021-10127-8 · Cited by 19
Desafios na avaliação da compreensão leitora: demanda cognitiva e leiturabilidade textual
de Sousa, L. B., & Hübner, L. C. (2015) · Revista Neuropsicologia Latinoamericana · doi:10.5579/rnl.2015.0237 · Cited by 19
Gaze-based predictive models of deep reading comprehension
Southwell, R. V., Mills, C., Caruso, M., & D’Mello, S. K. (2022) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-022-09346-7 · Cited by 18
“…a reliable and generalizable link between eye gaze and self-explanations, which measures comprehension at deeper levels by encompassing multiple components of reading such as inferencing, elaboration, and prior knowledge activation (McNamara, 2004; McNamara et al., 2007; Ozuru et al., 2013).”
BackgroundAutomating Academic Assessment: A Large Language Model Approach
Wangwiwattana, C., & Tongvivat, Y. (2023) · doi:10.1109/incit60207.2023.10412991 · Cited by 16
“There is a growing trend to shift evaluation methods toward open-ended questions rather than relying solely on multiple-choice items [9].”
BackgroundTreeQuestion: Assessing Conceptual Learning Outcomes with LLM-Generated Multiple-Choice Questions
Cheng, Z., Xu, J., & Jin, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686970 · Cited by 16
“Multiple-choice questions can also be effective assessment tools with a variety of benefits for both teachers and students [43, 66].”
Merzougui, W., Myers, M. A., Hall, S., Elmansouri, A., Parker, R., Robson, A., Kurn, O. R., Parrott, R., Geoghegan, K., Harrison, C. H., et al. (2021) · Anatomical Sciences Education · doi:10.1002/ase.2053 · Cited by 16
“…education, often resulting in mixed outcomes (Badger and Thomas, 1991; Bridgeman, 1992; Wolfson et al., 2001; Schuwirth and van der Vleuten, 2004a; Ozuru et al., 2013; Wooton et al., 2014; Brenner et al., 2015; Thompson and O’Loughlin, 2015; Biria and Dehghan, 2016; Adamu et al., 2018; Hauer et…”
BackgroundInfluentialTest format effects: a componential approach to second language reading
Lim, H. (2019) · Language Testing in Asia · doi:10.1186/s40468-019-0082-y · Cited by 16
ASAG2024: A Combined Benchmark for Short Answer Grading
Meyer, G., Breuer, P., & Fürst, J. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649409.3691083 · Cited by 14
“Among the various question types, open-ended questions can test a more thorough understanding compared to closed-ended questions such as multiple-choice [13].”
Contribution of metacognitive questions to accuracy of judgment of learning in a digital environment
Michalsky, T., & Bakrish, H. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e40055 · Cited by 14
Schauber, S. K., Hautz, S. C., Kämmer, J. E., Stroben, F., & Hautz, W. E. (2021) · Advances in Health Sciences Education · doi:10.1007/s10459-021-10052-z · Cited by 14
Vocabulary and text comprehension levels: what should be considered in assessments?
Kärbla, T., Männamaa, M., & Uibu, K. (2019) · Educational Psychology · doi:10.1080/01443410.2019.1703172 · Cited by 14
“This kind of task might not be as familiar to students as traditional multiple-choice items, which demand less from active memory insofar as students only need to recognise the right answer from a given set of options (Ozuru et al., 2013).”
Video‐blogs and linguistic simplification for students with intellectual disability
Fajardo, I., Clemente, V. Á., Delgado, P., Gómez‐Merino, N., & Salmerón, L. (2022) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13016 · Cited by 13
“We elaborated three literal questions and three inferential questions per blog post (Ozuru et al., 2013).”
BackgroundGarden-path sentences and the diversity of their (mis)representations
Ceháková, M., & Chromý, J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0288817 · Cited by 13
“[39], using open-ended comprehension questions can lead to a lower response accuracy as compared to multiple-choice questions.”
MethodAssessment of adults with low literacy skills: a review of methods
Chyl, K., Dębska, A., Pokropek, A., Szczerbiński, M., Tanaś, Ł., & Sitek, M. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1346073 · Cited by 13
Do Multimedia Instructional Designs Enhance Comprehension in College Students with Dyslexia
Kim, S. J., Wiseheart, R., & Walden, P. R. (2018) · The Journal of Postsecondary Education and Disability · Cited by 13
Oog voor diep begrip: Onderzoek naar het tekstbegrip van vwo-leerlingen
Rooijackers, P. J. H. (2023) · Utrecht University Repository (Utrecht University) · doi:10.33540/1643 · Cited by 13
Bae, M., & Lee, B. (2018) · English Teaching · doi:10.15858/engtea.73.4.201812.149 · Cited by 12
“Second, it is emphasized to assess reading skills diversifying the question types as well as reading comprehension tests (Alderson, 2000; Kang, 2005; Ozuru, Best, Bell, Witherspoon, & McNamara, 2007; Ozuru, Briner, Kurby, & McNamara, 2013; Ozuru, Rowe, O’Reilly, & McNamara, 2008).”
BackgroundNeumann, J., Simmrodt, S., Teichert, H., & Gergs, U. (2021) · Education Research International · doi:10.1155/2021/7767935 · Cited by 12
“+ey have shown using a short biology text in psychology students that VSA requires text processing and really understanding the new given text, whereas SC could be answered mainly based on prior biology high school knowledge [7].”
BackgroundMethodInfluentialDeveloping a Two-Tier Proportional Reasoning Skill Test: Validity and Reliability Studies
AÇIKGÜL, K. (2021) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.909316 · Cited by 12
Mwalongo, A. I. (2014) · Research Commons (University of Waikato) · Cited by 12
Assessment of text comprehension and teaching comprehension strategies in Estonian basic school
Kärbla, T. (2020) · DSpace repository (University of Tartu) · Cited by 12
Anatomy exam model for the circulatory and respiratory systems using GPT-4: a medical school study
ORHA, A. T., Karamus, N. F., & Çolak, T. (2025) · Surgical and Radiologic Anatomy · doi:10.1007/s00276-025-03667-z · Cited by 11
Livas, C., & Karali, N. (2022) · Higher Education Skills and Work-based Learning · doi:10.1108/heswbl-02-2022-0039 · Cited by 10
Alsharif, A. H., Kapfhammer, G. M., & McMinn, P. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00076 · Cited by 10
“While some studies use multiple-choice questions to assess problem-solving skill [26], others report that performance on a multiple-choice quiz correlates with knowledge of a written text [27].”
BackgroundYeari, M., Schlesinger, L. M., & Moshka, E. (2021) · Reading Research Quarterly · doi:10.1002/rrq.452 · Cited by 10
Llorens, A.-C., Vidal‐Abarca, E., Cerdán, R., & Clemente, V. Á. (2015) · Journal for the Study of Education and Development Infancia y Aprendizaje · doi:10.1080/02103702.2015.1076269 · Cited by 10
Buthelezi, L. L. (2023) · DUT Open Scholar (Durban University of Technology) · doi:10.51415/10321/4776 · Cited by 10
Learning or Boredom? Task Adaptation Effects in Sentence Processing Experiments
Chromý, J., & Tomaschek, F. (2024) · Open Mind · doi:10.1162/opmi_a_00173 · Cited by 9
Woodcock, S., Howard, S. J., & Ehrich, J. F. (2019) · School Psychology · doi:10.1037/spq0000340 · Cited by 9
Diaz, M. P., Steen, M., Brown, A. E., Fleet, J., & Williams, J. (2021) · JMIR Research Protocols · doi:10.2196/32911 · Cited by 9
“Multiple-choice questionnaires are a convenient and effective assessment tool that has been extensively used in research to test knowledge [35-38].”
Cárcamo, B. (2023) · International Journal of Applied Linguistics · doi:10.1111/ijal.12466 · Cited by 9
Akın, E. Ö., Mustafayev, R., Pekcici, E. B. B., & Ertem, İ. (2021) · Journal of Pediatrics and Congenital Disorders · doi:10.47275/2379-6707-107 · Cited by 8
Akgün, H., & Ünaldı, A. (2022) · Studies in Language Assessment · doi:10.58379/uixk2220 · Cited by 7
“However, except for Lim (2014), who used a mixed-method design, other studies used only qualitative methods for data collection such as cognitive interviews, self-explanation protocols, and concurrent or retrospective verbal reports (e.g. Cohen & Upton, 2006; Ozuru et al., 2013; Rupp et al., 2006).”
BackgroundMethodEffects of Adjunct Questions on L2 Reading Comprehension with Texts of Different Types
Sun, Y., Zhou, W., & Tang, S. (2024) · Behavioral Sciences · doi:10.3390/bs14020138 · Cited by 7
“Both written recalls and why questions are open-ended questions, the completion of which requires information recollection [30].”
BackgroundFun, experience or education? Learning efficiency – virtual reality lesson vs traditional lesson
Brylska, K., Gackowski, T., Kwiatkowska, A., & Dudziak-Kisio, M. (2024) · Information Technology and People · doi:10.1108/itp-08-2022-0631 · Cited by 6
Anghel, E., & von Davier, M. (2025) · Large-scale Assessments in Education · doi:10.1186/s40536-025-00241-2 · Cited by 6
“While our main interest was the comprehension level, the model included mode to account for how these item types diverge when measuring comprehension and the different response processes associated with them (Ozuru et al., 2013).”
Clark, S. A., Rivers, M. L., & Overono, A. L. (2025) · Behavioral Sciences · doi:10.3390/bs15040477 · Cited by 6
“However, they have been criticized for encouraging guesswork, relying on recognition rather than retrieval processes, and failing to measure complex understanding effectively (e.g., Brown & Abdulnabi, 2017; Greving & Richter, 2022; Ozuru et al., 2013).”
Tang, U. H., Tranberg, R., Sundberg, L., & Scandurra, I. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11674-w · Cited by 6
Evaluation of Interventions in Blended Learning Using a Communication Skills Serious Game
Lala, R., Corbalan, G., & Jeuring, J. T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-34350-7_31 · Cited by 6
“Ozuru et al. find a positive correlation of the performance on open text questions with the quality of an explanation, and a positive correlation of the performance on multiple choice questions with the level of prior knowledge related to the text.”
BackgroundResultInfluentialGioia, A. R., Ahmed, Y., Woods, S. P., & Cirino, P. T. (2022) · Reading and Writing · doi:10.1007/s11145-022-10274-6 · Cited by 6
“However, a small amount of research has used experimental measures assessing the two (Spivey, 1990; Spivey & King, 1989) and there exist state assessments that combine the two 1 3 725 A. R. Gioia et al. domains (e.g., Ozuru et al., 2013).”
BackgroundAptitude and achievement testing
Katz, L. J., & Brown, F. C. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-802203-0.00005-5 · Cited by 6
Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question AnsweringPreprint
Akpinar, N.-J., Lee, C., Murdock, V., & Perona, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.12925 · Cited by 6
Zhang, T. (2025) · Reading Research Quarterly · doi:10.1002/rrq.70032 · Cited by 5
“Future studies could incorporate open-ended or essay-type questions, which are argued to better capture higher order skills such as critical thinking, inference-making, and deeper comprehension beyond identifying correct options (Ko 2010; Ozuru et al. 2013).”
Learning from text and animations: A study into the need for cross-representational signaling
Désiron, J. C., Bétrancourt, M., & de Vries, E. (2021) · L’Année psychologique · doi:10.3917/anpsy1.214.0393 · Cited by 5
Wongthai, P., Photia, A., Traivaree, C., Monsereenusorn, C., Lertvivatpong, N., Sudnawa, K. K., & Rujkijyanont, P. (2022) · Pediatric Blood & Cancer · doi:10.1002/pbc.29785 · Cited by 5
Aldrich, R. S., Trammell, B. A., Poli, S., Potter, S., & Garringer, K. (2018) · InSight A Journal of Scholarly Teaching · doi:10.46504/14201807al · Cited by 5
The Concurrent Validity of the GORT-5 with the WJ-IV
Jennings, R. L., Midkiff, M., McCauley, E. N., Lopuch, J., Stroebel, S. S., James, R., Toler, M., Wendell, R., King, P., & Frampton, M. (2024) · Contemporary School Psychology · doi:10.1007/s40688-024-00502-x · Cited by 4
“The elimination of the multiple-choice items additionally lessened guessing the correct answer due to chance alone, while demanding that the reader generate accurate and on-topic responses to correctly answer the open-ended questions (Ozuru et al., 2013).”
Predicting Multi-document Comprehension: Cohesion Network Analysis
Nicula, B., Perret, C. A., Dascălu, M., & McNamara, D. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23204-7_30 · Cited by 4
Self-citationReading Comprehension Tests for Children: Test Equating and Specific Age-Interval Reports
Lúcio, P. S., Coutinho-Lourenço, F., Cogo‐Moreira, H., Bandalos, D. L., de Carvalho, C. A. F., Kida, A. D. S. B., & de Ávila, C. R. B. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.662192 · Cited by 4
“To overcome the limitations, multiple-choice tests format for evaluating active processing in reading comprehension (Ozuru et al., 2013), we used open-ended questions.”
MethodSemi-automatic short answers clustering and grading with K-Means and Keyword Matching algorithms
Wangwiwattana, C., & Tongvivat, Y. (2022) · doi:10.1109/incit56086.2022.10067412 · Cited by 4
“Nevertheless, it is undeniable that there is an impetus for change in the evaluation types, shifting towards the adoption of open-ended questions rather than evaluating learning outcomes from multiple-choice testing [8].”
BackgroundKumarasamy, T., Nam, S., & Fels, D. I. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-62849-8_4 · Cited by 4
Baussart, B., Prebot, J., Assié, G., Reina, V., Leclercq, D., Villa, C., & Gaillard, S. (2025) · Neurochirurgie · doi:10.1016/j.neuchi.2025.101652 · Cited by 4
Omori, M., & Yamamoto, J. (2022) · Behavior Analysis in Practice · doi:10.1007/s40617-021-00671-8 · Cited by 4
“Responses to closed-ended questions are often correlated with prior knowledge, whereas open-ended questions measure the quality of self-explanations (Ozuru et al., 2013).”
BackgroundAn Examination of the Testing and Spacing Effects in a Middle Grades Social Studies Classroom
Liming, M. C., & Cuevas, J. A. (2017) · Georgia Educational Researcher · doi:10.20429/ger.2017.140104 · Cited by 4
“Ozuru et al. (2013) also found that even after answering SA questions correctly, students did not necessarily answer the corresponding MC question correctly.”
BackgroundTécnicas de aprendizaje de conocimiento científico a partir de textos
Rubio-Peñarrubia, A. (2019) · Investigaciones Sobre Lectura · doi:10.37132/isl.v0i10.255 · Cited by 2
“AOQ will also encourage more complex cognitive processes as opposed to selforiented comprehension by SE (Ozuru et al., 2013), so AOQ students will present more elaborations than SE students (Hypothesis 2).”
BackgroundNatural Language Processing for scoring open-ended questions : a systematic review
Kerkhof, R. G. (2020) · Cited by 4
Human-in-the-Loop LLM Grading for Handwritten Mathematics AssessmentsPreprint
Vanhoyweghen, A., Holst, V., Mobini, M., de Voorde, L. V., Vanleke, T., Verbruggen, B., Verbeken, B., Algaba, A., Verboven, S., Guerry, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.13083 · Cited by 4
Who should do as AI say? Only non-task expert physicians benefit from correct explainable AI advice
Gaube, S., Suresh, H., Raue, M., Lermer, E., Koch, T., Hudecek, M., Ackery, A. D., Grover, S. C., Coughlin, J. F., Frey, D., et al. (2022) · doi:10.21203/rs.3.rs-1687219/v1 · Cited by 3
Roelofs, E. C., Farrokhnia, M., & Haeri, M. A. (2026) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2026.101599 · Cited by 3
Language Assessment Feedback Towards Pedagogical Conversational Agent
Cheeli, B. P., & Latha, M. (2024) · Sino-US English Teaching · doi:10.17265/1539-8072/2024.05.001 · Cited by 3
“MCQs are used to test low/basic levels of knowledge (Bremner et al., 2018), and open-ended questions are used to test high levels of knowledge (Ozuru et al., 2013).”
MethodDo Explanatory Prompts Enhance Retrieval Quality of Science Text Content?
McCarthy, K. S., Dahl, A. C., & Hinze, S. R. (2025) · The Journal of Experimental Education · doi:10.1080/00220973.2025.2476646 · Cited by 3
Enseñar y aprender a leer en inglés: creencias de futuros docentes de lengua extranjera chilenos
Ochoa, T. T., Muñoz, C. S., Rojas, V., Espinoza, M., Rosales, L. U., Flores, L., & Briesmaster, M. (2020) · Folios · doi:10.17227/folios.52-11778 · Cited by 3
Gay, K. D., Leal, A., Ruth, T. K., Lamm, A. J., & Rumble, J. N. (2015) · Journal of International Agricultural and Extension Education · doi:10.4148/2831-5960.1259 · Cited by 3
Development and Assessment of Self-explaining Skills in College Chemistry Instruction
Villalta‐Cerdas, A. (2014) · Digital Commons - University of South Florida (University of South Florida) · Cited by 3
Mayoral, B. R. D. (2015) · Buleria (Universidad de León) · doi:10.18002/10612/5901 · Cited by 3
Metadata
- Authors
- Ozuru, Y., Briner, S., Kurby, C. A., & McNamara, D. S.
- Affiliations
- University of Illinois Chicago (US)
- Grand Valley State University (US)
- Arizona State University (US)
- Venue
- Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale
- Published
- 2013-09
- Volume
- 67
- Issue
- 3
- Pages
- 215-227
- ISSN
- 1196-1961 (ISSN-L)
- 1878-7290
- OpenAlex
- W2202297923
- Cited by
- 134
- References
- 37
- Citations by year
- 2026: 14 · 2025: 7 · 2024: 12 · 2023: 12 · 2022: 17 · 2021: 18 · 2020: 10 · 2019: 13 · 2018: 10 · 2017: 7 · 2016: 5 · 2015: 6 · 2014: 3
- FWCI
- 9.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- text comprehension, multiple-choice questions, open-ended questions, self-explanation, prior knowledge, dual-process theory, reading comprehension
- MeSH terms
- Choice Behavior, Female, Humans, Language, Male, Surveys and Questionnaires, Reading, Mental Recall, Semantics, Statistics as Topic, Students, Universities, Vocabulary, Knowledge, Comprehension, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.