Comparing comprehension measured by multiple-choice and open-ended questions.
The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
This study compared the nature of text comprehension as measured by multiple-choice format and open-ended format questions. Participants read a short text while explaining preselected sentences. After reading the text, participants answered open-ended and multiple-choice versions of the same questions based on their memory of the text content. The results indicated that performance on open-ended questions was correlated with the quality of self-explanations, but performance on multiple-choice questions was correlated with the level of prior knowledge related to the text. These results suggest that open-ended and multiple-choice format questions measure different aspects of comprehension processes. The results are discussed in terms of dual process theories of text comprehension.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Indeed, verifying and falsifying suggestions is a different cognitive task than independently gathering findings and deducing a diagnosis15,16.”
From Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays (2023) · cites this paper as [15]
Citations
Cites
(1998) · Cited by 4,875
The role of knowledge in discourse comprehension: A construction-integration model.
Kintsch, W. (1988) · Psychological Review · doi:10.1037/0033-295x.95.2.163 · Cited by 4,363
The Nature of Recollection and Familiarity: A Review of 30 Years of Research
Yonelinas, A. P. (2002) · Journal of Memory and Language · doi:10.1006/jmla.2002.2864 · Cited by 3,840 · cited 1×
“This is because the target information or its variant is present as a part of the answer options (see Yonelinas, 2002 for discussions on recollection and familiarity).”
BackgroundTheoretical Models and Processes of Reading
Barnett, M. A., Singer, H., & Ruddell, R. B. (1987) · The Modern Language Journal · doi:10.2307/326762 · Cited by 3,086
Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention
Roediger, H. L., & Karpicke, J. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01693.x · Cited by 2,867
Levels of processing versus transfer appropriate processing
Morris, C. D., Bransford, J. D., & Franks, J. J. (1977) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(77)80016-9 · Cited by 2,296 · cited 1×
“Further, note that the transfer appropriate processing perspective (Morris et al., 1977) states that memory retrieval is best when the nature of the encoding process corresponds to the nature of the retrieval process.”
BackgroundMcNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996) · Cognition and Instruction · doi:10.1207/s1532690xci1401_1 · Cited by 1,341
Self-citationEliciting Self‐Explanations Improves Understanding
Chi, M. T. H., De Leeuw, N., Chiu, M., & Lavancher, C. (1994) · Cognitive Science · doi:10.1207/s15516709cog1803_3 · Cited by 757
Learning from texts: Effects of prior knowledge and text coherence
McNamara, D. S., & Kintsch, W. (1996) · Discourse Processes · doi:10.1080/01638539609544975 · Cited by 745 · cited 3×
“We presented the prior knowledge questions at the end of the experiment to avoid exposing participants to topic-related questions that could influence participants’ reading behaviour (see McNamara & Kintsch, 1996).”
BackgroundSelf-citationToward an Interactive Model of Reading
Rumelhart, D. (2013) · Theoretical Models and Processes of Reading · doi:10.1598/0710.29 · Cited by 628
Cited by
Items 101–187 of 187
Dis)continuing the continued influence effect of misinformation
Desai, S. C. (2018) · City Research Online (City University London) · Cited by 3
Schmauder, R., Daugherty, K. K., Rudolph, M. J., Hutson, B., McCarthy, K., McCauley, S., & Eagle, D. (2025) · Intersection A Journal at the Intersection of Assessment and Learning · doi:10.61669/001c.146079 · Cited by 2
“Other concerns address whether MCQs truly measure deep understanding of course material (Mehrens, 1987; Ozuru et al., 2013).”
Digital Technology and Artificial Intelligence in Mathematics Education Assessment
Geraniou, E., Crisan, C., & Mavrikis, M. (2025) · Routledge eBooks · doi:10.4324/9781003533764 · Cited by 2
Indicators of Reading Comprehension: Example of Narrative Text and Open Ended Question
Kuşdemir, Y. (2020) · International Journal of Progressive Education · doi:10.29329/ijpe.2020.277.2 · Cited by 2
Decanonizing Contemporary Culture Courses: Teaching Culture with Twitter
Maroun, D. N. (2022) · Springer eBooks · doi:10.1007/978-3-030-95357-7_13 · Cited by 2
Wulanningtyas, M. E., Arfin, A., Linling, Y., Setiana, D. S., & Boriboon, G. (2024) · Jurnal Pendidikan MIPA · doi:10.23960/jpmipa/v25i1.pp468-485 · Cited by 2
GEOMETRİDE ÖLÇME DEĞERLENDİRME: ÇOKTAN SEÇMELİ VE AÇIK UÇLU SINAVLARIN KARŞILAŞTIRILMASI
Sayın, V., & Orbay, K. (2024) · 3 SEKTÖR SOSYAL EKONOMİ DERGİSİ · doi:10.15659/3.sektor-sosyal-ekonomi.24.12.2564 · Cited by 2
Kirpouiki, A., & Agaliotis, I. (2025) · British Journal of Special Education · doi:10.1111/1467-8578.12575 · Cited by 2
The use of open-ended questions to verify the mathematical literacy. Suggestions from PISA 2012
Asquini, G. (2016) · Form re - Open Journal per la formazione in rete · doi:10.13128/formare-17904 · Cited by 2
THE DEVELOPMENT OF KNOWLEDGE TESTS WHEN STUDYING ACADEMIC TEXTS IN HIGHER EDUCATION
Bruyne, E. D., Aesaert, K., & Valcke, M. (2017) · EDULEARN proceedings · doi:10.21125/edulearn.2017.0991 · Cited by 2
CEFR-based language requirements in the European labour market
Carlsen, C. H., Deygers, B., Zeidler, B., & VÎLCU, D. (2019) · Ghent University Academic Bibliography (Ghent University) · Cited by 2
Capobianco, R. (2020) · FORMAZIONE & INSEGNAMENTO. Rivista internazionale di Scienze dell'educazione e della formazione · doi:10.7346/-feis-xviii-01-20_13 · Cited by 2
Nicula, B., Perret, C. A., Dascălu, M., & McNamara, D. S. (2020) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai50040.2020.00061 · Cited by 2
Self-citationDesigning Technological Content Curriculum Materials Supported by Logger Pro: An Action Research
KUMAŞ, A. (2022) · International Journal of Progressive Education · doi:10.29329/ijpe.2022.426.9 · Cited by 2
Cognitive Processes and Memory Differences in Recall and Recognition in Adults
Fitzgerald, L. E. (2016) · ScholarWorks (Walden University) · Cited by 2
Effects of Bold Key Terms on Judgments of Learning and Reading Comprehension
Roberts, A. (2016) · Cited by 2
배민령 (2017) · Seoul National University Open Repository (Seoul National University) · Cited by 2
Automated software testing of relational database schemas
Alsharif, A. H. (2020) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York) · Cited by 2
Open-Ended Testing Stays Relevant for Critical Knowledge Assessment
Zrínyi, M. (2019) · COJ Nursing & Healthcare · doi:10.31031/cojnh.2019.05.000619 · Cited by 1
“Research, however, confirmed that OE and MCQs measure different characteristics of the thought process [5].”
BackgroundAn Explication and Classroom Field Study of the Virtual Human Interaction Lab’s Expert (VHIL-E) LLM
Bailenson, J. N., You, J., Markowitz, D. M., Petersen, G. B., Ratan, R., Santoso, M., & Wang, P. (2026) · Cyberpsychology Behavior and Social Networking · doi:10.1177/21522715261423752 · Cited by 1
“19 Similarly, research shows that LLM accuracy on MCQs drops signi fi cantly (up to * 40 percentage points) when the correct answer is replaced with “ none of the other answers ” , suggesting a strong reliance on pattern matching as opposed to reasoning.”
Watson, W., Cho, N., Ganesh, S., & Veloso, M. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.251 · Cited by 1
Pham, N. B. T., Zeng, Y., Fazil, K. F., & Said, N. E. M. (2026) · Language Testing in Asia · doi:10.1186/s40468-026-00444-6 · Cited by 1
Lepp, G. A., Fierke, K. K., Schullo-Feulner, A. M., & Esposito, E. R. (2026) · Currents in Pharmacy Teaching and Learning · doi:10.1016/j.cptl.2026.102650 · Cited by 1
Çalık, M., Bağ, H., Ültay, N., & Ayas, A. (2026) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-026-10673-z · Cited by 1
“Although multiple-choice questions facilitate students’ responding processes in a structured manner with a correct response and distracters (Ozuru et al., 2013), the quality and design of them may have influenced the measurement of the students’ academic performance through the treatments (Annisa &…”
Baskaran, R., Sirikonda, S., Singh, A., Srinivasan, S., Leveridge, B., Casals-Farre, O., Kaur, H., Manivannan, S., Elangovan, K., Quek, C. W. N., et al. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1888765 · Cited by 1
“Since GPT-4 demonstrates a significant increase in UKMLA questions when presented as MCQs, this indicates that its familiarity-based understanding has improved (35, 36).”
Pennington, S. E. (2016) · Digital Commons - University of South Florida (University of South Florida) · Cited by 1
Uzamsal Yetenek Belirlemek İçin Hangi Tür Sorular Kullanılmalıdır?
Kepceoğlu, İ., & Ercan, N. Ö. (2018) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.2990 · Cited by 1
Varghese, S. S. (2021) · International Journal of Dentistry and Oral Science · doi:10.19070/2377-8075-21000397 · Cited by 1
Ratajczak, M., Indrarathne, B., & Kormos, J. (2021) · Journal of the European Second Language Association · doi:10.22599/jesla.78 · Cited by 1
", GÜMÜŞ, F. Ö., Aktaş, G. S., & Karaca, H. (2022) · Acta Didactica Napocensia · doi:10.24193/adn.15.1.11 · Cited by 1
Micai, M. (2018) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1
Muzenda, T. (2019) · Open University of Cape Town (University of Cape Town) · Cited by 1
Effects of field dependence-independence and passive highlights on comprehension
Dodson, S. (2016) · cIRcle (University of British Columbia) · doi:10.14288/1.0300056 · Cited by 1
(2020) · Arizona State University Library Digital Repository (Arizona State University) · Cited by 1
The effect of mathematics anxiety, question complexity and time pressure on mathematics performance
Slaney, L. (2025) · Repository@Hull (Worktribe) (University of Hull) · Cited by 1
Strategies Among Health Care Business Professionals to Increase Productivity and Revenue
Cal, A. C. (2016) · ScholarWorks (Walden University)
Investigating the Difference in recall between Video/Audio learning and Read/Write learning.
Mamphaga, I. (2016) · TRAP@NCI (National College of Ireland)
Mitigating the Tensions of Teacher Evaluation through Pragmatism
Sheedlo, S. (2017) · Language arts journal of Michigan · doi:10.9707/2168-149x.2167
Singer, L. S. (2018) · Digital Commons - University of South Florida (University of South Florida)
Exploring Stages of Recovery from Crack Cocaine Addiction
Regan, Z. S. (2019) · ScholarWorks (Walden University)
Promoting Students' Understanding of Argumentation in Multiple Texts
Borlaug, P. M. K. (2019) · NVA (Nasjonalt vitenarkiv)
Anxiety in young children: cognitive biases, development and assessment
Ravenscroft, S. (2018) · CentAUR (University of Reading)
A Dilemma in Turkish Examination System: Open-Ended or Multiple-Choice?
Birgili, B., & Kiraz, E. (2017) · MEF University Institutional Repository (MEF University)
Does reduced cough capacity in minor thoracic injury is associated with atelectasis development?Preprint
Hegg-Deloye, S., Batomen, B., Thériault, E., Boucher, V., & Émond, M. (2021) · Research Square · doi:10.21203/rs.3.rs-199491/v1
“While the QCM are quick and easy to use in medicine, the literature questions their validity since the answers given seem to be correlated to the level of prior knowledge(25).”
BackgroundLinguistic Features of Lectures and their Relationship with Student Performance
Young, T. (2021) · UWSpace (University of Waterloo)
Scenarios in a serious game for training communication skills
Lala, R. (2021) · Utrecht University Repository (Utrecht University) · doi:10.33540/299
Text Adjuncts and Comprehension with University Level Second Language Readers
Liu, H. (2021) · Open Scholarship Institutional Repository (Washington University in St. Louis) · doi:10.7936/tc97-9175
Pekcici, E. B. B. (2020) · Research Square · doi:10.21203/rs.3.rs-36577/v1
“Page 12/17 questions has been found to correlate with quality of self-explanations, whereas performance on multiple-choice questions correlates with prior knowledge on topic[33].”
BackgroundApplication of High-Order Questions as an Objective Assessment Tool During Distance Learning
Duzbayeva, M., Tussupzhanov, A., Mukhamadiyeva, S., & Katanova, Y. (2022) · Journal of Educational Sciences · doi:10.26577/jes.2022.v71.i2.013
A Utopian Need: Artificial Intelligence Systems to Evaluate Open-Ended Questions
Birgili, B. (2019) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1427060
Nasjonal prøve i lesing på ungdomstrinnet for en gruppe norskfødte elever med innvandrerforeldre
Arntzen, R., & Kvifte, B. H. (2023) · Nordic Journal of Literacy Research · doi:10.23865/njlr.v9.2602
“Sammenhenger mellom resultater og egenskaper ved tekst og oppgave En vanlig måte å teste leseforståelse på er å kombinere flervalgsoppgaver og åpne oppgaver (Ozuru et al., 2013).”
BackgroundŠmida, D., Čipková, E., & Belka, M. (2024) · Biologie Chemie Zeměpis · doi:10.14712/25337556.2023.3.3
CORPORATE SOCIAL RESPONSIBILITY IN POLAND DURING THE COVID-19 PANDEMIC
Nowotarska, A. (2024) · Annals of the Polish Association of Agricultural and Agribusiness Economists · doi:10.5604/01.3001.0054.4321
Interpreting Latent Student Knowledge Representations in Programming AssignmentsPreprint
Fernandez, N., & Lan, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.08213
“Open-ended problems, which require students to produce free-form responses, either as short answers or essays [3] or even code [41], serve as a highly meaningful form of assessment and complements closed-form problems such as multiple-choice questions [29].”
Kocaarslan, M., Çelik, Z., & Yıldırım, K. N. (2024) · Ana Dili Eğitimi Dergisi · doi:10.16916/aded.1511386
Primor, L., Cerdán, R., Máñez, I., & Yeari, M. (2024) · Written Language & Literacy · doi:10.1075/wll.00082.pri
Ungdommers forståelse av mistenktes rettigheter og plikter i avhør
Brodshaug, M. F., Logendran, A., & Melinder, A. M. D. (2025) · Tidsskrift for strafferett · doi:10.18261/strafferett.25.3.4
Metacomprehension Accuracy of Health-Related Information
Chadwick, S. (2023) · Lancaster EPrints (Lancaster University) · doi:10.17635/lancaster/thesis/2213
Kreutzer, T. (2023) · YorkSpace (York University)
Malmström, F., & Ellen, A. K. (2025) · DiVA (Digitala Vetenskapliga Arkivet)
Factors Influencing the Adoption of Location Based Identification in the Kurdistan Region of Iraq
Amin, S. R. M. (2017) · University of Huddersfield Repository (University of Huddersfield)
The Effects of Response Format and Question Type on Reading Comprehension Test Performance
성수빈 (2023) · Seoul National University Open Repository (Seoul National University)
Teacher comprehension and ratings of school based psychological reports
Douglas, E. (2020) · Mspace (University of Manitoba)
Benedičič, E. (2019) · PeFprints (University of Ljubljana)
NC DOCKS at The University of North Carolina at Greensboro, & Walker, R. (2017) · NC Digital Online Collection of Knowledge and Scholarship (The University of North Carolina at Greensboro)
Watson, W., Cho, N., Ganesh, S., & Veloso, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.20300
Translingual Practices in English Language Education in South Asia
Sultana, S., Sah, P. K., Mishra, S., Malik, R. C., Boruah, P. B., Mukhopadhyay, L., Rafi, A. S. M., Phyak, P. B., Ghimire, N. B., Mohamed, N., et al. (2026) · OAPEN (The OAPEN Foundation)
Kyte, L., Eide, G. E., Berg, O. K. B., Stensrud, H., Meyer, M. E., Miland, Å. O., Simonsen, S. G., Totland, I., & Kvam, F. I. (2026) · Uniped · doi:10.18261/uniped.49.1.3
BizCompass: Benchmarking the Reasoning Capabilities of LLMs in Business Knowledge and Applications
Hao, J., Wu, Y., Xu, Y., Meng, S., Yuan, S. S., Zeng, W., Wang, Z., & Zhang, G. (2026) · CityUHK Scholars · doi:10.48550/arxiv.2604.17305
“In contrast, the Multiple-Choice Question (MCQ), a typical SR format, evaluates recognition-based processes that rely more heavily on prior knowledge and familiarity (Ozuru et al., 2013).”
Neri, N. C., Nagy, G., Heppt, B., & Retelsdorf, J. (2026) · Learning and Individual Differences · doi:10.1016/j.lindif.2026.102954
Third and fifth graders’ self-regulation in multiple digital documents reading
Troncoso‐Ruiz, A., van de Ven, M., & Segers, E. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101127
Criterium: Assisting Teachers in Fair and Consistent Grading of Open-Ended Questions
D’Urso, S., & Sciarrone, F. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0042-9_7
Framework for Use of Multiple-Choice Questions
Hutson, B., Schmauder, R., Daugherty, K. K., Eagle, D., McCarthy, K., McCauley, S., & Taylor, J. N. (2026) · Intersection A Journal at the Intersection of Assessment and Learning · doi:10.61669/001c.165518
“Critics of MCQs argue that MCQs fail to allow accurate assessment of a learner's higher-order cognitive abilities (e.g., Mehrens, 1987; Ozuru, et al., 2013).”
Seitz, T., Leipold, F. M., & Protte, J. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107587
Algeciras, K. A. M., Elosua, P., & Giménez, J. F. (2026) · Revista signos · doi:10.4151/s0718-09342026012101406
Morgan, K. (2026) · ScholarWorks @ UTRGV (The University of Texas Rio Grande Valley)
日本人英語学習者の複数テキスト読解における統合:テストにおける特徴, テキスト関係とタスク介入に焦点をおいて
MIKAMI, Y., ヨウスケ, ミ., & MIKAMI, Y. (2026) · Terrestrial Environment Research Center (University of Tsukuba)
2013 Author Recognition Bibliography
Grand Valley State University Libraries (2013) · Language arts journal of Michigan
Cross-national comparability of reading task formats in the DELE B2 test
Elosua, P., Algeciras, K. A. M., & Soria, I. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100362
Håndskrevne og digitale notater. En studie av læring fra fagvideo
Abdikarim, A. (2026) · SPISS
Effectiveness of IMBDSME program among patients with Type 2 Diabetes
Albelbisi, Z. (2021) · Nottingham ePrints (University of Nottingham)
Bogdan, T. (2025) · University of Maribor digital library (University of Maribor)
Is easy-to-read easy for everyone? An eye-tracking study across the adult lifespan
Navas-León, S., & Duñabeitia, J. A. (2026) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2026.1916339
“…comprehension measure may have shaped the results, as different assessment formats, such as recall versus recognition, impose different retrieval demands and can yield divergent outcomes (Cao and Kim, 2021; Ozuru et al., 2013); future studies should compare E2R effects across comprehension measures.”
Lúcio, P. S., Lourenço, F. C., Cogo-Moreira, H., Bandalos, D., de Carvalho, C. A. F., Kida, A. D. S. B., & de Ávila, C. R. B. (2021) · doi:10.3389/fpsyg.2021.662192.s001
(2020) · Universitätsbibliographie, Universität Duisburg-Essen
Metadata
- Authors
- Ozuru, Y., Briner, S., Kurby, C. A., & McNamara, D. S.
- Affiliations
- University of Illinois Chicago (US)
- Grand Valley State University (US)
- Arizona State University (US)
- Venue
- Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale
- Published
- 2013-09
- Volume
- 67
- Issue
- 3
- Pages
- 215-227
- ISSN
- 1196-1961 (ISSN-L)
- 1878-7290
- OpenAlex
- W2202297923
- Cited by
- 134
- References
- 37
- Citations by year
- 2026: 14 · 2025: 7 · 2024: 12 · 2023: 12 · 2022: 17 · 2021: 18 · 2020: 10 · 2019: 13 · 2018: 10 · 2017: 7 · 2016: 5 · 2015: 6 · 2014: 3
- FWCI
- 9.62
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- text comprehension, multiple-choice questions, open-ended questions, self-explanation, prior knowledge, dual-process theory, reading comprehension
- MeSH terms
- Choice Behavior, Female, Humans, Language, Male, Surveys and Questionnaires, Reading, Mental Recall, Semantics, Statistics as Topic, Students, Universities, Vocabulary, Knowledge, Comprehension, Self Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.