Automatic Multiple Choice Question Generation From Text: A Survey
The DOI matches its Crossref record; 100 of the 106 references it cites were found in a registry or an index.
Abstract
Automatic multiple choice question (MCQ) generation from a text is a popular research area. MCQs are widely accepted for large-scale assessment in various domains and applications. However, manual generation of MCQs is expensive and time-consuming. Therefore, researchers have been attracted toward automatic MCQ generation since the late 90's. Since then, many systems have been developed for MCQ generation. We perform a systematic review of those systems. This paper presents our findings on the review. We outline a generic workflow for an automatic MCQ generation system. The workflow consists of six phases. For each of these phases, we find and discuss the list of techniques adopted in the literature. We also study the evaluation techniques for assessing the quality of the system generated MCQs. Finally, we identify the areas where the current research focus should be directed toward enriching the literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Additionally, while existing systems rely on textual context to generate reading-comprehension-like questions, [4, 30] we focus on generating a question well-aligned with the provided LO, allowing for easier large-scale evaluation.”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [4]
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Kumar, Banchs, and D’Haro [46] selected the distractors using semantic similarity computed through the word2vec tool [110].”
MethodAn introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880 · cited 1×
“Aldabe and Maritxalar [27] used Latent Semantic Analysis [108] to generate distractors.”
MethodIntroduction to WordNet: An On-line Lexical Database*
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990) · International Journal of Lexicography · doi:10.1093/ijl/3.4.235 · Cited by 4,197 · cited 1×
“WordNet: WordNet [105] is a lexical database that groups the words into sets of synonyms (called synsets), and records the relations among these synonym sets or their members.”
BackgroundJurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Automatic Labeling of Semantic Roles
Gildea, D., & Jurafsky, D. (2002) · Computational Linguistics · doi:10.1162/089120102760275983 · Cited by 1,626
Information filtering and information retrieval: two sides of the same coin?
Belkin, N. J., & Croft, W. B. (1992) · Communications of the ACM · doi:10.1145/138859.138861 · Cited by 1,308
Antoniou, G., & van Harmelen, F. (2004) · Handbook on Ontologies · doi:10.1007/978-3-540-24750-0_4 · Cited by 1,017
Interpreting TF-IDF term weights as making relevance decisions
Wu, H. C., Luk, R. W. P., Wong, K. F., & Kwok, K. L. (2008) · ACM Transactions on Information Systems · doi:10.1145/1361684.1361686 · Cited by 821 · cited 1×
“Statistical Analysis: Several systems used certain word level analysis including word frequency, n-gram frequency, Term frequency * inverse document frequency (Tf*Idf) [85], and co-occurrence statistics in various modules ([1], [6], [14], [16], [47], [49], [69], [76], [77], [86], [87], [88]).”
MethodStanford’s Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task
Lee, H., Peirsman, Y., Chang, A. L. S., Chambers, N., Surdeanu, M., & Jurafsky, D. (2011) · Lirias · Cited by 504
Using WordNet Based Context Vectors to Estimate the Semantic Relatedness of Concepts
Patwardhan, S., & Pedersen, T. (2006) · Cited by 375
Good Question! Statistical Ranking for Question Generation
Heilman, M., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473390 · Cited by 366
Normalization of non-standard words
Sproat, R., Black, A. W., Chen, S., Kumar, S., Ostendorf, M., & Richards, C. D. (2001) · Computer Speech & Language · doi:10.1006/csla.2001.0169 · Cited by 342 · cited 1×
“Text Normalization: Normalization refers to a conversion of the input text into the required format and removal of unnecessary content from the text [70].”
BackgroundMEAD - A Platform for Multidocument Multilingual Text Summarization
Radev, D., Allison, T., Blair-Goldensohn, S., Blitzer, J. C., Çelebi, A., Dimitrov, S., Drábek, E. F., Hakim, A., Lam, W. P., Liu, D., et al. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2nevc4eu3swt · Cited by 335
Computer-aided generation of multiple-choice tests
Mitkov, R., & Ha, L. A. (2003) · Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing - · doi:10.3115/1118894.1118897 · Cited by 252 · cited 8×
“We find the use of this pre-processing step in several systems including [6], [14], [76], [87].”
BackgroundMethodInfluentialAutomatic question generation for vocabulary assessment
Brown, J. C., Frishkoff, G. A., & Eskenazi, M. (2005) · Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing - HLT '05 · doi:10.3115/1220575.1220678 · Cited by 247 · cited 6×
“Frequency is used in [1], [14], [19], [49], [76], [99].”
BackgroundMethodInfluentialA computer-aided environment for generating multiple-choice test items
MITKOV, R., AN HA, L., & KARAMANIS, N. (2006) · Natural Language Engineering · doi:10.1017/s1351324906004177 · Cited by 229 · cited 10×
“We find the use of this pre-processing step in several systems including [6], [14], [76], [87].”
BackgroundMethodInfluentialAutomatic factual question generation from text
Smith, N. A., & Heilman, M. (2011) · Cited by 215
Analysis of Statistical Question Classification for Fact-Based Questions
Metzler, D., & Croft, W. B. (2005) · Information Retrieval · doi:10.1007/s10791-005-6995-3 · Cited by 139 · cited 1×
“Some authors named MCQs as factual questions ([6], [7], [8]).”
BackgroundGenerating Natural Language Questions to Support Learning On-Line
Lindberg, D., Popowich, F. P., Nesbit, J. C., & Winne, P. H. (2013) · Cited by 122
AUTOMATIC GENERATION OF MULTIPLE CHOICE QUESTIONS FROM DOMAIN ONTOLOGIES
Papasalouros, A., Kanaris, K., & Kotis, K. (2008) · Cited by 116
Sumita, E., Sugaya, F., & Yamamoto, S. (2005) · Proceedings of the second workshop on Building Educational Applications Using NLP - EdAppsNLP 05 · doi:10.3115/1609829.1609839 · Cited by 114 · cited 7×
“MCQ systems have been developed for a variety of applications, starting from language learning or vocabulary testing to assessment in active learning or e-learning framework ([6], [10], [11], [13], [14], [15], [16], [17], [18], [19], [20]).”
BackgroundMethodInfluentialSentence Simplification for Semantic Role Labeling
Vickrey, D., & Koller, D. (2008) · Cited by 110
Automatic Gap-fill Question Generation from Text Books
Agarwal, M., & Mannem, P. (2011) · Cited by 109
Generating Instruction Automatically for the Reading Strategy of Self-Questioning
Jack, M., & Wei, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-465 · Cited by 101 · cited 1×
“MCQ systems have been developed for a variety of applications, starting from language learning or vocabulary testing to assessment in active learning or e-learning framework ([6], [10], [11], [13], [14], [15], [16], [17], [18], [19], [20]).”
BackgroundNeural Models for Key Phrase Extraction and Question Generation
Subramanian, S., Wang, T., Yuan, X., Zhang, S., Trischler, A., & Bengio, Y. (2018) · Proceedings of the Workshop on Machine Reading for Question Answering · doi:10.18653/v1/w18-2609 · Cited by 95
Coniam, D. (1996) · CALICO Journal · doi:10.1558/cj.v14i2-4.15-33 · Cited by 94 · cited 7×
“Frequency is used in [1], [14], [19], [49], [76], [99].”
BackgroundMethodInfluentialLinguistic Considerations in Automatic Question Generation
Mazidi, K., & Nielsen, R. D. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2053 · Cited by 82 · cited 1×
“Mazidi and Nielsen [103] also used semantic role labeling based framework for question formation.”
MethodQuestion Generation from Paragraphs at UPenn: QGSTEC System Description
Mannem, P., Prasad, R., & Joshi, A. K. (2010) · Cited by 82
ArikIturri: An Automatic Question Generator Based on Corpora and NLP Techniques
Aldabe, I., de Lacalle, M. L., Maritxalar, M., Martinez, E., & Uria, L. (2006) · Lecture notes in computer science · doi:10.1007/11774303_58 · Cited by 79 · cited 4×
“Aldabe et al. [18], Agarwal and Mannem [49], Mitkov, Ha, and Karamanis [6], Chen, Liou, and Chang [15], Gates et al. [100] also used part-of-speech or parse information for key selection.”
BackgroundInfluentialAutomatic generation of multiple choice questions using dependency-based semantic relations
Afzal, N., & Mitkov, R. (2013) · Soft Computing · doi:10.1007/s00500-013-1141-4 · Cited by 79 · cited 5×
“Karamanis, Ha, and Mitkov [17], Mitkov et al. [107], Afzal and Mitkov [48], Smith, Sommers, and Kilgarriff [93] used distributional similarity based approach for distractor generation.”
BackgroundMethodInfluentialApplications of lexical information for algorithmically composing multiple-choice cloze items
Liu, C.-L., Wang, C.-H., Gao, Z.-M., & Huang, S.-M. (2005) · Proceedings of the second workshop on Building Educational Applications Using NLP - EdAppsNLP 05 · doi:10.3115/1609829.1609830 · Cited by 68 · cited 8×
“Frequency is used in [1], [14], [19], [49], [76], [99].”
BackgroundMethodInfluentialAutomatic generation system of multiple-choice cloze questions and its evaluation
Goto, T., Kojiri, T., Watanabe, T., Iwata, T., & Yamada, T. (2010) · Knowledge Management & E-Learning An International Journal · doi:10.34105/j.kmel.2010.02.016 · Cited by 67 · cited 8×
“Due to this property, the MCQs are also called as cloze questions in the literature ([6], [9], [10], [11], [12]).”
BackgroundMethodInfluentialGap-fill Tests for Language Learners: Corpus-Driven Item Generation
Smith, S., Avinesh, P. V. S., & Kilgarriff, A. (2010) · Coventry University Open Collections (Coventry university) · Cited by 67
A real-time multiple-choice question generation for language testing: a preliminary study
Hoshino, A., & Nakagawa, H. (2005) · Proceedings of the second workshop on Building Educational Applications Using NLP - EdAppsNLP 05 · doi:10.3115/1609829.1609832 · Cited by 65 · cited 4×
“Statistical Analysis: Several systems used certain word level analysis including word frequency, n-gram frequency, Term frequency * inverse document frequency (Tf*Idf) [85], and co-occurrence statistics in various modules ([1], [6], [14], [16], [47], [49], [69], [76], [77], [86], [87], [88]).”
BackgroundMethodInfluentialFAST: an automatic generation system for grammar tests
Chen, C.-Y., Liou, H.-C., & Chang, J. S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225404 · Cited by 65 · cited 6×
“For example, [1], [6], [15], [17], [18], [19], [49], [67], [68], [75], [76], [87], [89].”
BackgroundInfluentialOntology-Based Multiple Choice Question Generation
Alsubait, T., Bijan Parsia, A., & Sattler, U. (2015) · Künstliche Intell. · doi:10.1007/s13218-015-0405-9 · Cited by 64 · cited 1×
“([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12],…”
Semantic similarity of distractors in multiple-choice tests
Mitkov, R., Ha, L. A., Varga, A. E., & Rello, L. (2009) · doi:10.3115/1705415.1705422 · Cited by 63 · cited 2×
“Karamanis, Ha, and Mitkov [17], Mitkov et al. [107], Afzal and Mitkov [48], Smith, Sommers, and Kilgarriff [93] used distributional similarity based approach for distractor generation.”
BackgroundMultiple Choice Question Generation Utilizing An Ontology
Stasaski, K., & Hearst, M. A. (2017) · Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w17-5034 · Cited by 63 · cited 1×
“…grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active…”
Automatic generation of cloze items for prepositions
Lee, J., & Seneff, S. (2007) · Interspeech 2007 · doi:10.21437/interspeech.2007-592 · Cited by 62 · cited 3×
“Lee and Seneff [99], Agarwal and Mannem [49] used collocation based approach for distractor generation.”
BackgroundMethodGenerating multiple-choice test items from medical text: a pilot study
Karamanis, N., Ha, L. A., & Mitkov, R. (2006) · Proceedings of the Fourth International Natural Language Generation Conference on - INLG '06 · doi:10.3115/1706269.1706291 · Cited by 57 · cited 4×
“Some-times, Tf*Idf has been used instead of simple term frequency ([17], [89]).”
BackgroundInfluentialKnowledge Questions from Knowledge Graphs
Seyler, D., Yahya, M., & Berberich, K. (2017) · Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval · doi:10.1145/3121050.3121073 · Cited by 55 · cited 1×
“…[49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active research area.”
A System for Generating Multiple Choice Questions: With a Novel Approach for Sentence Selection
Majumder, M., & Saha, S. K. (2015) · Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications · doi:10.18653/v1/w15-4410 · Cited by 54 · cited 5×
“…[49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active research area.”
BackgroundMethodInfluentialSelf-citationAutomatic Chinese Factual Question Generation
Liu, M., Rus, V., & Liu, L. (2017) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2016.2565477 · Cited by 52 · cited 2×
“Some authors named MCQs as factual questions ([6], [7], [8]).”
BackgroundGenerating Diagnostic Multiple Choice Comprehension Cloze Questions
Mostow, J., & Jang, H. (2012) · Cited by 48
Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts.
Araki, J., Rajagopal, D., Sankaranarayanan, S., Holm, S., Yamakawa, Y., & Mitamura, T. (2016) · International Conference on Computational Linguistics · Cited by 48
Using cognitive models to develop quality multiple-choice questions
Pugh, D., De Champlain, A., Gierl, M., Lai, H., & Touchie, C. (2016) · Medical Teacher · doi:10.3109/0142159x.2016.1150989 · Cited by 47 · cited 1×
“…& vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning…”
Hill, J., & Simha, R. (2016) · Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w16-0503 · Cited by 46 · cited 1×
“Also automatic MCQ generation has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48],…”
BackgroundNeural Models for Key Phrase Detection and Question GenerationPreprint
Subramanian, S., Wang, T., Yuan, X., Zhang, S., Bengio, Y., & Trischler, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04560 · Cited by 45 · cited 1×
“However, there exist a few systems that take more than one sentences into account to form a question ([3], [4], [5]).”
BackgroundAutomatic Chinese Multiple Choice Question Generation Using Mixed Similarity Strategy
Liu, M., Rus, V., & Liu, L. (2018) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2017.2679009 · Cited by 44 · cited 1×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundQuestion Generation with Minimal Recursion Semantics
Yao, X. (2010) · Cited by 43
Automatic Distractor Generation for Domain Specific Texts
Aldabe, I., & Maritxalar, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14770-8_5 · Cited by 42 · cited 3×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundMethodAssisting cloze test making with a web application
Hoshino, A., & Nakagawa, H. (2007) · Cited by 41
On the Use of Semantic-Based AIG to Automatically Generate Programming Exercises
Zavala, L., & Mendoza, B. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159608 · Cited by 41 · cited 1×
“…science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]),…”
Automatic Question Tagging with Deep Neural Networks
Sun, B., Zhu, Y., Xiao, Y., Xiao, R., & Wei, Y. (2018) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2018.2808187 · Cited by 41 · cited 1×
“…language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56],…”
Automatic Generation of Multiple Choice Questions Using Wikipedia
Bhatia, A. S., Kirti, M., & Saha, S. K. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-45062-4_104 · Cited by 39 · cited 3×
“Statistical Analysis: Several systems used certain word level analysis including word frequency, n-gram frequency, Term frequency * inverse document frequency (Tf*Idf) [85], and co-occurrence statistics in various modules ([1], [6], [14], [16], [47], [49], [69], [76], [77], [86], [87], [88]).”
MethodSelf-citationAutomatic Factual Question Generation from Text
Heilman, M. (2025) · doi:10.1184/r1/28745492 · Cited by 39
Automatic Generation of English Vocabulary Tests
Susanti, Y., Iida, R., & Tokunaga, T. (2015) · International Conference on Computer Supported Education · doi:10.5220/0005437200770087 · Cited by 37 · cited 1×
“Also automatic MCQ generation has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48],…”
Background“Automatic generation of cloze question stems,”
Correia, R., Baptista, J., Eskenazi, M., & Mamede, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28885-2_19 · Cited by 35 · cited 3×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundMethodOntoQue: A Question Generation Engine for Educational Assesment Based on Domain Ontologies
Al-Yahya, M. M. (2011) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2011.124 · Cited by 34 · cited 1×
“…several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54],…”
The SIETTE Automatic Assessment Environment
Conejo, R., Guzmán, E., & Trella, M. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0078-4 · Cited by 34 · cited 1×
“…learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57],…”
Automatic fill-the-blank question generator for student self-assessment
Kumar, G., Banchs, R. E., & D’Haro, L. F. (2015) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2015.7344291 · Cited by 32 · cited 3×
“…learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10],…”
MethodFactual open cloze question generation for assessment of learner’s knowledge
Das, B., & Majumder, M. (2017) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-017-0060-3 · Cited by 32 · cited 1×
“Sentence simplification has been performed in various systems; like, [28], [48], [68], [75], [80], [81], [82].”
BackgroundSemantic Similarity Measures for the Generation of Science Tests in Basque
Aldabe, I., & Maritxalar, M. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2014.2355831 · Cited by 30 · cited 1×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundAutomatic question generation for intelligent tutoring systems
Shah, R., Shah, D., & Kurup, L. (2017) · 2017 2nd International Conference on Communication Systems, Computing and IT Applications (CSCITA) · doi:10.1109/cscita.2017.8066538 · Cited by 30 · cited 1×
“…has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51],…”
Automatic Cloze-Questions Generation
Narendra, A., Agarwal, M., & Shah, R. (2013) · International conference Recent advances in natural language processing · Cited by 28
Majumder, M., & Saha, S. K. (2014) · Knowledge Management & E-Learning An International Journal · doi:10.34105/j.kmel.2014.06.025 · Cited by 27 · cited 2×
“…[49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active research area.”
Self-citationPerez-Beltrachini, L., Gardent, C., Kruszewski, G., Exercises, G. G., & Lorraine, U. ́. D. (2012) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 27
FollowYou!: An Automatic Language Lesson Generation System
Shei, C.-C. (2001) · Computer Assisted Language Learning · doi:10.1076/call.14.2.129.5777 · Cited by 27 · cited 3×
“Statistical Analysis: Several systems used certain word level analysis including word frequency, n-gram frequency, Term frequency * inverse document frequency (Tf*Idf) [85], and co-occurrence statistics in various modules ([1], [6], [14], [16], [47], [49], [69], [76], [77], [86], [87], [88]).”
MethodImplementation and assessment of the automatic question generation module
Bednarik, L., & Kovács, L. (2012) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom.2012.6421938 · Cited by 27 · cited 3×
“Various types of text normalization and sentence normalization have been used in [71], [72], [73].”
MethodA Novel System for Generating Simple Sentences from Complex and Compound Sentences
Das, B., Majumder, M., & Phadikar, S. (2018) · International Journal of Modern Education and Computer Science · doi:10.5815/ijmecs.2018.01.06 · Cited by 26 · cited 1×
“Sentence simplification has been performed in various systems; like, [28], [48], [68], [75], [80], [81], [82].”
BackgroundEvaluation of automatically generated English vocabulary questions
Susanti, Y., Tokunaga, T., Nishikawa, H., & Obari, H. (2017) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-017-0051-y · Cited by 24 · cited 2×
“Also automatic MCQ generation has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science”
BackgroundMethodAutomatic Question Generation system
Pabitha, P., Mohana, M., Suganthi, S., & Sivanandhini, B. (2014) · International Conference on Recent Trends in Information Technology · doi:10.1109/icrtit.2014.6996216 · Cited by 23 · cited 2×
“Knowledge in Sentence: Pabitha et al. [84] also used a rule-based approach.”
MethodApplication of automatic thesaurus extraction for computer generation of vocabulary questions
Heilman, M., & Eskenazi, M. (2007) · Speech and Language Technology in Education (SLaTE 2007) · doi:10.21437/slate.2007-19 · Cited by 23 · cited 1×
“Sentence simplification has been performed in various systems; like, [28], [48], [68], [75], [80], [81], [82].”
BackgroundAutomatic generation of multiple choice questions for e-assessment
Santhanavijayan, A., Balasundaram, S. R., Narayanan, S. H., Kumar, S. V., & Prasad, V. V. (2017) · International Journal of Signal and Imaging Systems Engineering · doi:10.1504/ijsise.2017.10005435 · Cited by 22 · cited 1×
“…[48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active…”
Automatic generation of test questions by software agents using ontologies
Станчева, Н., Popchev, I. P., Stoyanova-Doycheva, A. G., & Stoyanov, S. N. (2016) · IEEE International Conference on Intelligent Systems · doi:10.1109/is.2016.7737395 · Cited by 20 · cited 1×
“…[49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active research area.”
WordGap - Automatic Generation of Gap-Filling Vocabulary Exercises for Mobile Learning
Knoop, S., & Wilske, S. (2013) · Cited by 18
How to Generate Cloze Questions from Definitions: A Syntactic Approach
Gates, D. (2011) · National Conference on Artificial Intelligence · Cited by 18
Item Difficulty Analysis of English Vocabulary Questions
Susanti, Y., Nishikawa, H., Tokunaga, T., & Hiroyuki, O. (2016) · Proceedings of the 8th International Conference on Computer Supported Education · doi:10.5220/0005775502670274 · Cited by 18 · cited 1×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundAutomatic generation of multiple choice questions for e-assessment
Santhanavijayan, A., Balasundaram, S. R., Narayanan, S. H., Kumar, S. V., & Prasad, V. V. (2017) · International Journal of Signal and Imaging Systems Engineering · doi:10.1504/ijsise.2017.084571 · Cited by 18
Stemming in the language modeling framework
Allan, J., & Kumaran, G. (2003) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/860435.860548 · Cited by 16 · cited 1×
“Therefore, stemming [83] that finds the root form of the word is performed in several systems.”
BackgroundAutomatic Generation of English Reference Question by Utilising Nonrestrictive Relative Clause
Satria, A. Y., & Tokunaga, T. (2017) · International Conference on Computer Supported Education · doi:10.5220/0006320203790386 · Cited by 16 · cited 1×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundA Study on the Automatic Selection of Candidate Sentences Distractors
Aldabe, I., Maritxalar, M., & Mitkov, R. (2009) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-028-5-656 · Cited by 15 · cited 4×
“Statistical Analysis: Several systems used certain word level analysis including word frequency, n-gram frequency, Term frequency * inverse document frequency (Tf*Idf) [85], and co-occurrence statistics in various modules ([1], [6], [14], [16], [47], [49], [69], [76], [77], [86], [87], [88]).”
BackgroundMethodInfluentialAn Automatic Quiz Generation System for English Text
Sung, L.-C., Lin, Y.-C., & Chen, M. C. (2007) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2007.56 · Cited by 15 · cited 1×
“Sung, Lin, and Chen [101] used a semantic network structure to represent the relationship among the players, actions, and attributes; and used this relationship to select the key.”
BackgroundAutomatic multiple-choice question generation from Thai text
Kwankajornkiet, C., Suchato, A., & Punyabukkana, P. (2016) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse.2016.7748891 · Cited by 15 · cited 1×
“…([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67],…”
An automatic generator and corrector of multiple choice tests with random answer keys
Zampirolli, F. D. A., Batista, V. R., & Quilici-Gonzalez, J. A. (2016) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2016.7757467 · Cited by 15 · cited 1×
“…history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports &…”
A Survey on Paraphrase Detection Techniques for Indian Regional Languages
Srivastava, S., & Govilkar, S. S. (2017) · International Journal of Computer Applications · doi:10.5120/ijca2017913757 · Cited by 14 · cited 1×
“Araki et al. [3] used various semantic processing including coreference resolution [95], paraphrase detection [96], and extraction of a relationship between the concepts.”
BackgroundFattoh, I. E. (2014) · Computer Engineering and Intelligent Systems · Cited by 13
Automatically Generating and Validating Reading-Check Questions
Feeney, C. M., & Heilman, M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69132-7_69 · Cited by 11 · cited 1×
“MCQ systems have been developed for a variety of applications, starting from language learning or vocabulary testing to assessment in active learning or e-learning framework ([6], [10], [11], [13], [14], [15], [16], [17], [18], [19], [20]).”
BackgroundA System for Generating Cloze Test Items from Russian-Language Text
Михайлович, К. А. (2013) · International conference Recent advances in natural language processing · Cited by 11
WebExperimenter for multiple-choice question generation
Hoshino, A., & Nakagawa, H. (2005) · doi:10.3115/1225733.1225743 · Cited by 10 · cited 1×
“…MCQ generation has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]),…”
BackgroundA Survey on Plagiarism Detection Techniques for Indian Regional Languages
Lamba, H., & Govilkar, S. S. (2017) · International Journal of Computer Applications · doi:10.5120/ijca2017913622 · Cited by 10
SARAC: A Framework for Automatic Item Generation
Liu, B. (2009) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2009.47 · Cited by 9 · cited 1×
“…([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48], [49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12],…”
Evaluation of Automatically Generated Pronoun Reference Questions
Satria, A. Y., & Tokunaga, T. (2017) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w17-5008 · Cited by 9 · cited 1×
“Also automatic MCQ generation has been performed in several domains, including, language learning ([5], [33], [34]), grammar & vocabulary learning ([35], [36], [37]), science ([38], [39]), history ([40], [41]), general science ([42], [43]), biology & medical ([9], [44], [45], [46], [47], [48],…”
BackgroundEvaluating and improving the distractor-generating heuristics
Aldabe, I., Maritxalar, M., & Martinez, E. (2007) · Cited by 8
AUTOMATIC GENERATION OF ENGLISH LANGUAGE TEST QUESTIONS USING MATHEMATICA
Malinova, A., & Rahneva, O. (2016) · CBU International Conference Proceedings · doi:10.12955/cbup.v4.794 · Cited by 7 · cited 1×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundChu, M.-H., Chen, W.-Y., & Lin, S.-D. (2012) · Conference on Technologies and Applications of Artificial Intelligence (TAAI) · doi:10.1109/taai.2012.38 · Cited by 5 · cited 2×
“In the literature, we found MCQ systems exist in many languages including English ([21], [22], [23], and many more), Portuguese ([24], [25]), Basque ([26], [27], [28]), French [29], Russian [30] and Chinese ([7], [31], [32]).”
BackgroundA Brief Introduction to Word Sense Disambiguation
Pen, J. (2010) · Terminology Standardization & Information Technology · Cited by 2
Web authoriser tool to build assessments using Wikipedia articles
Adithya, S. S. R., & Singh, P. K. (2017) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon.2017.8227909 · Cited by 2 · cited 1×
“…[49]), technology ([50], [51], [52], [53], [54], [55]), generic domain ([32], [56], [57], [58]), e-learning & active learning ([10], [11], [59], [60], [61], [62], [63]), sports & entertainment ([12], [64], [65], [66], [67], [68], [69]) etc. Automatic MCQ generation is still an active research…”
Automatic Generation of Question Bank Based on Pre-defined Templates
Awad, A. E., & Dahab, M. Y. (2014) · doi:10.13140/2.1.3633.3766 · Cited by 2
Research works printed but not linked: 11
- hoshino, 2010, Predicting the difficulty of multiple-choice close questions for computer-adaptive testing, Proc CICLing, 279
- correia, 2010, Automatic generation of cloze question distractors, Proc Interspeech Satellite Workshop 2nd Lang Stud Acquisition Learn Edu Technol, 1
- stancheva, 2017, A model for generation of test questions, Comptes Rendus de L'Academie Bulgare des Sciences, 70, 619
- bhale, 0, Automatic question generation using Wikipedia, Imperial Journal of Interdisciplinary Research (IJIR), 2, 943
- agarwal, 2012, Cloze and open cloze question generation systems and their evaluation guidelines, MS Thesis IIIT Hyderabad Hyderabad INDIA
- pino, 2008, A selection strategy to improve cloze question quality, Proc Workshop Intell Tutoring Syst Ill-Defined Domains 9th Int Conf Intell Tutoring Syst, 22
- kojiri, 2010, Statistical learning-based approach for automatic generation system of multiple-choice cloze questions, Proc Workshop Proc 18th Int Conf Comput Edu
- sung, 2007, The design of automatic quiz generation for ubiquitous English e-learning system, Proc Conf Technol Enhanced Learn, 161
- nielsen, 2008, A taxonomy of questions for question generation, Workshop on the Question Generation Shared Task and Evaluation Challenge, 15
- goto, 2009, An automatic generation of multiple-choice cloze questions based on statistical learning, Proc 17th Int Conf Comput Edu, 415
- lin, 2007, An automatic multiple-choice question generation scheme for English adjective understanding, Proc Workshop Model Manage Gener Problems/Questions eLearn 15th Int Conf Comput Edu, 137
Cited by
A Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514
Diwan, C., Srinivasa, S., Suri, G., Agarwal, S., & Ram, P. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100110 · Cited by 106
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
“Note that some of the LOs extracted from the courses were not well-defined (e.g., no action verb) and, hence, it was not possible to categorize them in terms of Bloom’s taxonomy.”
BackgroundMCQGen: A Large Language Model-Driven MCQ Generator for Personalized Learning
Hang, C. N., Tan, C. W., & Yu, P.-D. (2024) · IEEE Access · doi:10.1109/access.2024.3420709 · Cited by 151
“Research on automatic MCQ generation began over two decades ago, and since then, a significant amount of effort has been dedicated to its development [5].”
BackgroundAutomatic question generation and answer assessment: a survey
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-021-00151-1 · Cited by 123
Metadata
- Authors
- CH, D. R., & Saha, S. K.
- Affiliations
- Birla Institute of Technology, Mesra (IN)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2020-01-01
- Volume
- 13
- Issue
- 1
- Pages
- 14-25
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W2906432682
- Cited by
- 144
- References
- 106
- Citations by year
- 2026: 15 · 2025: 29 · 2024: 29 · 2023: 25 · 2022: 20 · 2021: 17 · 2020: 8
- FWCI
- 3.64
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- Funders
- Science and Engineering Research Board (YSS/2015/001948)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- large-scale assessment, automated assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.