Distractor Generation with Generative Adversarial Nets for Automatically Creating Fill-in-the-blank Questions
Abstract
Distractor generation is a crucial step for fill-in-the-blank question generation. We propose a generative model learned from training generative adversarial nets (GANs) to create useful distractors. Our method utilizes only context information and does not use the correct answer, which is completely different from previous Ontology-based or similarity-based approaches. Trained on the Wikipedia corpus, the proposed model is able to predict Wiki entities as distractors. Our method is evaluated on two biology question datasets collected from Wikipedia and actual college-level exams. Experimental results show that our context-based method achieves comparable performance to a frequently used word2vec-based method for the Wiki dataset. In addition, we propose a second-stage learner to combine the strengths of the two methods, which further improves the performance on both datasets, with 51.7% and 48.4% of generated distractors being acceptable.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reference [39] addresses fill-in-the-blank question generation by using GANs to create distractors.
“The problem of fill-in-the-blank (FITB)-type question generation is dealt with in [39]. The creation of important distractors is done using generative adversarial networks for training.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [39]
The method of [39] performed better than existing similarity-based methods.
“The proposed method performed better than already existing similarity-based methods.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [39]
Generated questions lack semantics and relevance to some extent, as reported in most studies through human evaluation.
“However, the questions generated lack to a certain extent in terms of semantics and relevance as reported in most studies through human evaluation [32, 35, 36, 39, 56, 61].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [39]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 2× · see citing works
“FortheDG problem, many systems utilize WordNet [19] to find synonyms or other related words as distractors [21].”
BackgroundMethodConditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863 · see citing works
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616 · see citing works
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,342 · see citing works
The Concrete Distribution: A Continuous Relaxation of Discrete Random VariablesPreprint
Maddison, C. J., Mnih, A., & Teh, Y. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00712 · Cited by 1,771 · see citing works
A computer-aided environment for generating multiple-choice test items
MITKOV, R., AN HA, L., & KARAMANIS, N. (2006) · Natural Language Engineering · doi:10.1017/s1351324906004177 · Cited by 229 · see citing works
Maddison, C. J., Tarlow, D., & Minka, T. (2014) · neural information processing systems · Cited by 154 · see citing works
Sumita, E., Sugaya, F., & Yamamoto, S. (2005) · doi:10.3115/1609829.1609839 · Cited by 117 · see citing works
Automatic Gap-fill Question Generation from Text Books
Agarwal, M., & Mannem, P. (2011) · Cited by 99 · see citing works
Multiple Choice Question Generation Utilizing An Ontology
Stasaski, K., & Hearst, M. A. (2017) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w17-5034 · Cited by 65 · see citing works
Automatic generation of cloze items for prepositions
Lee, J., & Seneff, S. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-592 · Cited by 64 · see citing works
Generating multiple-choice test items from medical text
Karamanis, N., Ha, L. A., & Mitkov, R. (2006) · doi:10.3115/1706269.1706291 · Cited by 57 · see citing works
RevUP: Automatic Gap-Fill Question Generation from Educational Texts
Kumar, G., Banchs, R. E., & D’Haro, L. F. (2015) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.3115/v1/w15-0618 · Cited by 51 · see citing works
Mind the Gap: Learning to Choose Gaps for Question Generation
Becker, L. A., Basu, S., & Vanderwende, L. (2012) · Cited by 49 · see citing works
Hill, J., & Simha, R. (2016) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w16-0503 · Cited by 47 · see citing works
Questimator: generating knowledge assessments for arbitrary topics
Guo, Q., Kulkarni, C. E., Kittur, A., Bigham, J. P., & Brunskill, E. (2016) · International Joint Conference on Artificial Intelligence · Cited by 33 · see citing works
Semi-automatic generation of cloze question distractors effect of students² L1
Pino, J., & Eskénazi, M. (2009) · ISCA Workshop on Speech and Language Technology in Education (SLaTE) · doi:10.21437/slate.2009-27 · Cited by 29 · see citing works
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514 · see citing works
“Others, including Liang et al. (2017, 2018) and Liu et al. (2018), used ML-models to rank distractors based on a combination of the previous features.”
MethodMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
“The problem of fill-in-the-blank (FITB)-type question generation is dealt with in [39].”
BackgroundDistractor Generation for Multiple Choice Questions Using Learning to Rank
Liang, C., Yang, X., Dave, N., Wham, D. C., Pursel, B., & Giles, C. L. (2018) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w18-0533 · Cited by 67 · see citing works
“Prior methods were evaluated on different question sets collected from textbooks (Agarwal and Mannem, 2011), Wikipedia (Liang et al., 2017), ESL corpuses (Sakaguchi et al.”
MethodInfluentialSelf-citationGenerating Distractors for Reading Comprehension Questions from Real Examinations
Gao, Y., Bing, L., Li, P., King, I., & Lyu, M. R.-T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016423 · Cited by 65 · see citing works
Quiz-Style Question Generation for News Stories
Lelkes, A. D., Tran, V. Q., & Yu, C. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449892 · Cited by 47 · see citing works
“There has also been work that used GANs for distractor generation [27].”
BackgroundKnowledge-Driven Distractor Generation for Cloze-Style Multiple Choice Questions
Ren, S., & Zhu, K. Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16559 · Cited by 43 · see citing works
Rizvi, S. K. J., Azad, M. A., & Fraz, M. M. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09543-4 · Cited by 37 · see citing works
“[67] 2017 Create useful distractors Advantage: achieves comparable performance to a frequently used word2vec-based method for the Wiki dataset”
BackgroundMethodChung, H.-L., Chan, Y.-H., & Fan, Y.-C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.393 · Cited by 37 · see citing works
Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
Zhou, X., Luo, S., & Wu, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6522 · Cited by 34 · see citing works
Learning to Reuse Distractors to Support Multiple-Choice Question Generation in Education
Bitew, S. K., Hadifar, A., Sterckx, L., Deleu, J., Develder, C., & Demeester, T. (2022) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3226523 · Cited by 33 · see citing works
“Other works such as [39, 35, 36, 38] use machine learning models to generate distractors by using a combination of the previous features and other types of information such as tf-idf scores.”
MethodMaurya, K. K., & Desarkar, M. S. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411997 · Cited by 28 · see citing works
“The model proposed in [14] learns distractor-distribution conditioned on the question using generative adversarial nets (GANs).”
BackgroundRevita: a Language-learning Platform at the Intersection of ITS and CALL
Katinskaia, A., Nouri, J., & Yangarber, R. (2018) · International Conference on Language Resources and Evaluation · doi:10.63317/5f2yjftgogkj · Cited by 37 · see citing works
“Various approaches to the problem of generation of reliable distractors are described in (Lee and Luo, 2016; Correia et al., 2010; Rakangor and Ghodasara, 2014; Sakaguchi et al., 2013; Hill and Simha, 2016; Liang et al., 2017).”
BackgroundDeena, G., Raja, K., Banu, P. K. N., & Kannan, K. (2020) · International Journal of Service Science Management Engineering and Technology · doi:10.4018/ijssmet.2020040106 · Cited by 11 · see citing works
Mulla, N., & Gharpure, P. (2023) · International Journal of Information Technology · doi:10.1007/s41870-023-01262-2 · Cited by 10 · see citing works
Lu, J., Ye, X., Ren, Y. C., & Yang, Y. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00539 · Cited by 11 · see citing works
Ranking Multiple Choice Question Distractors using Semantically Informed Neural Networks
Sinha, M., Dasgupta, T., & Mandav, J. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3417468 · Cited by 6 · see citing works
“…similarities [5] [10][8], n-gram co-occurrence likelihood [6], phonetic and morphological similarities [16], structural similarities in an ontology [7] [19] [4] [1] [17] [2][15] [3] [13][12] [18], context similarity [16], context-sensitive inference [21], and syntactic similarity [5].”
BackgroundLin, R., Xu, D., Wang, H., Chen, Z., Wang, Y., & Liu, S. (2024) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc54092.2024.10831835 · Cited by 2 · see citing works
“Concurrently, other works explore applying neural network-based methods in selecting distractors [9]–[12].”
BackgroundDong, X., Li, Z., Su, H., Xue, J., & Dang, X. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11237-3 · Cited by 2 · see citing works
“Early works perceived DG as a classification problem (Liang et al. 2017; Sakaguchi et al. 2013).”
Using Learning Analytics for Adaptive Exercise Generation
Heck, T., & Meurers, D. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.4 · Cited by 1 · see citing works
“else generated based on machine learning (Liang et al., 2017; Sakaguchi et al., 2013) or on transformation rules (Žitko et al.”
MethodBhuvan, N. T., Jisha, G., & Shamna, N. V. (2024) · Intelligent Decision Technologies · doi:10.3233/idt-230629 · Cited by 1 · see citing works
“In various works [27,28,29,30,31], and [32], several similarities have been used, namely word2vec similarity, WordNet similarity, n-gram co-occurrence likelihood, and so on.”
Constructing Cloze Questions Generatively
Sun, Y., & Wang, J. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191481
“Researchers also considered semantic relatedness of distractors with the whole stem and domain restrictions [15], [16], and explored machine-learning ranking models to select distractors and quantitatively evaluate the top generated distractors [2], [17], [18].”
BackgroundQadg: Generating question–answer-distractors pairs for real examination
Zhou, H. Y., & Li, L. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10658-5
Distractor Generation for Multiple-Choice Question Using Existing Distractor
Higa, K., Watanabe, T., & Baba, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7178-6_17
- Authors
- Liang, C., Yang, X., Wham, D., Pursel, B., Passonneaur, R., & Giles, C. L.
- Affiliations
- Pennsylvania State University (US)
- Venue
- Proceedings of the Knowledge Capture Conference
- Publisher
- ACM
- Published
- 2017-12-04
- Pages
- 1-4
- OpenAlex
- W2781428482
- Cited by
- 23
- References
- 23
- Citations by year
- 2025: 2 · 2024: 3 · 2023: 4 · 2022: 2 · 2021: 3 · 2020: 5 · 2019: 2 · 2018: 2
- FWCI
- 1.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Knowledge Capture (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- distractor generation, generative adversarial networks, automatic question generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
