Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text
Open accessAbstract
Automatic question generation (QG) is a useful yet challenging task in NLP.Recent neural network-based approaches represent the stateof-the-art in this task.In this work, we attempt to strengthen them significantly by adopting a holistic and novel generator-evaluator framework that directly optimizes objectives that reward semantics and structure.The generator is a sequence-to-sequence model that incorporates the structure and semantics of the question being generated.The generator predicts an answer in the passage that the question can pivot on.Employing the copy and coverage mechanisms, it also acknowledges other contextually important (and possibly rare) keywords in the passage that the question needs to conform to, while not redundantly repeating words.The evaluator model evaluates and assigns a reward to each predicted question based on its conformity to the structure of ground-truth questions.We propose two novel QG-specific reward functions for text conformity and answer conformity of the generated question.The evaluator also employs structure-sensitive rewards based on evaluation measures such as BLEU, GLEU, and ROUGE-L, which are suitable for QG.In contrast, most of the previous works only optimize the cross-entropy loss, which can induce inconsistencies between training (objective) and testing (evaluation) measures.Our evaluation shows that our approach significantly outperforms state-of-the-art systems on the widelyused SQuAD benchmark as per both automatic and human evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reference [53] introduces two reward functions, QSS and ANSS, for evaluating generated question quality.
“The authors also introduce two new reward functions for evaluating the quality of the generated questions, namely Question Sentence Overlap Score (QSS) and Predicted and Encoded Answer Overlap Score (ANSS).”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [53]
The [53] model variant using ROUGE, QSS, and ANSS outperformed two state-of-the-art baselines on BLEU, METEOR, and ROUGE.
“After comparison, the authors observed that their model variant which included ROUGE, QSS, and ANSS outperformed the two state-of-the-art baselines on automatic evaluation using BLEU, METEOR, and ROUGE.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [53]
Reference [53] creates a deep reinforcement learning framework for generating intelligent questions in conversational systems.
“In [53], authors have attempted to create a framework for generating intelligent questions in the context of conversational systems.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
(2004) · Cited by 9,718 · see citing works
(2005) · PubMed · Cited by 7,011 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · neural information processing systems · Cited by 2,024 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Incorporating Copying Mechanism in Sequence-to-Sequence Learning
Gu, J., Lu, Z., Li, H., & Li, V. O. K. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1154 · Cited by 1,472 · see citing works
A Decomposable Attention Model for Natural Language Inference
Parikh, A. P., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,453 · see citing works
Deep Reinforcement Learning for Dialogue Generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1127 · Cited by 1,221 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
Modeling Coverage for Neural Machine Translation
Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1008 · Cited by 702 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · cited 12× · see citing works
“, 2016), our system considerably outperforms state-of-the-art question generation models (Du et al., 2017; Kumar et al., 2018; Song et al., 2018) in automatic and human evaluation.”
BackgroundMethodResultInfluentialRandolph, J. (2005) · Cited by 395 · see citing works
Deep Communicating Agents for Abstractive Summarization
Çelikyılmaz, A., Bosselut, A., He, X., & Choi, Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1150 · Cited by 327 · see citing works
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks
Zhao, Y., Ni, X., Ding, Y., & Ke, Q. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1424 · Cited by 288 · see citing works
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · see citing works
Leveraging Context Information for Natural Question Generation
Song, L., Wang, Z., Hamza, W., Zhang, Y., & Gildea, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2090 · Cited by 148 · see citing works
Question Generation from Paragraphs at UPenn: QGSTEC System Description
Mannem, P., Prasad, R., & Joshi, A. K. (2010) · Cited by 68 · see citing works
Difficulty-Controllable Multi-hop Question Generation from Knowledge Graphs
Kumar, V., Hua, Y., Ramakrishnan, G., Qi, G., Gao, L., & Li, Y.-F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30793-6_22 · Cited by 62 · see citing works
Self-citationCross-Lingual Training for Automatic Question Generation
Kumar, V., Joshi, N., Mukherjee, A., Ramakrishnan, G., & Jyothi, P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1481 · Cited by 58 · see citing works
Self-citationAutomating Reading Comprehension by Generating Question and Answer Pairs
Kumar, V., Boorla, K., Meena, Y. K., Ramakrishnan, G., & Li, Y.-F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93040-4_27 · Cited by 49 · see citing works
Self-citationTeaching Machines to Ask Questions
Yao, K., Zhang, L., Luo, T., Tao, L., & Wu, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/632 · Cited by 44 · cited 1× · see citing works
“(Yao et al., 2018) proposed generative adversarial network (GAN) framework with modified discriminator to predict question type.”
BackgroundA Reinforcement Learning Framework for Natural Question Generation using Bi-discriminators
Fan, Z., Wei, Z., Wang, S., Liu, Y., & Huang, X. (2018) · International Conference on Computational Linguistics · Cited by 36 · see citing works
Automating Reading Comprehension by Generating Question and Answer PairsPreprint
Kumar, V., Boorla, K., Meena, Y., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.03664 · Cited by 11 · see citing works
Self-citationSequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732
Flagged first, then by citations
Research works printed but not linked: 2
- Vishwajeet Kumar, N. Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. 2019b. Crosslingual training for automatic question generation. In ACL .
- Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointergenerator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages 1073-1083.
Cited by
Sorted by citations
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
- Authors
- Kumar, V., Ramakrishnan, G., & Li, Y.-F.
- Affiliations
- Indian Institute of Technology Bombay (IN)
- Monash University (AU)
- Venue
- Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
- Published
- 2019
- Pages
- 812-821
- OpenAlex
- W2983655111
- Cited by
- 46
- References
- 28
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 8 · 2022: 5 · 2021: 8 · 2020: 11 · 2019: 8
- FWCI
- 3.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Computational Natural Language Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- automatic question generation, question structure, sequence-to-sequence models, SQuAD, BLEU, ROUGE-L
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
