The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index.
Learning to Ask: Neural Question Generation for Reading Comprehension
Open accessAbstract
We study automatic question generation for sentences from text passages in reading comprehension.We introduce an attention-based sequence learning model for the task and investigate the effect of encoding sentence-vs.paragraph-level information.In contrast to all previous work, our model does not rely on hand-crafted rules or a sophisticated NLP pipeline; it is instead trainable end-to-end via sequenceto-sequence learning.Automatic evaluation results show that our system significantly outperforms the state-of-the-art rule-based system.In human evaluations, questions generated by our system are also rated as being more natural (i.e., grammaticality, fluency) and as more difficult to answer (in terms of syntactic and lexical divergence from the original text and reasoning needed to answer).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The authors performed both automatic evaluations using BLEU, METEOR, and ROUGE as well as human-based evaluation and found that their model outperformed the baseline model in [32] that they considered.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [32]
In the citing paper
“Other approaches make use of naturalness [32, 35] and difficulty [35].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [32]
In the citing paper
“However, the questions generated lack to a certain extent in terms of semantics and relevance as reported in most studies through human evaluation [32, 35, 36, 39, 56, 61].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [32]
In the citing paper
“While ITSs have traditionally been defined and constructed by hand, recent approaches have applied ML techniques such as multi-armed bandit techniques to adaptively personalize sequences of learning activities [135], LSTMs to generate questions to evaluate language comprehension [205], and RL to improve the strategies used within the ITS [365,441].”
From Tackling climate change with machine learning (2023) · cites this paper as [205]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“W s and W t are parameters to be learned. here, LSTM is the Long Short-Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997).”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“300d pre-trained embeddings (Pennington et al., 2014) for initialization.”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 2×
“We model the conditional probability us-ing RNN encoder-decoder architecture (Bahdanau et al., 2015; Cho et al., 2014), and adopt the global attention mechanism (Luong et al., 2015a) to make the model focus on certain elements of the input when generating each word during decoding.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“The package includes BLEU 1, BLEU 2, BLEU 3, BLEU 4 (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and ROUGE L (Lin, 2004) evaluation scripts.”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 4×
“More specifically, inspired by the recent success in neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), summarization (Rush et al.”
BackgroundInfluentialNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 4×
“We model the conditional probability us-ing RNN encoder-decoder architecture (Bahdanau et al., 2015; Cho et al., 2014), and adopt the global attention mechanism (Luong et al., 2015a) to make the model focus on certain elements of the input when generating each word during decoding.”
BackgroundMethodInfluentialBinary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387
(2004) · Cited by 9,718
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 3×
“We model the conditional probability us-ing RNN encoder-decoder architecture (Bahdanau et al., 2015; Cho et al., 2014), and adopt the global attention mechanism (Luong et al., 2015a) to make the model focus on certain elements of the input when generating each word during decoding.”
MethodShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“, 2016), and image caption generation (Xu et al., 2015), we tackle question generation using a conditional neural language model with a global attention mechanism (Luong et al.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,252 · cited 2×
“We first run Stanford CoreNLP (Manning et al., 2014) for pre-processing: tokenization and sentence splitting.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 11×
“In evaluations on the SQuAD dataset (Ra-jpurkar et al., 2016) using three automatic evaluation metrics, we find that our system significantly outperforms a collection of strong base-lines, including an information retrieval-based system (Robertson and Walker, 1994), a statistical machine translation approach (Koehn et…”
BackgroundMethodInfluentialELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1044 · Cited by 2,932 · cited 3×
“More specifically, inspired by the recent success in neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), summarization (Rush et al., 2015; Iyer et al., 2016), and image caption generation (Xu et al., 2015), we tackle question generation using a conditional neural language…”
BackgroundMethodTeaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“Hermann et al. (2015) released a corpus of cloze style questions by replacing entities with place-holders in abstractive summaries of CNN/Daily Mail news articles.”
BackgroundELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,807
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · cited 3×
“The package includes BLEU 1, BLEU 2, BLEU 3, BLEU 4 (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and ROUGE L (Lin, 2004) evaluation scripts.”
MethodSome simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval
Robertson, S., & Walker, S. (1994) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1007/978-1-4471-2099-5_24 · Cited by 1,947 · cited 4×
“, 2016) using three automatic evaluation metrics, we find that our system significantly outperforms a collection of strong baselines, including an information retrieval-based system (Robertson and Walker, 1994), a statistical machine translation approach (Koehn et al.”
BackgroundMethodInfluentialOpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,916
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 1×
“We use the evaluation package released by Chen et al. (2015), which was originally used to score image captions.”
MethodMoses : open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,547
Abstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1012 · Cited by 896 · cited 2×
“(2014) and Chopra et al. (2016), we factorize the the conditional in equation 1 into a product of word-level predictions:”
BackgroundMCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 1×
“Richardson et al. (2013) curated MCTest, in which crowdworker questions are paired with four answer choices.”
BackgroundSummarizing Source Code using a Neural Attention Model
Iyer, S. V., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1195 · Cited by 739 · cited 2×
“…inspired by the recent success in neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), summarization (Rush et al., 2015; Iyer et al., 2016), and image caption generation (Xu et al., 2015), we tackle question generation using a conditional neural language model with a…”
BackgroundMethodTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 1×
“For example, bAbI (Weston et al., 2016) is a fully synthetic dataset featuring 20 different tasks.”
BackgroundAddressing the Rare Word Problem in Neural Machine Translation
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2015) · doi:10.3115/v1/p15-1002 · Cited by 626 · cited 3×
“We model the conditional probability us-ing RNN encoder-decoder architecture (Bahdanau et al., 2015; Cho et al., 2014), and adopt the global attention mechanism (Luong et al., 2015a) to make the model focus on certain elements of the input when generating each word during decoding.”
MethodA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · cited 2×
“We use the evaluation package released by Chen et al. (2015), which was originally used to score image captions.”
BackgroundMethodScalable Modified Kneser-Ney Language Model Estimation
Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013) · Cited by 442
Good Question! Statistical Ranking for Question Generation
Heilman, M., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473390 · Cited by 366
Colby, K. M., Weber, S., & Hilf, F. D. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90002-6 · Cited by 317 · cited 1×
“, 2016)) or, arguably, as a clinical tool for evaluating or improving mental health (Weizenbaum, 1966; Colby et al., 1971).”
BackgroundGenerating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · cited 1×
“Serban et al. (2016) propose generating simple factoid questions from logic triple (subject, relation, object).”
BackgroundGenerating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · cited 3×
“Mostafazadeh et al. (2016) introduce visual question generation task, to explore the deep connection between language and vision.”
BackgroundMethodComputer-aided generation of multiple-choice tests
Mitkov, R., & Ha, L. A. (2003) · doi:10.3115/1118894.1118897 · Cited by 262 · cited 1×
“For the most part, question generation has been tackled in the past via rule-based approaches ( e.g. , Mitkov and Ha (2003); Rus et al. (2010).”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261
Deep Questions without Deep Understanding
Labutov, I., Basu, S., & Vanderwende, L. (2015) · doi:10.3115/v1/p15-1086 · Cited by 141 · cited 1×
“Labutov et al. (2015) use crowdsourcing to collect a set of templates and then rank the relevant templates for the text of another domain.”
BackgroundGenerating Natural Language Questions to Support Learning On-Line
Lindberg, D., Popowich, F. P., Nesbit, J. C., & Winne, P. H. (2013) · Cited by 122
Generating Instruction Automatically for the Reading Strategy of Self-Questioning
Jack, M., & Wei, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-465 · Cited by 101 · cited 1×
“A lot of research has focused on first manually constructing question templates, and then applying them to generate questions (Mostow and Chen, 2009; Lindberg et al., 2013; Mazidi and Nielsen, 2014).”
BackgroundThe first question generation shared task evaluation challenge
Rus, V., Brendan Wyse, A., Piwek, P., Lintean, M. C., Stoyanchev, S., & Moldovan, C. (2010) · Open Research Online (The Open University) · Cited by 95
Linguistic Considerations in Automatic Question Generation
Mazidi, K., & Nielsen, R. D. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-2053 · Cited by 88 · cited 2×
“A lot of research has focused on first manually constructing question templates, and then applying them to generate questions (Mostow and Chen, 2009; Lindberg et al., 2013; Mazidi and Nielsen, 2014).”
BackgroundThe Importance of Being Important: Question Generation
Vanderwende, L. (2008) · Cited by 34
Research works printed but not linked: 2
- Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attentionbased neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Lisbon, Portugal, pages 14121421. http://aclweb.org/anthology/D15-1166.
- Ruslan Mitkov and Le An Ha. 2003. Computeraided generation of multiple-choice tests. In Jill Burstein and Claudia Leacock, editors, Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing . pages 17-22. http://www.aclweb.org/anthology/W03-0203.pdf.
Cited by
Sorted by citations
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,068
“Considering the available baseline results, we use the Gigaword dataset (Rush et al., 2015) for abstractive summarization and the SQuAD 1.1 dataset (Rajpurkar et al., 2016) for question generation (Du et al., 2017) as the benchmarks for models pretrained on BookCorpus and Wikipedia.”
MethodInfluentialEvent Extraction by Answering (Almost) Natural Questions
Du, X., & Cardie, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.49 · Cited by 459
Self-citationProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training
Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., & Zhou, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.217 · Cited by 459
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Metadata
- Authors
- Du, X., Shao, J., & Cardie, C.
- Affiliations
- Cornell University (US)
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Cardie, Claire
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1342-1352
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2962717047
- Cited by
- 696
- References
- 42
- Citations by year
- 2026: 25 · 2025: 36 · 2024: 54 · 2023: 95 · 2022: 101 · 2021: 120 · 2020: 111 · 2019: 90 · 2018: 56 · 2017: 8
- FWCI
- 33.06
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- automatic question generation, reading comprehension, end-to-end learning, question difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
