Infusing NLU into Automatic Question Generation
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; none of the 10 sources its claims cite is checked for support yet.
Abstract
We present a fresh approach to automatic question generation that significantly increases the percentage of acceptable questions compared to prior state-of-the-art systems.In our evaluation of the top 20 questions, our system generated 71% more acceptable questions by informing the generation process with Natural Language Understanding techniques.The system also introduces our DeconStructure algorithm which creates an intuitive and practical structure for easily accessing sentence functional constituents in NLP applications.
Overview
CoChat Summary
## TL;DR
The paper presents an automatic question generation system that infuses Natural Language Understanding into generation by classifying sentence patterns with DeconStructure before applying templates. In evaluation on the top 20 questions per passage, it raised acceptable questions from 42% to 72% versus Heilman and Smith (2011), a 71% increase.
## Key findings
- Across 10 open-source textbook passages, the system’s average MTurk rating for the top 20 questions was 3.7, compared with 2.9 for Heilman & Smith (2011), and the difference was statistically significant (p < 0.001).
- Using an acceptability threshold of > 3.0, 72% of the paper’s questions were acceptable versus 42% for the prior system.
- The paper reports that this is a 71% increase in acceptable questions among the top-ranked questions.
- The system generated far fewer questions than Heilman & Smith: the dataset table reports an average of 50 questions per passage for this system versus 406 for H&S, with generation percentages of 60% and 488%, respectively.
## Methods
- The system first applies the DeconStructure algorithm: it parses each sentence with dependency parsing and semantic role labeling, and also gathers lemmas, POS tags, and named entities.
- It then divides sentences into independent clauses, labels clause components, and matches the resulting sentence patterns against approximately 60 templates; filtering conditions check features such as verb type and voice.
- Evaluation used 10 science and humanities passages from OpenStax and Saylor textbooks, with each generated question rated on a 1–5 scale by 4 Amazon Mechanical Turk workers and compared with Heilman & Smith (2011) on the top 20 questions per input file.
## Limitations
- The paper states that some errors come from idiomatic language, where literal parsing yields awkward questions; it gives “What do few members spend?” as an example.
- It also says some templates work better for some topics than others, and some generated questions are vague out of context when a sentence is analyzed in isolation.
- Evident from the text: the evaluation is limited to 10 textbook passages and the top 20 questions from each system, so the reported results reflect a narrow slice of output rather than all generated questions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 9 Not checked · 30 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our evaluation of the top 20 questions, our system generated 71% more acceptable questions by informing the generation process with Natural Language Understanding techniques.”
- Also stated in: § 5.2 Results · plain
“This is an increase in the acceptablity percentage of the top questions of 71%.”
- Also stated in: § 6 Discussion · plain
“The QG system achieved a 71% increase in the percentage of acceptable questions from among the top system-ranked questions compared to the most cited prior state-of-the-art system.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5.2 Results · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, our system generates an average closer to one question per every 2 input sentences by focusing on the important content in each sentence but not generating questions when conditions are not favorable for generating a good question.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another problem is insufficient preprocessing to remove sentences such as: Different episodes of monetary policy are indicated in the figure , which generated the question: Where are different episodes of monetary policy indicated?”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another issue is that some templates work better with some topics other than others.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our system had a higher rating for every topic.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
8 more claims
Create a free account to see all 40 claims and to verify the 10 references still open.
- Probably not a substantial claim. 23
- It may state two separate claims as one. 11
Citations
Cites
TextRank: Bringing Order into Text
Mihalcea, R. F., & Tarau, P. (2004) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 3,631
Self-citationCheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887 · cited 1×
“Recent work in QG and other NLP applications favors evaluation by crowdsourcing which has proven to be both cost and time ef-ficient and to achieve results comparable to human evaluators (Snow et al., 2008; Heilman and Smith, 2010).”
ResultNatural Language Understanding
(2021) · AI Assistants · doi:10.7551/mitpress/12761.003.0007 · Cited by 1,271
Syntax: A Generative Introduction
Carnie, A. (2002) · Cited by 735
Universal Dependency Annotation for Multilingual Parsing
McDonald, R., Nivre, J., Quirmbach-Brundage, Y., Goldberg, Y., Das, D., Ganchev, K., Hall, K., Petrov, S., Zhang, H., Täckström, O., et al. (2013) · Cited by 716
Analyzing Grammar: An Introduction
Kroeger, P. R. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511801679 · Cited by 663
Swan, M. (2007) · ELT Journal · doi:10.1093/elt/ccl048 · Cited by 575
Ginzburg, J. (2012) · doi:10.1093/acprof:oso/9780199697922.001.0001 · Cited by 389
Carter (2006) · Cited by 361
Automatic factual question generation from text
Smith, N. A., & Heilman, M. (2011) · Cited by 215
Analyzing Grammar: An Introduction
Kroeger, P. R. (2005) · Cited by 187
NATURAL LANGUAGE UNDERSTANDING
(2021) · The Myth of Artificial Intelligence · doi:10.2307/j.ctv322v43j.8 · Cited by 159
Deep Questions without Deep Understanding
Labutov, I., Basu, S., & Vanderwende, L. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1086 · Cited by 132 · cited 1×
“Another recent innovative approach (Labutov et al., 2015) used crowd sourcing to develop QG templates by leveraging the structure of their source data, Wikipedia.”
Generating Natural Language Questions to Support Learning On-Line
Lindberg, D., Popowich, F. P., Nesbit, J. C., & Winne, P. H. (2013) · Cited by 122
Automatic Question Generation for Educational Applications – The State of Art
Le, N.-T., Kojiri, T., & Pinkwart, N. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-06569-4_24 · Cited by 85 · cited 1×
“In a recent survey of question generation approaches for educational applications, Le et al. (2014) observed that template-based approaches tended to perform better than systems that syntactically rearrange the source text.”
BackgroundLinguistic Considerations in Automatic Question Generation
Mazidi, K., & Nielsen, R. D. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2053 · Cited by 82 · cited 1×
“A QG system can then extract arguments and modifiers for question construction (Mannem et al., 2010; Lindberg et al., 2013; Mazidi and Nielsen, 2014; Chali and Hasan, 2015).”
BackgroundSelf-citationQuestion Generation from Paragraphs at UPenn: QGSTEC System Description
Mannem, P., Prasad, R., & Joshi, A. K. (2010) · Cited by 82
G-Asks: An Intelligent Automatic Question Generation System for Academic Writing Support
Liu, M., Calvo, R. A., & Rus, V. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.205 · Cited by 77 · cited 2×
“…approaches involves parsing text with a PSG (phrase structure grammar) parser and then forming questions using templates (Rus et al., 2007; Wyse and Piwek, 2009; Liu et al., 2010; Liu et al., 2012) or transformation rules and tree manipulation tools (Gates, 2008; Heilman, 2011; Ali et al., 2010).”
MethodAutomatic question generation from text - an aid to independent study
Wolfe, J. H. (1976) · Proceedings of the ACM SIGCSE-SIGCUE technical symposium on Computer science and education - · doi:10.1145/800107.803459 · Cited by 66
Automatic Question Generation for Literature Review Writing Support
Liu, M., Calvo, R. A., & Rus, V. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_9 · Cited by 64 · cited 1×
“…parsing text with a PSG (phrase structure grammar) parser and then forming questions using templates (Rus et al., 2007; Wyse and Piwek, 2009; Liu et al., 2010; Liu et al., 2012) or transformation rules and tree manipulation tools (Gates, 2008; Heil-man, 2011; Ali et al., 2010 Arg1 acomp…”
MethodGenerating Questions from OpenLearn study units
Brendan Wyse, A., & Piwek, P. (2009) · Open Research Online (The Open University) · Cited by 49
Experiments on Generating Questions About Facts
Rus, V., Cai, Z., & Graesser, A. C. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-70939-8_39 · Cited by 46
Rating Computer-Generated Questions with Mechanical Turk
Heilman, M., & Smith, N. A. (2010) · KiltHub Repository · doi:10.1184/r1/6473645 · Cited by 39
The Importance of Being Important: Question Generation
Vanderwende, L. (2008) · Cited by 35
Leveraging Multiple Views of Text for Automatic Question Generation
Mazidi, K., & Nielsen, R. D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_26 · Cited by 34 · cited 1×
“Although the dependency parse had been used as an ancilliary tool and for sentence simpli-fication, Mazidi et al. (2015) was the first to fully exploit dependency relations in question generation.”
MethodSelf-citationA Detailed Account of The First Question Generation Shared Task Evaluation Challenge
Rus, V., Wyse, B., Piwek, P., Lintean, M., Stoyanchev, S., & Moldovan, C. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.208 · Cited by 33
MSR SPLAT, a language analysis toolkit
Quirk, C., Choudhury, P., Gao, J., Suzuki, H., Toutanova, K., Gamon, M., Yih, W.-T., Cherry, C. A., & Vanderwende, L. (2012) · Cited by 32
Automatic question generation from text - an aid to independent study
Wolfe, J. H. (1976) · ACM SIGCSE Bulletin · doi:10.1145/952989.803459 · Cited by 26
Reading Retention as a Function of Method for Generating Interspersed Questions.
Wolfe, J. H. (1977) · Defense Technical Information Center (DTIC) · Cited by 5
Research works printed but not linked: 5
- Husam Ali, Yllias Chali, and Sadid A Hasan. 2010. Automation of question generation from sentences. In Proceedings of QG2010: The Third Workshop on Question Generation , pages 58-67.
- Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2012. Abstract meaning representation (amr) 1.0 specification. In Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: ACL , pages 1533-1544.
- Kristy Elizabeth Boyer and Paul Piwek. 2010. Proceedings. In Proceedings of QG2010: The Third Workshop on Question Generation. Pittsburgh: questiongeneration. org .
- D Gates. 2008. Generating look-back strategy questions from expository texts. In The Workshop on the Question Generation Shared Task and Evaluation Challenge, NSF, Arlington, VA. http://www. cs. memphis. edu/˜ vrus/questiongeneration//1-Gates-QG08. pdf .
- Paul Kroeger. 2004. Analyzing syntax: a lexicalfunctional approach . Cambridge University Press.
Cited by
A Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514
“This classification was carried out by analysing POS and dependency labels, as in Mazidi and Tarau (2016a) and Mazidi and Tarau (2016b) or by using a machine learning (ML) model and a set of rules, as in Basuki and Kusuma (2018).”
BackgroundMethodA Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 153
“Later, Eldesoky [61] and Mazidi and Tarau [145] first classified sentences before generating questions to determine the question type, which significantly increases the percentage of acceptable questions.”
BackgroundMulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132
“Although some research work uses such metrics in the form of human evaluation [2, 28, 31, 32, 35, 43, 57, 65, 67, 75, 84–86,”
MethodMultiple Choice Question Generation Utilizing An Ontology
Stasaski, K., & Hearst, M. A. (2017) · Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w17-5034 · Cited by 63
“Approaches have included factual generation directly from text (Brown et al., 2005; Mannem et al., 2010; Mazidi and Tarau, 2016; Yao et al., 2012) as well as generation from knowledge bases (Olney et al., 2012a).”
Background
Metadata
- Authors
- Mazidi, K., & Tarau, P.
- Affiliations
- University of North Texas (US)
- ORCID
- Tarau, Paul
- Venue
- Proceedings of the 9th International Natural Language Generation conference
- Published
- 2016
- Pages
- 51-60
- OpenAlex
- W2562215231
- Cited by
- 36
- References
- 29
- Citations by year
- 2025: 2 · 2024: 3 · 2023: 6 · 2022: 4 · 2021: 5 · 2020: 2 · 2019: 4 · 2018: 5 · 2017: 4 · 2016: 1
- FWCI
- 2.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Generation (publishedVersion, cc-by)
- University of North Texas Digital Library (University of North Texas) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- automatic question generation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.