Automatic generation of short answer questions for reading comprehension assessment
Abstract
Writing items for reading comprehension assessment is time-consuming. Automating part of the process can help test-designers to develop assessments more efficiently and consistently. This paper presents an approach to automatically generating short answer questions for reading comprehension assessment. Our major contribution is to introduce Lexical Functional Grammar (LFG) as the linguistic framework for question generation, which enables systematic utilization of semantic and syntactic information. The approach can efficiently generate questions of better quality than previous high-performing question generation systems, and uses paraphrasing and sentence selection to improve the cognitive complexity and effectiveness of questions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 5× · see citing works
“As WordNet ranks the senses of a word according to their frequencies in a reference corpus, we set a fixed threshold on the sense numbers to restrict sense frequencies.”
MethodInfluentialAdvances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Landauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282 · see citing works
Taxonomy of Educational Objectives. Handbook I: Cognitive Domain
Bloom, B., & Krathwohl, D. R. (1966) · Cited by 4,624 · see citing works
Thematic Proto-Roles and Argument Selection
Dowty, D. R. (1991) · Language · doi:10.1353/lan.1991.0021 · Cited by 4,494 · see citing works
The Proposition Bank: An Annotated Corpus of Semantic Roles
Palmer, M. S., Gildea, D., & Kingsbury, P. (2005) · Computational Linguistics · doi:10.1162/0891201053630264 · Cited by 2,317 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Language Testing in Practice: Designing and Developing Useful Language Tests
Bachman, L. F., & Palmer, A. S. (1996) · Cited by 1,539 · see citing works
Essentials of Educational Measurement
Garvin, A. D., & Ebel, R. L. (1980) · Educational Researcher · doi:10.2307/1175572 · Cited by 1,492 · see citing works
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Levy, O., Goldberg, Y., & Dagan, I. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00134 · Cited by 1,307 · see citing works
Taxonomy of Educational Objectives. Handbook I: Cognitive Domain
Blyth, W. A. L., Bloom, B., & Krathwohl, D. R. (1966) · British Journal of Educational Studies · doi:10.2307/3119730 · Cited by 1,111 · see citing works
Chomsky, N. (1977) · Medical Entomology and Zoology · Cited by 1,012 · see citing works
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics
Gimpel, K., Schneider, N., O’Connor, B., Das, D., Mills, D. Q., Eisenstein, J., Heilman, M., Yogatama, D., Flanigan, J., & Smith, N. A. (2011) · The Association for Computational Linguistics · Cited by 890 · see citing works
An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet
Banerjee, S., & Pedersen, T. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45715-1_11 · Cited by 863 · see citing works
Lexical-Functional Grammar: A Formal System for Grammatical Representation
Kaplan, R. M., & Bresnan, J. (1995) · Cited by 725 · see citing works
Vanderwende, L., Daumé, H., & Kirchhoff, K. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 685 · see citing works
Using latent semantic analysis to improve access to textual information
Dumais, S., Furnas, G. W., Landauer, T. K., Deerwester, S. C., & Harshman, R. A. (1988) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/57167.57214 · Cited by 682 · see citing works
Ganitkevitch, J., Durme, B. V., & Callison-Burch, C. (2013) · Cited by 659 · see citing works
The Representation Problem in Reading Acquisition
Perfetti, C. A. (2017) · Routledge eBooks · doi:10.4324/9781351236904-6 · Cited by 645 · see citing works
Essentials of educational measurement
Hanson, R. A. (1973) · Journal of School Psychology · doi:10.1016/0022-4405(73)90057-5 · Cited by 532 · see citing works
Semantic taxonomy induction from heterogenous evidence
Snow, R., Jurafsky, D. S., & Ng, A. Y. (2006) · doi:10.3115/1220175.1220276 · Cited by 456 · see citing works
The Importance of Syntactic Parsing and Inference in Semantic Role Labeling
Punyakanok, V., Roth, D., & Yih, W.-T. (2008) · International Conference on Computational Logic · doi:10.1162/coli.2008.34.2.257 · Cited by 453 · see citing works
Good Question! Statistical Ranking for Question Generation
Heilman, M., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473390 · Cited by 366 · see citing works
Automatic summarising: The state of the art
Jones, K. S. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.03.009 · Cited by 354 · see citing works
Applied morphological processing of English
Minnen, G., Carroll, J. M., & Pearce, D. (2001) · Natural Language Engineering · doi:10.1017/s1351324901002728 · Cited by 345 · see citing works
Immediate-head parsing for language models
Charniak, E. (2001) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1073012.1073029 · Cited by 297 · see citing works
How to Construct Achievement Tests to Assess Comprehension
Anderson, R. C. (1972) · Review of Educational Research · doi:10.3102/00346543042002145 · Cited by 294 · see citing works
Explorations in context space: Words, sentences, discourse
Burgess, C., Livesay, K. L., & Lund, K. (1998) · Discourse Processes · doi:10.1080/01638539809545027 · Cited by 274 · see citing works
Computer-aided generation of multiple-choice tests
Mitkov, R., & Ha, L. A. (2003) · doi:10.3115/1118894.1118897 · Cited by 258 · see citing works
A computer-aided environment for generating multiple-choice test items
MITKOV, R., AN HA, L., & KARAMANIS, N. (2006) · Natural Language Engineering · doi:10.1017/s1351324906004177 · Cited by 229 · see citing works
Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger
Ciaramita, M., & Altün, Y. (2006) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1610075.1610158 · Cited by 210 · see citing works
Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing
Jurafsky, D., & Gaussier, É. (2006) · Cited by 196 · see citing works
The Evolution of Human Language
(2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511817755 · Cited by 143 · see citing works
Generating Natural Language Questions to Support Learning On-Line
Lindberg, D., Popowich, F. P., Nesbit, J. C., & Winne, P. H. (2013) · Cited by 122 · see citing works
Your theory of language evolution depends on your theory of language
Jackendoff, R. S. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511817755.004 · Cited by 122 · see citing works
Performance‐Compatible Competence Grammar
Sag, I. A., & Wasow, T. (2011) · doi:10.1002/9781444395037.ch10 · Cited by 118 · see citing works
Question-Answering Strategies for Children.
Raphael, T. E. (1982) · The Reading Teacher · Cited by 114 · see citing works
Borsley, R. D., & Börjars, K. (2011) · Wiley eBooks · doi:10.1002/9781444395037 · Cited by 102 · see citing works
Generating Instruction Automatically for the Reading Strategy of Self-Questioning
Jack, M., & Wei, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-465 · Cited by 101 · see citing works
Proceedings International Conference on Computers in Education
(2002) · doi:10.1109/cie.2002.1185849 · Cited by 101 · see citing works
Question Generation from Concept Maps
Olney, A. M., Graesser, A. C., & Person, N. K. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.204 · Cited by 99 · see citing works
Automatic Gap-fill Question Generation from Text Books
Agarwal, M., & Mannem, P. (2011) · Cited by 99 · see citing works
Lascarides, A., Gardent, C., & Nivre, J. (2009) · Cited by 96 · see citing works
Linguistic Considerations in Automatic Question Generation
Mazidi, K., & Nielsen, R. D. (2014) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/v1/p14-2053 · Cited by 88 · see citing works
EM works for pronoun anaphora resolution
Charniak, E., & Elsner, M. (2009) · doi:10.3115/1609067.1609083 · Cited by 85 · see citing works
G-Asks: An Intelligent Automatic Question Generation System for Academic Writing Support
Liu, M., Calvo, R. A., & Rus, V. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.205 · Cited by 77 · see citing works
Semantics-based Question Generation and Implementation
Yao, X., Bouma, G., & Zhang, Y. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.202 · Cited by 72 · see citing works
Ockey, G. J. (2009) · Modern Language Journal · doi:10.1111/j.1540-4781.2009.00976.x · Cited by 71 · see citing works
Question Generation from Paragraphs at UPenn: QGSTEC System Description
Mannem, P., Prasad, R., & Joshi, A. K. (2010) · Cited by 68 · see citing works
Effects of question type and question placement on problem-solving ability from prose material.
Felker, D. B., & Dapra, R. A. (1975) · Journal of Educational Psychology · doi:10.1037/h0076616 · Cited by 68 · see citing works
Automatic question generation from text - an aid to independent study
Wolfe, J. H. (1976) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/800107.803459 · Cited by 66 · see citing works
Mind the Gap: Learning to Choose Gaps for Question Generation
Becker, L. A., Basu, S., & Vanderwende, L. (2012) · Cited by 49 · see citing works
Generating Diagnostic Multiple Choice Comprehension Cloze Questions
Mostow, J., & Jang, H. (2012) · Cited by 48 · see citing works
Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications
Miaschi, A., Davidson, S., Brunato⋄, D., Dell’Orletta⋄, F., Sagae, K., Sánchez‐Gutiérrez, C., & Venturi⋄, G. (2016) · doi:10.18653/v1/w16-05 · Cited by 45 · see citing works
Multilingual WSD with Just a Few Lines of Code: the BabelNet API
Navigli, R., & Ponzetto, S. P. (2012) · MADOC (University of Mannheim) · Cited by 44 · see citing works
Question Generation with Minimal Recursion Semantics
Yao, X. (2010) · Cited by 43 · see citing works
Question Generation based on Lexico-Syntactic Patterns Learned from the Web
Curto, S., Mendes, A. C., & Coheur, L. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.207 · Cited by 36 · see citing works
Mostow, J., Beck, J. E., Bey, J., Cuneo, A., Sison, J., Tobin, B., & Valeri, J. (2004) · Cited by 36 · see citing works
A Detailed Account of The First Question Generation Shared Task Evaluation Challenge
Rus, V., Brendan Wyse, A., Piwek, P., Lintean, M. C., Stoyanchev, S., & Moldovan, C. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.208 · Cited by 33 · see citing works
Proceedings of the 14th European Workshop on Natural Language Generation
Basile, V., & Bos, J. (2013) · The Association for Computational Linguistics · Cited by 29 · see citing works
Question Generation for French: Collating Parsers and Paraphrasing Questions
Bernhard, D., de Viron, L., Moriceau, V., & Tannier, X. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.203 · Cited by 20 · see citing works
Automatically Generating Discussion Questions
Adamson, D., Bhartiya, D., Gujral, B., Kedia, R., Singh, A., & Rosé, C. P. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_9 · Cited by 20 · see citing works
Automatic question generation from text - an aid to independent study
Wolfe, J. H. (1976) · ACM SIGCUE Outlook · doi:10.1145/953026.803459 · Cited by 20 · see citing works
Varieties of Question Generation: Introduction to this Special Issue
Piwek, P., & Boyer, K. E. (2012) · Open Research Online - ORO (The Open University) · Cited by 19 · see citing works
A Comparison of Objective-Based and Modified-Bormuth Item Writing Techniques
Roid, G. H., & Haladyna, T. M. (1978) · Educational and Psychological Measurement · doi:10.1177/001316447803800104 · Cited by 11 · see citing works
Piwek, P., & Boyer, K. E. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.201 · Cited by 11 · see citing works
The impact of deep linguistic processing on parsing technology
Baldwin, T., Dras, M., Hockenmaier, J., King, T. H., & van Noord, G. (2007) · International Workshop/Conference on Parsing Technologies · doi:10.3115/1621410.1621415 · Cited by 9 · see citing works
Text as a Construct of Aggregations
Hřebíček, L. (1993) · Springer eBooks · doi:10.1007/978-94-011-1769-2_3 · Cited by 5 · see citing works
Flagged first, then by citations
4 of the 75 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Systematic Review of Automatic Question Generation for Educational Purposes
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00186-y · Cited by 514 · see citing works
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
A Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 91 · see citing works
Investigating the capability of ChatGPT for generating multiple-choice reading comprehension items
Lin, Z., & Chen, H. (2024) · System · doi:10.1016/j.system.2024.103344 · Cited by 68 · see citing works
A Semantic Role-based Approach to Open-Domain Automatic Question Generation
Flor, M., & Riordan, B. (2018) · Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w18-0530 · Cited by 39 · see citing works
Gorgun, G., & Bulut, O. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12771-3 · Cited by 18 · see citing works
Gašpar, A., Grubišić, A., & Šarić‐Grgić, I. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09672-1 · Cited by 18 · see citing works
Coordinated LLM multi-agent systems for collaborative question-answer generation
Saadaoui, S., & Alonso, E. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114627 · Cited by 20 · see citing works
Question Generator System of Sentence Completion in TOEFL Using NLP and K-Nearest Neighbor
Riza, L. S., Pertiwi, A. D., Rahman, E. F., Munir, M., & Abdullah, C. U. (2019) · Indonesian Journal of Science and Technology · doi:10.17509/ijost.v4i2.18202 · Cited by 14 · see citing works
Steuer, T., Bongard, L., Uhlig, J., & Zimmer, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86436-1_22 · Cited by 12 · see citing works
Tomikawa, Y., Suzuki, A., & Uto, M. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3491801 · Cited by 10 · see citing works
Development of system for generating questions, answers, distractors using transformers
Barlybayev, A., & Matkarimov, B. T. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i2.pp1851-1863 · Cited by 15 · see citing works
Reading bots: The implication of deep learning on guided reading
Huang, B., Dou, J., & Zhao, H. Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.980523 · Cited by 9 · see citing works
A Fuzzy WordNet graph based approach to find key terms for students short answer evaluation
Vij, S., Tayal, D. K., & Jain, A. (2019) · International Conference on Internet of Things: Smart Innovation and Usages (IoT-SIU) · doi:10.1109/iot-siu.2019.8777600 · Cited by 6 · see citing works
Steuer, T., Filighera, A., Meuser, T., & Rensing, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04123 · Cited by 11 · see citing works
A literature review of research on question generation in education
Dong, X., Zhang, X., Li, Z., Hou, Q., Xue, J., & Li, X. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3203 · Cited by 3 · see citing works
GenScore: Agent-Based Short-Answer Question Generation and Scoring in Software Engineering Courses
Duong, T. N. B., & Shar, L. K. (2025) · IEEE Access · doi:10.1109/access.2025.3647111 · Cited by 3 · see citing works
“Questomatic”: Automated Question Formulation System
Jadhav, P., Deore, S., Aute, S., Gaikwad, R., Barphe, S. S., & Meshram, E. (2024) · International Conference on Sustainable Expert Systems (ICSES) · doi:10.1109/icses63445.2024.10763124 · Cited by 3 · see citing works
Afzaal, M., Nouri, J., & Aayesha, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42682-7_2 · Cited by 3 · see citing works
Diverse Content Selection for Educational Question Generation
Hadifar, A., Bitew, S. K., Deleu, J., Hoste, V., Develder, C., & Demeester, T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2023.eacl-srw.13 · Cited by 2 · see citing works
Automatic item generation for personality situational judgment tests with large language models
Li, C., Zhang, J., Tang, Y., & Li, J. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.100964 · Cited by 2 · see citing works
Knowledge-Aware Arabic Question Generation: A Transformer-Based Framework
Jabr, R. B., & Azmi, A. M. (2025) · Mathematics · doi:10.3390/math13182975 · Cited by 1 · see citing works
ViQG: Web Tool for Automatic Question Generation from Code for Viva Preparation
Gaur, H., Tayal, D. K., & Jain, A. (2023) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda60357.2023.10482988 · Cited by 1 · see citing works
Rule-Based Automatic Question Generationusing Dependency ParsingPreprint
Sewunetie, W. T., & Kovács, L. (2023) · Research Square · doi:10.21203/rs.3.rs-2693893/v1 · Cited by 1 · see citing works
Good-Quality Question Generation for Academic Support
Divate, M. S., & Salgaonkar, A. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-10-8237-5_75
Automatic Web-Based Question Answer Generation System for Online Feedable New-Born Chatbot
Abdul-Kader, S. A., Woods, J., & Thabet, T. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-01174-1_7
Deep question generation model based on dual attention guidance
Li, J., Zhang, X., Wang, J., & Zhou, X. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02249-6
Impact of OCR Noise on BERT-Based Question Type Classification in Digital Test Banks
Seo, J.-H., & Park, J.-H. (2025) · Journal of Digital Contents Society · doi:10.9728/dcs.2025.26.10.2871
A Rule-Based Automatic Question Generation Framework for Educational Text
Tonde, J. P., & Sankaye, S. (2026) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2613127
- Authors
- HUANG, Y., & HE, L.
- Affiliations
- University of Cambridge (GB)
- Zhejiang University (CN)
- Venue
- Natural Language Engineering
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-01-13
- Volume
- 22
- Issue
- 3
- Pages
- 457-489
- ISSN
- 1351-3249 (ISSN-L)
- 1469-8110
- OpenAlex
- W2336465850
- Cited by
- 30
- References
- 75
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 6 · 2023: 7 · 2022: 1 · 2021: 3 · 2019: 3 · 2018: 3
- FWCI
- 1.65
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Natural Language Engineering (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- automatic question generation, short-answer questions, reading comprehension assessment, Lexical-Functional Grammar, test item generation, paraphrase, sentence selection, cognitive complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review