Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

May combine two claims.

Rule-based algorithms often rely on extracted features and add a classifier model, extracting syntactic and semantic parts and using templates.

“These algorithms rely often on extracted features and later add a classifier model. They extract syntactic and semantic parts of text and make use of templates to construct the question.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AQG systems are gaining popularity due to needs in machine reading comprehension, conversational systems, and education.

“Such systems are becoming more popular of late, with their requirement in machine reading comprehension applications, conversational systems, and even educational applications.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

AQG systems are built using varied approaches.

“Automatic question generation systems are realized using varied approaches.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The field has advanced considerably with the arrival of recent deep learning architectures.

“However, this field has advanced to a great extent over the years with introduction of recent deep learning architectures.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

AQG in natural language has many applications.

“Question generation in natural language has a wide variety of applications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generating questions now includes creating questions for images or video.

“Generating questions in natural language is now, a more evolved task, which also includes generating questions for an image or video.”

May combine two claims.

In rule-based standalone question generation, most approaches use Wikipedia as dataset and evaluation metrics include f1-score or precision.

“As seen from the table, most approaches make use of Wikipedia as their dataset and the evaluation metrics for automated evaluation include f1-score or precision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Neural-based approaches have become very popular for AQG due to the large number of datasets available in recent years.

“With the humongous number of datasets available in current years, neural-based approaches have become very popular for automatic question generation.”

May combine two claims.

Evaluation in rule-based standalone question generation is not strongly made with a well-defined metric, though human ratings have been explored.

“Evaluation is not strongly made in terms of a well-defined metric though human based ratings have been explored.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rule-based algorithms usually consider smaller target topics where specific question types are required.

“Usually, smaller target topics are considered where specific types of questions are required to be generated.”

May combine two claims.

GPT-2-based models gave the best results on CoQA, the most used dataset.

“We observe that GPT-2-based models gave the best results on CoQA, the most used dataset.”

• AssumptionRests on an assumption: The automatic evaluation scores reported by different primary studies are comparable enough to rank models across studies.

Probably not a substantial claim: may not pass S3 (load-bearing).

BLEU, METEOR, and ROUGE are the more favored automatic metrics for standalone question generation and conversational question generation.

“Among the automatic metrics, BLEU, METEOR and ROUGE are the more favored metrics for Standalone QG and Conversational QG.”

• AssumptionRests on an assumption: The automatic evaluation scores reported by different primary studies are comparable enough to rank models across studies.

Probably not a substantial claim: may not pass S3 (load-bearing).

Most available datasets for question generation are curated from open-domain data such as Wikipedia, Reddit, and social networking platforms.

“What happens in case of training data is that most of the datasets available are curated from open domain data like Wikipedia, reddit, social networking platforms and the like.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Video question generation is an upcoming area with some recent works.

“Video QG is an upcoming area, where some works have been introduced lately.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Only a few datasets are available for conversational question generation, and as a result not much work has been done in this use-case.

“It would also be important to note that for conversational question generation, only a few datasets are available. As a result, not much work is done in this use-case.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Human-based evaluation is currently used for grammatical correctness, question well-formedness, and domain-centric aspects, though some recent efforts address these.

“Human-based evaluation is currently in use for the above aspects, although some efforts in this direction are being made recently.”

May combine two claims.

Most currently available datasets for training question generation systems are crowd-sourced, which largely affects the quality of generated questions.

“Most datasets that are currently available for training question generation systems are crowd-sourced [42, 47, 84, 92, 114, 115, 119, 120, 122, 123], which largely impacts the quality of generated questions.”

• cites [42] Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ Questions for Machine Comprehension of Text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 2383–2392. Association for Computational Linguistics, Stroudsburg, PA, USA (2016) sourceNot checked yet

  Not checked in the free run.

• cites [47] Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., Suleman, K.: NewsQA: A Machine Comprehension Dataset. pp. 191–200 (2017). https://doi.org/10.18653/v1/w17-2623 sourceNot checked yet

  Not checked in the free run.

• cites [84] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision. 2015 Inter, pp. 2425–2433 (2015). https://doi.org/10.1109/ICCV.2015.279 sourceNot checked yet

  Not checked in the free run.

• cites [92] Reddy, S., Chen, D., Manning, C.D.: CoQA: A Conversational Question Answering Challenge. In: Transactions of the Association for Computational Linguistics. pp. 249–266 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [114] Choi, E., He, H., Yatskar, M., Yih, W., Choi, Y., Liang, P., Zettlemoyer, L.: QuAC : Question answering in context. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2174–2184 (2018) sourceNot checked yet

  Not checked in the free run.

• cites [115] Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., Riedel, S.: Interpretation of natural language rules in conversational machine reading. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. 1, 2087–2097 (2020). https://doi.org/10.18653/v1/d18-1233 sourceNot checked yet

  Not checked in the free run.

• cites [119] Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA Matter: elevating the role of image understanding in visual question answering. Int. J. Comput. Vis. 127, 398–414 (2019). https://doi.org/10.1007/s11263-018-1116-0 sourceNot checked yet

  Not checked in the free run.

• cites [120] Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L.: Visual7W: Grounded question answering in images. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2016-Decem, pp. 4995–5004 (2016). https://doi.org/10.1109/CVPR.2016.540 sourceNot checked yet

  Not checked in the free run.

• cites [122] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123, 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7 sourceNot checked yet

  Not checked in the free run.

• cites [123] Johnson, J., Fei-Fei, L., Hariharan, B., Zitnick, C.L., van der Maaten, L., Girshick, R.: CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pp. 1988–1997 (2017). Doi: https://doi.org/10.1109/CVPR.2017.215 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

The standard metrics used to evaluate generated questions are those generally used for text generation.

“Although the standard metrics used to evaluate the generated questions are the ones used generally for text generation.”

May combine two claims.

No existing approach completely solves generating relevant questions for text given a topic, although a few topic-based approaches exist.

“Although there are a few approaches where topic-based questions [99] have been looked at, there is no existing approach that completely solves the problem.”

• cites [99] Qi, P., Zhang, Y., Manning, C.D.: Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations. pp. 25–40 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Most datasets are curated by crowdsourcing, and SQuAD and NewsQA have sufficiently large numbers of training examples.

“Most datasets are curated by crowdsourcing and the number of training examples are sufficiently large in datasets (SQuAD, NewsQA).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

BLEU is often used for visual question generation, while other metrics have been replaced by CIDEr for visual question generation.

“BLEU is even used often for Visual QG but the other metrics have been replaced by CIDEr for Visual QG.”

• AssumptionRests on an assumption: The automatic evaluation scores reported by different primary studies are comparable enough to rank models across studies.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For images, most systems identify objects in scenes or images, and this task is largely solved, but models that extract meaningful information from images are limited to trivial use-cases.

“If we consider the case of images, most systems try to identify objects placed in various scenes or images, this task has been solved to quite an extent, although the kind of models which are able to extract meaningful information from an image are limited to trivial use-cases (refer Table 7).”

Classified as probably not a substantial claim. May combine two claims.

Most question generation datasets are crowd-sourced or borrowed from open-source communities such as Wikipedia and Reddit, making them unsuitable for domain-based question generation.

“As most of the datasets for question generation are either crowd-sourced or borrowed from open-sourced communities like Wikipedia, Reddit and others. These are not suitable to domain-based question generation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most question generation systems today have worked on generating questions from text.

“As presented in the survey, most question generation systems today have worked on generating questions from the text.”

• AssumptionRests on an assumption: The surveyed literature is representative of the field of automatic question generation.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most works used RNN-based architectures with additional features, and a few used transformer models.

“Most works used RNN-based architectures with additional features and a few used transformer models.”

• AssumptionRests on an assumption: The automatic evaluation scores reported by different primary studies are comparable enough to rank models across studies.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Generated questions lack naturalness and are sometimes meaningless for information extraction.

“For example, questions generated lack naturalness and sometimes are meaningless in the sense of information extraction.”

• AssumptionRests on an assumption: The surveyed literature is representative of the field of automatic question generation.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Justifiable metrics for evaluating question quality are still a work in progress.

“Also, justifiable metrics for evaluating the quality of questions is still a work in progress.”

• AssumptionRests on an assumption: The surveyed literature is representative of the field of automatic question generation.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Multiple input modalities are being considered recently, and the impact of incorporating them is being studied.

“Multiple input modalities are being considered of late and the impact of incorporating them is being studied.”

• AssumptionRests on an assumption: The surveyed literature is representative of the field of automatic question generation.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Closed-domain question generation targets a specific domain and depends on domain-specific knowledge limited by an ontology.

“In closed-domain question generation, questions are generated for a specific domain like medicine [5, 6], educational text [7]. Here, the questions usually rely on some domain-specific knowledge restricted by an ontology.”

• cites [5] Wang, W., Hao, T., Liu, W.: Automatic question generation for learning evaluation in medicine. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 4823 LNCS, pp. 242–251 (2008). https://doi.org/10.1007/978-3-540-78139-4_22 sourceNot checked yet

  Not checked in the free run.

• cites [6] Shen, S., Li, Y., Du, N., Wu, X., Xie, Y., Ge, S., Yang, T., Wang, K., Liang, X., Fan, W.: On the generation of medical question-answer pairs. In: AAAI 2020 - 34th AAAI Conference on Artificial Intelligence. pp. 8822–8829 (2020). https://doi.org/10.1609/aaai.v34i05.6410 sourceNot checked yet

  Not checked in the free run.

• cites [7] Araki, J., Rajagopal, D., Sankaranarayanan, S., Holm, S., Yamakawa, Y., Mitamura, T.: Generating questions and multiple-choice answers using semantic analysis of texts. In: COLING 2016 - 26th International Conference on Computational Linguistics, Proceedings of COLING 2016: Technical Papers. pp. 1125–1136 (2016)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The encoder-decoder architecture was introduced by Google in [31].

“The encoder–decoder architecture was introduced by Google in [31].”

• cites [31] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst. 4, 3104–3112 (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [48] generates clarification questions on text using a GAN-based approach.

“In [48], an attempt is made to generate clarification questions on text to extract useful information that captures the context of the text.”

• cites [48] Rao, S., Daumé, H.: Answer-based adversarial training for generating clarification questions. In: NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference. vol. 1, pp. 143–155 (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A few specific reviews exist for particular domain areas of question generation.

“There are a few specific reviews, in particular domain areas of question generation. For example, the work done for question generation in the educational domain is discussed in [15].”

• cites [15] Kurdi, G., Leo, J., Parsia, B., Sattler, U., Al-Emari, S.: A systematic review of automatic question generation for educational purposes. Int. J. Artif. Intell. Educ. 30, 121–204 (2020). https://doi.org/10.1007/s40593-019-00186-y sourceNot checked yet

  Not checked in the free run.

• cites [16] Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: a survey. Res. Pract. Technol. Enhanc. Learn. (2021). https://doi.org/10.1186/s41039-021-00151-1 sourceNot checked yet

  Not checked in the free run.

• cites [17] Patil, C., Patwardhan, M.: Visual question generation: the state of the art. ACM Comput. Surv. (2020). https://doi.org/10.1145/3383465 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Question categorization helps determine the exact use-case to be realized.

“Question categorization helps in chalking out the exact use-case that has to be realized.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sometimes the goal is to generate questions on a topic of interest or from different spans of a passage.

“For instance, in some cases, the goal is to generate questions on a topic of interest or based on different spans of text in a passage [1].”

• cites [1] Kumar, V., Chaki, R., Talluri, S.T., Ramakrishnan, G., Li, Y.-F., Haffari, G.: Question generation from paragraphs: a tale of two hierarchical models. (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [39] addresses fill-in-the-blank question generation by using GANs to create distractors.

“The problem of fill-in-the-blank (FITB)-type question generation is dealt with in [39]. The creation of important distractors is done using generative adversarial networks for training.”

• cites [39] Liang, C., Yang, X., Wham, D., Pursel, B., Passonneau, R., Giles, C.L.: Distractor generation with generative adversarial nets for automatically creating fill-in-the-blank questions. In: Proceedings of the Knowledge Capture Conference, K-CAP 2017. 4–7 (2017). https://doi.org/10.1145/3148011.3154463 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Question Generation Systems are either closed domain or open domain.

“Question Generation Systems fall into one of the following domain categories: closed domain or open domain.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The method of [39] performed better than existing similarity-based methods.

“The proposed method performed better than already existing similarity-based methods.”

• cites [39] Liang, C., Yang, X., Wham, D., Pursel, B., Passonneau, R., Giles, C.L.: Distractor generation with generative adversarial nets for automatically creating fill-in-the-blank questions. In: Proceedings of the Knowledge Capture Conference, K-CAP 2017. 4–7 (2017). https://doi.org/10.1145/3148011.3154463 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Lehnert's classification is an important contribution to question categorization.

“An important contribution towards this direction is Lehnert’s classification [18].”

• cites [18] Lehnert, W.: A Conceptual Theory of Question Answering. Ijcai-77. (1977)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Factual questions are simple objective questions starting with what, which, when, who, how, and expect a word or group of words from the text.

“Factual questionsThis category of questions is simple objective questions that start with what, which, when, who, how. Here, the expected answer is a word or a group of words from sentences on a paragraph of text.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [36] extracts natural-language question-answer pairs from a knowledge graph using an RNN-based question generation model.

“In [36], question–answer pairs in natural language are extracted through a knowledge graph making use of the RNN-based model for question generation.”

• cites [36] Indurthi, S., Raghu, D., Khapra, M.M., Joshi, S.: Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model. In: 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 - Proceedings of Conference. vol. 1, pp. 376–385 (2017). https://doi.org/10.18653/v1/e17-1036 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

For any question generation system, identifying the types of questions it can generate is important.

“Thus, for any question generation system, it is important to identify the types of questions which can be generated by it.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Yes/no type questions require a Boolean response and some reasoning.

“Yes/no type questionsThese are questions that require a Boolean response, yes/no. These require some level of reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Questions can be classified as extractive or abstractive.

“Other classifications could determine whether the questions are extractive or abstractive.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Multiple sentences spanning questions require multiple sentences as the answer, with facts spread across several sentences.

“Multiple sentences spanning questionsSome questions can require multiple sentences of a paragraph as the answer. The facts are present in several sentences.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The review focused more on articles using machine learning and deep learning architectures.

“We focused more on articles which used various machine learning and deep learning-based architectures.”

• AssumptionRests on an assumption: The journals and conferences from which papers were selected are of repute.
• AssumptionRests on an assumption: The selected papers are representative of the field.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Deep understanding questions are inference-oriented and require deriving a fact from several related facts.

“Deep understanding questionsThese are inference-oriented questions that require a proper inference mechanism. These questions might require deriving a fact from several related facts in a piece of text.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Complex natural language processing is not needed to answer factual questions.

“Complex natural processing is not required for answering factual questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Various taxonomies for question categories have been proposed.

“When we consider the question categories, various taxonomies were proposed.”

• cites [18] Lehnert, W.: A Conceptual Theory of Question Answering. Ijcai-77. (1977)Not checked yet

  Not checked in the free run.

• cites [19] Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994). https://doi.org/10.3102/00028312031001104 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Questions can be characterized by the cognitive level of the expected answer.

“Questions may also be characterized, on the basis of cognitive levels of the answer expected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [35] proposes a neural network-based question generation method that produces a question about a target aspect from input text.

“The authors propose a novel neural network-based question generation technique that generates a question towards a target aspect from an input piece of text [35].”

• cites [35] Hu, W., Liu, B., Ma, J., Zhao, D., Yan, R.: Aspect-based question generation. In: 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings. pp. 1–10 (2018)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

AQG has been widely tried in educational settings.

“AQG has been widely experimented with, in educational settings.”

• cites [3] Kunichika, H., Katayama, T., Hirashima, T., Takeuchi, A.: Automated question generation methods for intelligent english learning systems and its evaluation. Spectrochim. Acta. A. Mol. Biomol. Spectrosc. 62, 1209–1215 (2005) sourceNot checked yet

  Not checked in the free run.

• cites [4] Mostow, J., Chen, W.: Generating instruction automatically for the reading strategy of self-questioning. Front. Artif. Intell. Appl. 200, 465–472 (2009). https://doi.org/10.3233/978-1-60750-028-5-465 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The ultimate aim of AQG systems is to produce questions that are syntactically and semantically correct and meaningful for the use-case.

“The eventual goal of AQG systems is the capability to generate questions that are correct syntactically and semantically as well as meaningful in the context of the use-case.”

May combine two claims.

The model of [35] outperformed the baseline from [32] in both automatic and human evaluations.

“The authors performed both automatic evaluations using BLEU, METEOR, and ROUGE as well as human-based evaluation and found that their model outperformed the baseline model in [32] that they considered.”

• cites [35] Hu, W., Liu, B., Ma, J., Zhao, D., Yan, R.: Aspect-based question generation. In: 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings. pp. 1–10 (2018)Not checked yet

  Not checked in the free run.

• cites [32] Du, X., Shao, J., Cardie, C.: Learning to ask: Neural question generation for reading comprehension. In: ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). vol. 1, pp. 1342–1352 (2017). https://doi.org/10.18653/v1/P17-1123 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

The review excluded articles with incomplete experiments, insufficient proof, no comparisons, and non-English articles.

“We excluded the remaining articles which had incomplete experiments or less sufficient proof of results and those with no comparison among various methodologies. Also, we excluded the articles that were not written in English.”

• AssumptionRests on an assumption: The journals and conferences from which papers were selected are of repute.
• AssumptionRests on an assumption: The inclusion criteria used in the review ensure the quality of the surveyed content.
• AssumptionRests on an assumption: The selected papers are representative of the field.

Probably not a substantial claim: may not pass S3 (load-bearing).

Encoder-decoder architecture promotes end-to-end learning for sequence-to-sequence tasks.

“This architecture promotes end-to-end learning for tasks that require a sequence of tokens as input and a sequence of tokens as output.”

• cites [31] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst. 4, 3104–3112 (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Most factual questions are asked by choosing a single sentence and expect a known fact.

“Most of these questions are asked by choosing a single sentence from a paragraph and expect a known fact as an answer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are three types of question generation systems: standalone, visual, and conversational.

“Broadly, we identify three types of question generation systems: standalone question generation (SQG), visual question generation (VQG), and conversational question generation (CQG) (Fig. 1).”

• AssumptionRests on an assumption: The three use-cases (standalone, visual, conversational) cover the main types of AQG systems.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Open-domain question generation is domain-independent and needs only universal ontologies.

“Open-domain question generation does not depend on any domain and allows you to generate questions irrespective of the domain requiring only universal ontologies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When used for SQG, the encoder-decoder model takes input passage and answer and attempts to produce a question similar to the labelled question.

“When used for SQG, the encoder–decoder model takes input passage (and answer) as input and attempts to produce a question similar to the labelled question.”

May combine two claims.

The RNN model of [36] achieved higher BLEU4 scores than phrase-based machine translation [34] and a template-based method [37].

“The RNN model gave higher BLEU4 scores over other compared baselines, phrase-based machine translation [34] and template-based method [37].”

• cites [36] Indurthi, S., Raghu, D., Khapra, M.M., Joshi, S.: Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model. In: 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 - Proceedings of Conference. vol. 1, pp. 376–385 (2017). https://doi.org/10.18653/v1/e17-1036 sourceNot checked yet

  Not checked in the free run.

• cites [34] Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C.: Moses: open source toolkit for statistical machine translation. In: Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL ’07. p. 177 (2007) sourceNot checked yet

  Not checked in the free run.

• cites [37] Zhao, S., Wang, H., Li, C., Liu, T., Guan, Y.: Automatically generating questions from queries for community-based question answering. In: Proceedings of 5th International Joint Conference on Natural Language Processing. pp. 929–937 (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The type of questions a system generates matters when selecting an approach.

“Also, the kind of questions that such a system generates is important when choosing an approach.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The model of [45] gave better results than existing methods.

“Their experimentation proved that their model gave better results than existing methods.”

• cites [45] Bao, J., Gong, Y., Duan, N., Zhou, M., Zhao, T.: Question generation with doubly adversarial nets. IEEE/ACM Trans. Audio Speech Lang. Process. 26, 2230–2239 (2018). https://doi.org/10.1109/TASLP.2018.2859777 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The review included only papers with sufficient experimental proof on benchmarked datasets, state-of-the-art methodology, comparisons with existing work, and dataset introductions.

“We included only those papers exhibiting sufficient experimental proof with their models experimented on benchmarked datasets, papers which introduced the state-of-the-art methodology used for the purpose of question generation, the articles which compare their proposed models with existing work. We also included the articles which introduced various datasets for benchmarking this problem area.”

• AssumptionRests on an assumption: The journals and conferences from which papers were selected are of repute.
• AssumptionRests on an assumption: The inclusion criteria used in the review ensure the quality of the surveyed content.
• AssumptionRests on an assumption: The selected papers are representative of the field.

May combine two claims.

Two main approaches exist for open-domain questions: one uses syntactic structure and NLP operations, the other uses an end-to-end neural machine translation-like method.

“There are two major approaches for open-domain questions, which have been researched upon. The first approach makes use of the syntactic structure of sentences and other natural language processing operations to produce a question from a specified sentence, and the second approach makes use of an end-to-end method which uses an approach similar to machine language translation using neural networks”

• cites [8] Varga, A., Ha, L.: Wlv: A question generation system for the qgstec 2010 task b. Proceedings of the Third Workshop on Question Generation. pp. 80–83 (2010)Not checked yet

  Not checked in the free run.

• cites [10] Flor, M., Riordan, B.: A Semantic Role-based Approach to Open-Domain Automatic Question Generation. pp. 254–263 (2018). https://doi.org/10.18653/v1/w18-0530 sourceNot checked yet

  Not checked in the free run.

• cites [11] Serban, I.V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., Bengio, Y.: Generating factoid questionswith recurrent neural networks: the 30M factoid question-answer corpus. In: 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers. 1, 588–598 (2016). https://doi.org/10.18653/v1/p16-1056 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

In conversational systems, consistency with the conversation context and maintaining user interest are essential.

“On the other hand, in conversational systems, say, a question-asking bot, it is imperative to be consistent with the context of the conversation and also at the same time maintain the interest of the user in the conversation [2].”

• cites [2] Gu, J., Mirshekari, M., Yu, Z., Sisto, A.: ChainCQG: Flow-aware conversational question generation. In: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2061–2070 (2021) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Generative adversarial networks, introduced in [38], consist of a generator and a discriminator that compete adversarially.

“The generative adversarial networks (GANs) were introduced in [38]. This class of deep neural networks makes use of two different networks, the generator and discriminator, which compete against each other in an adversarial set-up.”

• cites [38] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. In: Neural Information Processing Systems (2014)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The review searched for relevant articles using key phrases such as automatic question generation, question generation, and visual question generation.

“We used key phrases like automatic question generation, question generation, visual question generation and the like for searching for relevant research articles in this field.”

• AssumptionRests on an assumption: The journals and conferences from which papers were selected are of repute.
• AssumptionRests on an assumption: The selected papers are representative of the field.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [57] uses a joint question answering and question generation approach in which the question is both asked and answered.

“For instance, in [57], the approach used is one where the question is asked as well as answered.”

• cites [57] Wang, T., Yuan, X., Trischler, A.: A Joint Model for Question Answering and Question Generation. Presented at the (2017)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In [99], informativeness is measured by the extra information in a student's answer that was not in the conversation history.

“The informativeness is measured using what additional information a given answer by the student provides which was not present the history of the conversation so far.”

• cites [99] Qi, P., Zhang, Y., Manning, C.D.: Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations. pp. 25–40 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The GPT-2-based model of [63] gave average results on BLEU, METEOR, and ROUGE.

“The model was evaluated on automatic metrics of BLEU, METEOR, and ROUGE and found to give average results.”

• cites [63] Lopez, L.E., Cruz, D.K., Cruz, J.C.B., Cheng, C.: Simplifying Paragraph-level Question Generation via Transformer Language Models. (2020) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [65] combines the GPT-2 transformer decoder with the BERT transformer encoder.

“In [65], the authors use a combination of the transformer-based decoder of the GPT-2 [66] model with the transformer encoder of BERT [67].”

• cites [65] Klein, T., Nabi, M.: Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds. (2019)Not checked yet

  Not checked in the free run.

• cites [66] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners. Presented at the (2018)Not checked yet

  Not checked in the free run.

• cites [67] Devlin, J., Chang, M.-W., Lee, K., Google, K.T., Language, A.I.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Naacl-Hlt 2019 (2018)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In [61], the model using ELMO, placeholding, and copying gave better results on SQuAD.

“The models were evaluated on automatic metrics (BLEU, ROUGE) as well as human evaluation for correctness, fluency, soundness, answerability, and relevance and found that the model employing the ELMO+placeholding+copy mechanism gave better results on SQuAD.”

• cites [61] Scialom, T., Piwowarski, B., Staiano, J.: Self-attention architectures for answer-agnostic neural question generation. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 6027–6032 (2020). https://doi.org/10.18653/v1/p19-1604 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [53] introduces two reward functions, QSS and ANSS, for evaluating generated question quality.

“The authors also introduce two new reward functions for evaluating the quality of the generated questions, namely Question Sentence Overlap Score (QSS) and Predicted and Encoded Answer Overlap Score (ANSS).”

• cites [53] Kumar, V., Ramakrishnan, G., Li, Y.-F.: Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text. arXiv:1808.04961 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The [96] experiments used CoQA and BLEU scores, and its authors recommend new metrics because existing automatic metrics are weak for conversational question generation.

“The experiments were performed on the CoQA dataset and evaluated using BLEU scores although the authors suggest the development of new metrics for conversational question generation due to weakness of the existing automatic metrics for evaluation.”

• cites [96] Nakanishi, M., Kobayashi, T., Hayashi, Y.: Towards Answer-unaware Conversational Question Generation. pp. 63–71 (2019). Doi: https://doi.org/10.18653/v1/d19-5809 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The recurrent BERT-based model of [68] gave better results on sentence-level and paragraph-level question generation.

“In comparison with other models using standard evaluation metrics, their model gave better results on both sentence-level and paragraph-level question generation.”

• cites [68] Chan, Y.-H., Fan, Y.-C.: A Recurrent BERT-based Model for Question Generation. pp. 154–162 (2019). Doi: https://doi.org/10.18653/v1/d19-5821 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The [99] architecture has two parts: an automatic question generation model and a model measuring informativeness and specificity of the generated question.

“The architecture used in this work consists of two fragments: the automatic question generation model and the informativeness and specificity measurement model for the generated question.”

• cites [99] Qi, P., Zhang, Y., Manning, C.D.: Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations. pp. 25–40 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [60] attempted question generation from passages on SQuAD using transformers.

“In [60], the task of question generation from passages was attempted on the SQuAD dataset using transformers.”

• cites [60] Kriangchaivech, K., Wangperawong, A.: Question Generation by Transformers. (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The [2] model outperforms several baselines, including T5 and BART-large.

“The model outperforms several baselines considered, including T5 model and BART-large.”

• cites [2] Gu, J., Mirshekari, M., Yu, Z., Sisto, A.: ChainCQG: Flow-aware conversational question generation. In: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2061–2070 (2021) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [53] creates a deep reinforcement learning framework for generating intelligent questions in conversational systems.

“In [53], authors have attempted to create a framework for generating intelligent questions in the context of conversational systems.”

• cites [53] Kumar, V., Ramakrishnan, G., Li, Y.-F.: Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text. arXiv:1808.04961 (2018) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In [98], question classification based on Lehnert's classification is used to tag questions and then in a conditional neural network model.

“An approach using question classification based on Lehnert’s classification [97] is used to tag questions and later on used in a conditional neural network-based model in [98].”

• cites [97] Colclough, M., Lehnert, W.G.: The Process of Question Answering -- A Computer Simulation of Cognition. (1979) sourceNot checked yet

  Not checked in the free run.

• cites [98] Krishna, K., Iyyer, M.: Generating question-answer hierarchies. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2321–2334 (2020). https://doi.org/10.18653/v1/p19-1224 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[98] introduces SQUASH, a task that converts text into a hierarchy of question-answer sets from broad high-level questions to more refined questions.

“The authors introduce a new task called SQUASH (Specificity-controlled Question Answer Hierarchies) which converts the text into a hierarchy consisting of question–answer sets which start at broader “high-level” questions and keep proceeding with more refined questions down the hierarchy.”

• cites [98] Krishna, K., Iyyer, M.: Generating question-answer hierarchies. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2321–2334 (2020). https://doi.org/10.18653/v1/p19-1224 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [56] proposes a graph-to-sequence architecture guided by deep reinforcement learning.

“A graph-to-sequence-based architecture guided by deep reinforcement learning is proposed in [56].”

• cites [56] Chen, Y., Wu, L., Zaki, M.J.: Reinforcement Learning Based Graph-To-Sequence Model for Natural Question Generation. In: Iclr 2020. pp. 498–515 (2020)Not checked yet

  Not checked in the free run.

May combine two claims.

Reference [72] was the first paper to discuss VQG and released three public datasets.

“This was the first paper that discussed the task of VQG and released 3 public datasets for the research community to solve this problem using different models.”

• cites [72] Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L.: Generating natural questions about an image. In: 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers. 3, 1802–1813 (2016). https://doi.org/10.18653/v1/p16-1170 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Transformers remember sequences better than encoder-decoder architectures based on recurrent networks.

“This is because transformers perform much better at remembering sequences than encoder–decoder architectures based on recurrent nets.”

May combine two claims.

Because automatic evaluation is not an effective measure of question quality, human evaluation is also performed.

“Hence, evaluation is also performed by human evaluation techniques.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Other human evaluation approaches use naturalness and difficulty.

“Other approaches make use of naturalness [32, 35] and difficulty [35].”

• cites [32] Du, X., Shao, J., Cardie, C.: Learning to ask: Neural question generation for reading comprehension. In: ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). vol. 1, pp. 1342–1352 (2017). https://doi.org/10.18653/v1/P17-1123 sourceNot checked yet

  Not checked in the free run.

• cites [35] Hu, W., Liu, B., Ma, J., Zhao, D., Yan, R.: Aspect-based question generation. In: 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings. pp. 1–10 (2018)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[99] addresses generating informative questions.

“The problem of generating informative questions is dwelled upon in [99].”

• cites [99] Qi, P., Zhang, Y., Manning, C.D.: Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations. pp. 25–40 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[2] uses GPT-2 to encode the answer and generate questions, and its flow-propagation training considers losses after n turns to improve conversational flow.

“For encoding the answer and for question generation, the GPT-2 model [66] is used.”

• cites [2] Gu, J., Mirshekari, M., Yu, Z., Sisto, A.: ChainCQG: Flow-aware conversational question generation. In: EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2061–2070 (2021) sourceNot checked yet

  Not checked in the free run.

• cites [66] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language Models are Unsupervised Multitask Learners. Presented at the (2018)Not checked yet

  Not checked in the free run.

May combine two claims.

Attention-based models are used in VQG and give good results.

“Also, attention-based models are used giving good results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The model of [56] outperformed earlier state-of-the-art models on automatic and human evaluation by a substantial margin on SQuAD.

“They evaluated the models on the SQuAD dataset and found that it outperformed the earlier state-of-the-art models on automatic evaluation metrics as well as human evaluation methods by a substantial margin.”

• cites [56] Chen, Y., Wu, L., Zaki, M.J.: Reinforcement Learning Based Graph-To-Sequence Model for Natural Question Generation. In: Iclr 2020. pp. 498–515 (2020)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reinforcement learning with reward-based training shows promising results in a few cases.

“In a few cases, reinforcement learning, which employs a reward-based training mechanism, also shows promising results.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The [98] pipeline was tested on SQuAD, QuAC, and CoQA and produced promising results.

“The authors test their proposed pipeline on the 3 datasets (SQuAD, QuAC, CoQA) and get promising results.”

• cites [98] Krishna, K., Iyyer, M.: Generating question-answer hierarchies. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. pp. 2321–2334 (2020). https://doi.org/10.18653/v1/p19-1224 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

For extracting image-related features, ResNet variants have been explored in most VQG techniques.

“For extracting image-related features, ResNet variants have been explored in most techniques.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In [12, 94], three crowd-workers rate questions on a 1-to-5 scale for fluency and relevance.

“An approach in which three crowd-workers are used for rating questions based on a scale of 1 to 5 (5 being good) on two parameters fluency and relevance is employed in [12, 94].”

• cites [12] Du, X., Cardie, C.: Identifying where to focus in reading comprehension for neural question generation. In: EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings. pp. 2067–2073 (2017). https://doi.org/10.18653/v1/d17-1219 sourceNot checked yet

  Not checked in the free run.

• cites [94] Du, X., Cardie, C.: Harvesting paragraph-level question-answer pairs from wikipedia. In: ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). 1, 1907–1917 (2018). https://doi.org/10.18653/v1/p18-1177 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The primary use-case of conversational question generation systems is conversational chatbots.

“The primary use-case of such a system is conversational chatbots.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In [60], word error rate was low for shorter questions and increased for longer questions.

“WER was low for the shorter questions, while it increased for longer questions.”

• cites [60] Kriangchaivech, K., Wangperawong, A.: Question Generation by Transformers. (2019)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A neural-network method in [91] uses coreference alignment and conversational flow so that questions in later turns relate to the conversation history.

“In [91], a neural-network-based approach is proposed which makes use of coreference alignment along with maintaining a conversational flow.”

• cites [91] Gao, Y., Li, P., King, I., Lyu, M.R.: Interconnected question generation with coreference alignment and conversation flow modeling. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. pp. 4853–4862 (2020). https://doi.org/10.18653/v1/p19-1480 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The [91] model uses a multi-source encoder and an attention-and-copy decoder, and was tested on CoQA against baselines and ablations.

“A multi-source encoder along with a decoder based on attention and copy mechanism is employed for this task.”

• cites [91] Gao, Y., Li, P., King, I., Lyu, M.R.: Interconnected question generation with coreference alignment and conversation flow modeling. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. pp. 4853–4862 (2020). https://doi.org/10.18653/v1/p19-1480 sourceNot checked yet

  Not checked in the free run.

• cites [92] Reddy, S., Chen, D., Manning, C.D.: CoQA: A Conversational Question Answering Challenge. In: Transactions of the Association for Computational Linguistics. pp. 249–266 (2019) sourceNot checked yet

  Not checked in the free run.

• cites [93] See, A., Liu, P.J., Manning, C.D.: Get To The Point: Summarization with Pointer-Generator Networks. In: 55th Annual Meeting of the Association for Computational Linguistics. pp. 1073–1083 (2017) sourceNot checked yet

  Not checked in the free run.

• cites [94] Du, X., Cardie, C.: Harvesting paragraph-level question-answer pairs from wikipedia. In: ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). 1, 1907–1917 (2018). https://doi.org/10.18653/v1/p18-1177 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Answer-unaware conversational question generation is explored in [96].

“Answer-unaware conversational question generation is explored in [96].”

• cites [96] Nakanishi, M., Kobayashi, T., Hayashi, Y.: Towards Answer-unaware Conversational Question Generation. pp. 63–71 (2019). Doi: https://doi.org/10.18653/v1/d19-5809 sourceNot checked yet

  Not checked in the free run.

May combine two claims. The quote only partly states this claim.

The [95] model also tested on SQuAD for multi-turn question-answer conversations and outperformed several baselines on automatic and human evaluation metrics.

“The experimental analysis proves that the model gives better results than several compared baselines using automatic and human-based evaluation metrics.”

• cites [95] Pan, B., Li, H., Yao, Z., Cai, D., Sun, H.: Reinforced dynamic reasoning for conversational question generation. In: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 2114–2124 (2020). https://doi.org/10.18653/v1/p19-1203 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors of [55] conclude that their approach could produce comparatively readable questions.

“The authors conclude that their approach could produce comparatively readable questions.”

• cites [55] Liu, Y., Zhang, C., Yan, X., Chang, Y., Yu, P.S.: Generative question refinement with deep reinforcement learning in retrieval-based QA system. In: International Conference on Information and Knowledge Management, Proceedings. pp. 1643–1652 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Reference [55] proposes refining ill-formed questions into well-formed questions using a reinforcement learning reward-based mechanism.

“In [55], the authors propose a refinement of ill-formed questions generated to well-formed questions by using a reward-based mechanism using reinforcement learning for training a deep learning model.”

• cites [55] Liu, Y., Zhang, C., Yan, X., Chang, Y., Yu, P.S.: Generative question refinement with deep reinforcement learning in retrieval-based QA system. In: International Conference on Information and Knowledge Management, Proceedings. pp. 1643–1652 (2019) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Closed-domain question generation can be applied to healthcare bots that interview patients about specific symptoms for preliminary disease investigation.

“Closed-domain question generation can be applied to healthcare bots where the bots can interview patients for specific symptoms for the preliminary investigation of diseases.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some applications used datasets such as SQuAD and MS-MARCO, while some works curated their own datasets for distinctive applications.

“For a few of these applications, some datasets were used like SQuAD, MS-MARCO and the like. Some works curated their own dataset owing to the distinctive application involved [122, 124].”

• cites [122] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123, 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7 sourceNot checked yet

  Not checked in the free run.

• cites [124] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access. 10, 63522–63531 (2022). https://doi.org/10.1109/ACCESS.2022.3180838 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[124] found a significant difference in how answers were selected in educational versus non-educational question generation.

“This study found that there was a significant difference in which the answers were selected in educational and non-educational question generation.”

• cites [124] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access. 10, 63522–63531 (2022). https://doi.org/10.1109/ACCESS.2022.3180838 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Open-domain generation systems can work across use-cases that are not restricted to a limited set of scenarios.

“Open-domain generation systems can be used for working across use-cases that are not restricted to a limited collection of scenarios.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The types of questions generated range from typical short answer span questions to multiple spans for text question generation.

“The type of questions to be generated range from the typical short answer span questions to multiple spans for question generation from the text [42, 47, 106, 107, 112].”

• cites [42] Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ Questions for Machine Comprehension of Text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 2383–2392. Association for Computational Linguistics, Stroudsburg, PA, USA (2016) sourceNot checked yet

  Not checked in the free run.

• cites [47] Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., Suleman, K.: NewsQA: A Machine Comprehension Dataset. pp. 191–200 (2017). https://doi.org/10.18653/v1/w17-2623 sourceNot checked yet

  Not checked in the free run.

• cites [106] Chen, G., Yang, J., Hauff, C., Houben, G.J.: LearningQ: A large-scale dataset for educational question generation. In: 12th International AAAI Conference on Web and Social Media, ICWSM 2018. pp. 481–490 (2018) sourceNot checked yet

  Not checked in the free run.

• cites [107] Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K.M., Melis, G., Grefenstette, E.: The NarrativeQA reading comprehension challenge. Trans. Assoc. Comput. Linguist. 6, 317–328 (2018). https://doi.org/10.1162/tacl_a_00023 sourceNot checked yet

  Not checked in the free run.

• cites [112] Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: RACE: Large-scale ReAding comprehension dataset from examinations. In: EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings. pp. 785–794 (2017). https://doi.org/10.18653/v1/d17-1082 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Standalone question generation has been applied to education, news, and social media.

“If we consider the different domains, standalone question generation has been applied to education [123], news [124] and social media [126].”

• cites [123] Johnson, J., Fei-Fei, L., Hariharan, B., Zitnick, C.L., van der Maaten, L., Girshick, R.: CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pp. 1988–1997 (2017). Doi: https://doi.org/10.1109/CVPR.2017.215 sourceNot checked yet

  Not checked in the free run.

• cites [124] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access. 10, 63522–63531 (2022). https://doi.org/10.1109/ACCESS.2022.3180838 sourceNot checked yet

  Not checked in the free run.

• cites [126] Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., Hajishirzi, H.: Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pp. 5376–5384 (2017) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[132] discusses desiderata for generating cloze and WH-questions.

“In [132], a desiderata for generating cloze type and WH-questions has been discussed.”

• cites [132] Corbett, A., Mostow, J.: Automating comprehension questions : lessons from a reading tutor. In: Workshop on the Question Generation Shared Task and Evaluation Challenge (2008)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In [104], human evaluators rate generated questions on syntactic correctness, semantic correctness, and relevance.

“In [104], human evaluators were asked to rate the quality of the generated questions based on three factors, syntactic correctness, semantic correctness, and relevance.”

• cites [104] Liu, Z., Huang, K., Huang, D., Zhao, J.: Semantics-reinforced networks for question generation. Front. Artif. Intell. Appl. 325, 2078–2084 (2020). https://doi.org/10.3233/FAIA200330 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Generating questions for passages is useful in education because it saves the time and effort needed to set question papers.

“A very important use case is generating questions for passages. This can be useful in an educational setup where the input will be passages of text and will result in saving the time and effort required for setting question papers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The [105] authors showed that their proposed metrics correlate better with human evaluation scores.

“They further proved that their proposed metrics had a better correlation with the human evaluation scores.”

• cites [105] Nema, P., Khapra, M.M.: Towards a better metric for evaluating question generation systems. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. pp. 3950–3959 (2020). https://doi.org/10.18653/v1/d18-1429 sourceNot checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: Human evaluation scores are a valid criterion for question quality.

May combine two claims.

Most works used factoid-based questions, but Yes/No, MCQ, and reasoning-type questions were also generated for some applications.

“Also, most works used factoid-based questions but Yes/No [133], MCQ type [124] and reasoning type questions [122] were also generated for some applications.”

• cites [133] Varga, A., Ha, L.: Wlv: A question generation system for the qgstec 2010 task b. In: Proceedings of the Third Workshop on Question Generation. pp. 80–83 (2010)Not checked yet

  Not checked in the free run.

• cites [124] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access. 10, 63522–63531 (2022). https://doi.org/10.1109/ACCESS.2022.3180838 sourceNot checked yet

  Not checked in the free run.

• cites [122] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123, 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[133] describes a system that generates simple factoid questions using syntactic rules by question type.

“In [133], a system for generating questions is discussed, by generating simple factoid questions using syntactic rules by question types.”

• cites [133] Varga, A., Ha, L.: Wlv: A question generation system for the qgstec 2010 task b. In: Proceedings of the Third Workshop on Question Generation. pp. 80–83 (2010)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Visual question generation has been used to generate meaningful questions on radiology images in [136].

“Visual question generation, for instance, has been used for generating meaningful questions on radiology images in [136].”

• cites [136] Sarrouti, M., ben Abacha, A., Demner-Fushman, D.: Visual question generation from radiology images. In: Proceedings of the First Workshop on Advances in Language and Vision Research. pp. 12–18. Association for Computational Linguistics, Stroudsburg, PA, USA (2020) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Visual question generation has been deployed in visual dialogue generation in a few works.

“Visual question generation has been deployed in visual dialogue generation in a few works [131, 133, 134].”

• cites [131] Ji, T., Lyu, C., Jones, G., Zhou, L., Graham, Y.: QAScore—an unsupervised unreferenced metric for the question generation evaluation. Entropy 24, 1514 (2022). https://doi.org/10.3390/e24111514 sourceNot checked yet

  Not checked in the free run.

• cites [133] Varga, A., Ha, L.: Wlv: A question generation system for the qgstec 2010 task b. In: Proceedings of the Third Workshop on Question Generation. pp. 80–83 (2010)Not checked yet

  Not checked in the free run.

• cites [134] Graesser, A.C., Rus, V., Cai, Z.: Question classification schemes. In: The Workshop on Question Generation. pp. 8–9 (2008)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[128] investigated a generator and pre-tester architecture for joint QA-QG from videos, using a video encoder for 20 frames and faster R-CNN and Resnet101.

“In [128], using a newly constructed architecture of a generator, which generates a question given a video clip and an answer, and a pre-tester, which tries to answer the generated question, has been investigated for joint QA-QG from videos.”

• cites [128] Su, H.T., Chang, C.H., Shen, P.W., Wang, Y.S., Chang, Y.L., Chang, Y.C., Cheng, P.J., Hsu, W.H.: End-to-end video question-answer generation with generator-pretester network. IEEE Trans. Circuits Syst. Video Technol. 31, 4497–4507 (2021). https://doi.org/10.1109/TCSVT.2021.3051277 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Conversational question generation has largely been applied to generating questions from multiple documents and in the medical domain.

“Conversational question generation largely has seen applications in the generation of questions from multiple documents field as also the medical domain.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[134] presents classification schemes for questions, and [135] discusses having students generate questions to improve meta-cognitive abilities.

“Classification schemes for questions are presented in [134] while asking students to generate questions themselves for improving meta-cognitive abilities has been discussed in [135].”

• cites [134] Graesser, A.C., Rus, V., Cai, Z.: Question classification schemes. In: The Workshop on Question Generation. pp. 8–9 (2008)Not checked yet

  Not checked in the free run.

• cites [135] Yu, F.Y., Liu, Y.H., Chan, T.W.: A web-based learning system for question-posing and peer assessment. Innov. Educ. Teach. Int. 42, 337–348 (2005). https://doi.org/10.1080/14703290500062557 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some datasets are open-domain, such as general English passages from Wikipedia, while closed-domain datasets use news, educational content, or search queries.

“Some datasets are open domain like general English passages curated from Wikipedia, while other closed-domain datasets make use of news(NewsQA, DeepMind), educational content (RACE, NarrativeQA) and some are from search queries(WIKIQA, MSMARCO).”

• cites NewsQANot checked yet

  Not checked in the free run.

• cites DeepMindNot checked yet

  Not checked in the free run.

• cites RACE, NarrativeQANot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Using multiple modalities can target more real-world applications, including educational questions based on pictures and diagrams and helping visually challenged people identify objects.

“Using multiple modalities, more real-world applications can be targeted, including generating questions based on pictures and diagrams in educational domain [126] and helping the visually challenged identify objects around them [127].”

• cites [126] Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., Hajishirzi, H.: Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pp. 5376–5384 (2017) sourceNot checked yet

  Not checked in the free run.

• cites [127] Patel, A., Bindal, A., Kotek, H., Klein, C., Williams, J.: Generating natural questions from images for multimodal assistants. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. pp. 2270–2274 (2021) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[125] used pre-trained transformer models on the curated dataset and achieved promising results.

“After analysis of the type of questions, they used pre-trained transformer models to train on the dataset and achieved promising results.”

• cites [125] Lehman, E., Lialin, V., Legaspi, K.Y., Sy, A.J.R., Pile, P.T.S., Alberto, N.R.I., Ragasa, R.R.R., Puyat, C.V.M., Alberto, I.R.I., Alfonso, P.G.I., Taliño, M., Moukheiber, D., Wallace, B.C., Rumshisky, A., Liang, J.J., Raghavan, P., Celi, L.A., Szolovits, P.: Learning to Ask Like a Physician. In: ClinicalNLP 2022 - 4th Workshop on Clinical Natural Language Processing, Proceedings. pp. 74–86 (2022). https://doi.org/10.18653/v1/2022.clinicalnlp-1.8 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[137] attempts to use a reference-less metric for text-to-text evaluation tasks.

“For this purpose, a recent work, which was proposed in [137], attempts to use a reference less metric for text-to-text evaluation tasks.”

• cites [137] Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., Gallinari, P.: Data-QuestEval: A reference-less metric for data-to-text semantic evaluation. In: EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. 8029–8036 (2021). https://doi.org/10.18653/v1/2021.emnlp-main.633 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A conversational bot able to generate relevant natural-language questions can speed up the process of diagnosing a patient.

“A conversational bot with the ability to generate relevant questions in natural language can thus aid in speeding up the process of diagnosing a patient.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In [124], transfer learning trains on non-educational datasets SQuAD and NQA and evaluates on the author-curated TQA-A dataset of educational text with answer tags.

“For instance, in [124], transfer learning is performed by training on non-educational datasets like SQuAD and NQA (Natural Questions) and the evaluation is performed on an author curated dataset called TQA-A with questions based on educational text and tagged with answers.”

• cites [124] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access. 10, 63522–63531 (2022). https://doi.org/10.1109/ACCESS.2022.3180838 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

[128] obtained promising results on the TVQA and ActivityNetQA datasets.

“Overall, they obtained promising results on the TVQA [129] and ActivityNetQA [130] datasets.”

• cites [129] Lei, J., Yu, L., Bansal, M., Berg, T.L.: TVQA: Localized, compositional video question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. pp. 1369–1379 (2018). https://doi.org/10.18653/v1/d18-1167 sourceNot checked yet

  Not checked in the free run.

• cites [130] Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: ActivityNet-QA: A dataset for understanding complex web videos via question answering. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019. pp. 9127–9134 (2019). Doi: https://doi.org/10.1609/aaai.v33i01.33019127 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Generating meaningful questions is a challenge because most existing techniques focus more on syntax than on information-extracting questions.

“Also, generating meaningful questions is a challenge as most existing techniques focus more on the syntactical aspects rather than information extracting questions.”

Classified as probably not a substantial claim.

Transfer learning is useful when training data are limited and pre-trained models exist on similar datasets.

“Transfer learning helps in cases where the data for training is less, and we have pre-trained models which exist on other similar datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In [131], QAScore is proposed using the pre-trained language model RoBERTa; it is reference-free and correlates well with human judgments.

“In [131], an evaluation metric called QAScore is proposed which makes use of pre-trained language model RoBERTa (Robustly Optimized BERT Pre-training Approach).”

• cites [131] Ji, T., Lyu, C., Jones, G., Zhou, L., Graham, Y.: QAScore—an unsupervised unreferenced metric for the question generation evaluation. Entropy 24, 1514 (2022). https://doi.org/10.3390/e24111514 sourceNot checked yet

  Not checked in the free run.