CoChat
Sign Up
PreprintCited by 16Natural Language Processing TechniquesTopic ModelingArtificial Intelligence in Education

The DOI matches its DataCite record.

Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation

Open access
Laban, Philippe, Wu, Chien-Sheng, Murakhovs'ka, Lidiya, Liu, Wenhao, Xiong, Caiming·2022·arXiv
DOI 10.48550/arxiv.2205.01730arXiv 2205.01730
Open this paper in CoChat — free accountOpen PDF

Abstract

Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting, focusing on the use case of helping teachers automate the generation of reading comprehension quizzes. In our study, teachers building a quiz receive question suggestions, which they can either accept or refuse with a reason. Even though we find that recent progress in QGen leads to a significant increase in question acceptance rates, there is still large room for improvement, with the best model having only 68.4% of its questions accepted by the ten teachers who participated in our study. We then leverage the annotations we collected to analyze standard NLG metrics and find that model performance has reached projected upper-bounds, suggesting new automatic metrics are needed to guide QGen research forward.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2205.01730

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies.

    “Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality.”

    From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Laban et al., 2022

  • Existing QG evaluation benchmarks usually employ a limited variety of models to generate questions, resulting in a lack of diversity in the data.

    “3) employing a limited variety of models to generate questions, resulting in a lack of diversity in the data.”

    From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Laban et al., 2022

    Section 3.3 says this study used seven QGen models of varying size and architecture, limiting the number for latency and user burden. The Introduction and Limitations do not say that existing evaluation benchmarks usually use a limited variety of models or that this causes a lack of data diversity. Searches for benchmark model variety and question diversity found no passage addressing that claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Evaluation methodologies in automatic question gen- eration 2013-2018. In Proceedings of the 11th Inter- national Conference on Natural Language Genera- tion, pages 307-317.
  2. Jun Araki, Dheeraj Rajagopal, Sreecharan Sankara- narayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. 2016. Generating questions and multiple-choice answers using semantic analysis of texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 1125-1136.
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
  4. Jacob Benesty, Jingdong Chen, Yiteng Huang, and Is- rael Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer.
  5. Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? analyzing clarification questions in cqa. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pages 345-348.
  6. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 643-653.
  7. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1342-1352.
  8. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artifi- cial Intelligence Research, 61:65-170.
  9. Tom Hosking and Sebastian Riedel. 2019. Evaluating rewards for question generation models. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2278-2283.
  10. Sathish Reddy Indurthi, Dinesh Raghu, Mitesh M Khapra, and Sachindra Joshi. 2017. Generating natu- ral language question-answer pairs from a knowledge graph using a rnn based question generation model. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 376-385.
  11. Panayiota Kendeou, Kristen L McMaster, and Theodore J Christ. 2016. Reading comprehension: Core components and processes. Policy Insights from the Behavioral and Brain Sciences, 3(1):62-69.
  12. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
  13. Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of auto- matic question generation for educational purposes. International Journal of Artificial Intelligence in Ed- ucation, 30(1):121-204.
  14. Philippe Laban, John Canny, and Marti A Hearst. 2020. What's the latest? a question-driven news chatbot. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 380-387.
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
  16. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
  17. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica, pages 0455-463.
  18. Robert B Miller. 1968. Response time in man-computer conversational transactions. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I, pages 267-277.
  19. Lidiya Murakhovs'ka, Chien-Sheng Wu, Philippe La- ban, Tong Niu, Wenhao Liu, and Caiming Xiong. 2022. Mixqg: Neural question generation with mixed answer types. Findings of the North American Chap- ter of the Association for Computational Linguistics: NAACL 2022.
  20. Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950-3959.
  21. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252.
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
  23. P David Pearson and Margaret C Gallagher. 1983. The instruction of reading comprehension. Contempo- rary educational psychology, 8(3):317-344.
  24. Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2401- 2410.
  25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog 1.8 (2019): 9.
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
  27. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions us- ing neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2737-2746.
  28. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. · arXiv
  29. Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & Analgesia, 126(5):1763- 1768.
  30. Tim Steuer, Anna Filighera, Tobias Meuser, and Christoph Rensing. 2021. I do not understand what i cannot define: Automatic question generation with pedagogically-driven content selection. arXiv preprint arXiv:2110.04123. ten epochs on the training portion of the SQuAD dataset (Rajpurkar et al., 2016), using the ADAM optimizer (Kingma and Ba, 2015), with hyper- parameter tuning based on model loss on the vali- dation set. The model checkpoint that achieves the lowest validation loss is selected as the final model. Selected hyper-parameters were: Distil-GPT2: batch-size 32, learning rate 2 * 10 -5 . GPT2-base: batch-size 32, learning rate 2 * 10 -5 . GPT2-medium: batch-size 16, learning rate 2 * 10 -5 . BART-base: batch-size 32, learning rate 1 * 10 -4 . BART-large: batch-size 32, learning rate 2 * 10 -5 . Finally, the last two QGen we used are publicly available on the HuggingFace model hub (Wolf et al., 2020), and we use them as is: ProphetNet: microsoft/ prophetnet-large- uncased-squad-qg MixQG: Salesforce/mixqg-large With all models, we used beam search to gen- erate candidate questions, using a beam-size of 2, and a sequence length maximum of 30. · arXiv

Cited by

Sorted by citations

  • QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

    Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16

    “Laban et al. (2022) utilized seven QG models to generate 1k+ questions to be evaluated, however, they merely assessed whether the generated questions could be accepted as reading comprehension quiz questions, rather than scoring them on multiple dimensions.”
    BackgroundMethod

Metadata

Authors
Laban, P., Wu, C.-S., Murakhovs'ka, L., Liu, W., & Xiong, C.
Affiliations
Salesforce (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
102-111
DOI
10.48550/arxiv.2205.01730
arXiv
2205.01730
OpenAlex
W4229084000
Cited by
16
References
0
Citations by year
2025: 1 · 2024: 6 · 2023: 6 · 2022: 3
FWCI
2.21
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
question generation, teacher feedback, natural language generation evaluation, n-gram overlap, automatic evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review