Evaluation methodologies in Automatic Question Generation 2013-2018
Open accessAbstract
In the last few years Automatic Question Generation (AQG) has attracted increasing interest.In this paper we survey the evaluation methodologies used in AQG.Based on a sample of 37 papers, our research shows that the systems' development has not been accompanied by similar developments in the methodologies used for the systems' evaluation.Indeed, in the papers we examine here, we find a wide variety of both intrinsic and extrinsic evaluation methodologies.Such diverse evaluation practices make it difficult to reliably compare the quality of different generation systems.Our study suggests that, given the rapidly increasing level of research in the area, a common framework is urgently needed to compare the performance of AQG systems and NLG systems more generally.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics.
“However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics.”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Amidei et al., 2018
Section 4 (Discussion) says human evaluations use different criteria and scales, making it difficult to compare question quality across generation systems. It also says the lack of comparisons between human and automatic evaluations makes it difficult to assess how well automatic metrics capture system quality. The paper does not specifically say that the lack of unified human criteria causes that difficulty with automatic metrics. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 28
- Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Con- ference on Computational Linguistics, pages 3318- 3329.
- Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computa- tional Linguistics, 34(4):555-596.
- Satanjeev Banerjee and Alon Laviel. 2005. Meteor: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72.
- Petra S. Bayerl and Karsten I. Paul. 2011. What deter- mines inter-coder agreement in manual annotations? a meta-analytic investigation. Computational Lin- guistics, 37(4):699-725.
- Jean Carletta. 1996. Assessing agreement on classi- fication tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
- Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar- garet Mitchell, Jianfeng Gao, and Bill Dolan. 2015. δbleu: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 26-31.
- Albert Gatt and Anja Belz. 2010. Introducing shared tasks to NLG: The TUNA shared task evaluation challenges. in E. Krahmer and Mariët Theune (Eds.), Empirical Methods in Natural Language Generation Springer-Verlag, Berlin Heidelberg.
- Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artifi- cial Intelligence Research, 61(1):65-170.
- Dimitra Gkatzia and Saad Mahamood. 2014. A snap- shot of NLG evaluation practices 2005 -2014. In Proceedings of the 15th European Workshop on Nat- ural Language Generation (ENLG), pages 57-60.
- Keith Godwin and Paul Piwek. 2016. Collecting reli- able human judgements on machine-generated lan- guage: The case of the qg-stec data. In Proceedings INLG16, pages 212-216.
- Chung Yong Lim Hwee Tou Ng and Shou King Foo. 1999. A case study on inter-annotator agreement for word sense disambiguation. In Proceedings of the ACL SIGLEX Workshop: Standardizing Lexical Re- sources, pages 9-13.
- Emiel Krahmer and Mariët Theune (Eds.). 2010. Em- pirical Methods in Natural Language Generation. Springer-Verlag, Berlin Heidelberg.
- Klaus Krippendorff. 1980. Content Analysis: An In- troduction to Its Methodology. Sage Publications, Beverly Hills, CA.
- Nguyen-Thinh Le, Tomoko Kojiri, and Niels Pinkwart. 2014. Automatic Question Generation for Educa- tional Applications The State of Art. In: van Do T., Thi H., Nguyen N. (eds) Advanced Computational Methods for Knowledge Engineering. Springer, Cham.
- Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using us- ing longest common subsequence and skip-bigram statistics. In Proceedings ACL'04, pages 605-612.
- Arle Lommel, Maja Popovic, and Aljoscha Burchardt. 2014. Assessing inter-annotator agreement for translation error annotation. In: MTE: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation, Reykjavik, Iceland.
- Jekaterina Novikova, Ondrȓej Dusek, and Verena Rieser. 2018. RankMe: Reliable human ratings for natural language generation. In Proceedings of NAACL-HLT, pages 72-78.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual meeting of the Association for Computational Linguistics., pages 311-318.
- Paul Piwek and Kristy. E. Boyer. 2012. Varieties of question generation: Introduction to this special is- sue. dialogue and discourse. Dialogue and Dis- course, 3(2):1-9.
- Sheetal Rakangor and Y. R. Ghodasara. 2015. Liter- ature review of automatic question generation sys- tems. International Journal of Scientific and Re- search Publications, 5(1):1-5.
- Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 44(3):393-401.
- Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evalu- ating natural language generation systems. Compu- tational Linguistics, 35(4):529-558.
- Ehud Reiter, Roma Robertson, and Liesl M.Osman. 2003. Lessons from a failure: Generating tailored smoking cessation letters. Artificial Intelligence, 144(1-2):41-58.
- Vasile Rus, Zhiqiang Cai, and Art Graesser. 2008. Question generation: Example of a multi-year eval- uation campaign. In Rus, V. and A. Graesser (eds.), online Proceedings of 1st Question Genera- tion Workshop, pages 25-26.
- Vasile Rus and Mihai C. Lintean. 2012. A compari- son of greedy and optimal assessment of natural lan- guage student input using word-to-word similarity metrics. In In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP.
- Vasile Rus, Brendan Wyse, Paul Piwek, Mihai Lintean, Svetlana Stoyanchev, and Cristian Moldovan. 2010. The first question generation shared task evaluation challenge. In Proceedings of the Sixth International Natural Language Generation Conference (INLG), pages 7-9.
- Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating nat- ural language generation. arXiv:1706.09799. · arXiv
- Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessan- dro Sordoni, Philip Bachman, and Saizheng Zhang. 2017. Machine comprehension by text-to-text neu- ral question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 15-25.
Cited by
Sorted by citations
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
Automatic question generation and answer assessment: a survey
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-021-00151-1 · Cited by 123 · see citing works
Leaf: Multiple-Choice Question Generation
Vachev, K., Hardalov, M., Karadzhov, G., Georgiev, G., Koychev, I., & Nakov, P. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-99739-7_41 · Cited by 45 · see citing works
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16 · see citing works
“However, the criteria of human evaluations are varied in existing research, leading to inconsistent and unreliable evaluations of QG models (Ji et al., 2022) and automatic metrics (Amidei et al., 2018; Mulla and Gharpure, 2023).”
Background
- Authors
- Amidei, J., Piwek, P., & Willis, A.
- Affiliations
- The Open University (GB)
- Venue
- Proceedings of the 11th International Conference on Natural Language Generation
- Published
- 2018
- Pages
- 307-317
- OpenAlex
- W2902583760
- Cited by
- 42
- References
- 0
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 4 · 2023: 9 · 2022: 10 · 2021: 6 · 2020: 7 · 2019: 3
- FWCI
- 1.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Generation (publishedVersion, cc-by)
- Open Research Online - ORO (The Open University) (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- automatic question generation, evaluation methodology, intrinsic evaluation, external evaluation, natural language generation, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
