Towards a Better Metric for Evaluating Question Generation Systems
Open accessAbstract
There has always been criticism for using ngram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems.However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc.Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task.In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc.In this work, we show that current automatic evaluation metrics based on n-gram similarity do not always correlate well with human judgments about answerability of a question.To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments.The scripts and data developed as a part of this work are made publicly available.1
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automatic evaluation of QG is still dominated by reference-based metrics such as BLEU and Q-Metric, which compute the similarity between generated questions and references.
“Automatic evaluation of QG is still dominated by reference-based metrics such as BLEU (Papineni et al., 2002) and Q-Metric”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Nema and Khapra, 2018
Existing QG evaluation benchmarks usually focus only on specific dimensions like answerability.
“1) focusing only on specific dimensions like answerability;”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Nema and Khapra, 2018
In [105], human-evaluated answerability is used to modify automatic metrics to include relevant words, question types, function words, and named entities.
“One notable contribution is made in [105], where the authors first use the answerability of a question through human evaluation to modify the existing automatic evaluation metrics to include the influence of relevant words, question types, function words, and named entities.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [105]
The [105] authors showed that their proposed metrics correlate better with human evaluation scores.
“They further proved that their proposed metrics had a better correlation with the human evaluation scores.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [105]
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
(2004) · Cited by 9,718 · see citing works
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,248 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
NLTK: The Natural Language ToolkitPreprint
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,893 · see citing works
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. R. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289273 · Cited by 1,603 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
MS MARCO: A Human Generated MAchine Reading COmprehension DatasetPreprint
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T. G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2025.5 · Cited by 1,190 · see citing works
(2016) · doi:10.18653/v1/d16-1147 · Cited by 1,035 · see citing works
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · cited 3× · see citing works
“…problem and create even more training data, there is growing interest in developing techniques that can automatically generate questions from a given source, say a document (Du et al., 2017; Du and Cardie, 2017), knowledge base (Reddy et al., 2017; Serban et al., 2016), or image (Li et al., 2017) .”
BackgroundRe-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
Liu, C., Lowe, R., Serban, I. V., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08023 · Cited by 603 · see citing works
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · see citing works
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 537 · see citing works
Large-scale Simple Question Answering with Memory Networks
Bordes, A., Usunier, N., Chopra, S., & Weston, J. (2015) · arXiv · doi:10.48550/arxiv.1506.02075 · Cited by 528 · see citing works
Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk
Callison-Burch, C. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09 · doi:10.3115/1699510.1699548 · Cited by 412 · see citing works
Good Question! Statistical Ranking for Question Generation
Heilman, M., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473390 · Cited by 366 · see citing works
Confidence estimation for machine translation
Blatz, J., Fitzgerald, E., Foster, G. F., Gandrabur, S., Goutte, C., Kulesza, A., Sanchís, A., & Ueffing, N. (2004) · International Conference on Computational Linguistics · doi:10.3115/1220355.1220401 · Cited by 356 · see citing works
The Meteor metric for automatic evaluation of machine translation
Lavie, A., & Denkowski, M. (2009) · Machine Translation · doi:10.1007/s10590-009-9059-4 · Cited by 351 · see citing works
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · see citing works
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · see citing works
Question Generation for Question Answering
Duan, N., Tang, D., Chen, P., & Zhou, M. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1090 · Cited by 271 · see citing works
Saha, A., Pahuja, V., Khapra, M. M., Sankaranarayanan, K., & Chandar, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11332 · Cited by 179 · see citing works
Self-citationVisual Question Generation as Dual Task of Visual Question Answering
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., & Zhou, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00640 · Cited by 171 · see citing works
Machine Comprehension by Text-to-Text Neural Question Generation
Yuan, X., Wang, T., Gülçehre, Ç., Sordoni, A., Bachman, P., Zhang, S., Subramanian, S., & Trischler, A. (2017) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w17-2603 · Cited by 158 · see citing works
Question Answering and Question Generation as Dual TasksPreprint
Tang, D., Duan, N., Qin, T., Yan, Z., & Zhou, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02027 · Cited by 155 · see citing works
Semi-Supervised QA with Generative Domain-Adaptive Nets
Yang, Z., Hu, J., Salakhutdinov, R., & Cohen, W. W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1096 · Cited by 142 · see citing works
Deep Questions without Deep Understanding
Labutov, I., Basu, S., & Vanderwende, L. (2015) · doi:10.3115/v1/p15-1086 · Cited by 141 · see citing works
Creativity: Generating Diverse Questions Using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.575 · Cited by 131 · see citing works
Generating Natural Language Questions to Support Learning On-Line
Lindberg, D., Popowich, F. P., Nesbit, J. C., & Winne, P. H. (2013) · Cited by 122 · see citing works
DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Saha, A., Aralikatte, R., Khapra, M. M., & Sankaranarayanan, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1156 · Cited by 111 · see citing works
Self-citationGenerating Instruction Automatically for the Reading Strategy of Self-Questioning
Jack, M., & Wei, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-465 · Cited by 101 · see citing works
Identifying Where to Focus in Reading Comprehension for Neural Question Generation
Du, X., & Cardie, C. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1219 · Cited by 100 · see citing works
Reddy, S., Raghu, D., Khapra, M. M., & Joshi, S. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1036 · Cited by 93 · see citing works
Self-citationA Joint Model for Question Answering and Question GenerationPreprint
Wang, T., Yuan, X., & Trischler, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01450 · Cited by 89 · see citing works
Creativity: Generating Diverse Questions using Variational AutoencodersPreprint
Jain, U., Zhang, Z., & Schwing, A. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03493 · Cited by 32 · see citing works
Saha, A., Pahuja, V., Khapra, M. M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10314 · Cited by 23 · see citing works
Self-citationAsking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate RewardsPreprint
Zhang, J., Wu, Q., Shen, C., Zhang, J., Lu, J., & van den Hengel, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07614 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
Self-citationHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
MCQGen: A Large Language Model-Driven MCQ Generator for Personalized Learning
Hang, C. N., Tan, C. W., & Yu, P.-D. (2024) · IEEE Access · doi:10.1109/access.2024.3420709 · Cited by 112 · see citing works
Reinforcement Learning Based Graph-to-Sequence Model for Natural Question GenerationPreprint
Chen, Y., Wu, L., & Zaki, M. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04942 · Cited by 127 · see citing works
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Zhang, S., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1253 · Cited by 115 · see citing works
Evaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105 · see citing works
Recent Advances in Neural Question GenerationPreprint
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 79 · see citing works
Current and future state of evaluation of large language models for medical summarization tasks
Croxford, E., Gao, Y., Pellegrino, N., Wong, K. K. Y., Wills, G. J., First, E., Liao, F., Goswami, C., Patterson, B. W., & Afshar, M. (2025) · npj Health Systems · doi:10.1038/s44401-024-00011-2 · Cited by 70 · see citing works
Lee, D. B., Lee, S., Jeong, W. T., Kim, D., & Hwang, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.20 · Cited by 64 · see citing works
Semantic Answer Similarity for Evaluating Question Answering Models
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.15 · Cited by 62 · see citing works
To Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 62 · see citing works
Let’s Ask Again: Refine Network for Automatic Question Generation
Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1326 · Cited by 57 · see citing works
Self-citationLearning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT WorldsPreprint
Klein, T., & Nabi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02365 · Cited by 58 · see citing works
Multi-hop Question Generation with Graph Convolutional Network
Su, D., Xu, Y., Dai, W., Ji, Z., Yu, T., & Fung, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.416 · Cited by 38 · see citing works
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 41 · see citing works
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.575 · Cited by 34 · see citing works
Self-citationIt is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Yao, B., Wang, D., Wu, T., Zhang, Z., Li, T., Yu, M., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.54 · Cited by 46 · see citing works
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 79 · see citing works
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016220 · Cited by 33 · see citing works
Self-citationControllable Open-ended Question Generation with A New Question Type Ontology
Cao, S., & Wang, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.502 · Cited by 35 · see citing works
MixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31 · see citing works
A Feasibility Study of Answer-Unaware Question Generation for Education
Dugan, L. D., Miltsakaki, E., Upadhyay, S., Ginsberg, E., Gonzalez, H., Choi, D., Yuan, C., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.151 · Cited by 26 · see citing works
Hwang, M.-H., Shin, J., Seo, H., Im, J.-S., Cho, H., & Lee, C.-K. (2023) · Applied Sciences · doi:10.3390/app13020903 · Cited by 25 · see citing works
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR
Opitz, J., & Frank, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.129 · Cited by 22 · see citing works
A question–answer generation system for an asynchronous distance learning platform
Wang, H., Maslim, M., & Kan, C.-H. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11675-y · Cited by 19 · see citing works
Evaluation metrics on text summarization: comprehensive survey
Davoodijam, E., & Meybodi, M. A. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02217-0 · Cited by 20 · see citing works
Gašpar, A., Grubišić, A., & Šarić‐Grgić, I. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09672-1 · Cited by 18 · see citing works
A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation Using GPT
Maity, S., Deroy, A., & Sarkar, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56063-7_18 · Cited by 20 · see citing works
Rule-Based Automatic Question Generation Using Semantic Role Labeling
Keklik, O., Tuğlular, T., & Tekır, S. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018edp7199 · Cited by 18 · see citing works
Ask to Learn: A Study on Curiosity-driven Question Generation
Scialom, T., & Staiano, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.202 · Cited by 22 · see citing works
Maity, S., Deroy, A., & Sarkar, S. (2024) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3734947.3734949 · Cited by 22 · see citing works
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 24 · see citing works
Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation
Laban, P., Wu, C.-S., Murakhovs’ka, L., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.9 · Cited by 17 · see citing works
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story GenerationPreprint
Chhun, C., Colombo, P., Clavel, C., & Suchanek, F. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11646 · Cited by 29 · see citing works
RNN based question answer generation and ranking for financial documents using financial NER
Jayakumar, H., Krishnakumar, M. S., Peddagopu, V. V. V., & Sridhar, R. (2020) · Sadhana · doi:10.1007/s12046-020-01501-3 · Cited by 15 · see citing works
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16 · see citing works
“For instance, Nema and Khapra (2018) generated monotonous questions using rule-based methods for evaluation and focused primarily on the answer-ability of questions, neglecting other dimensions.”
BackgroundMethodInfluentialMulti-hop Reasoning Question Generation and Its Application
Yu, J., Su, Q., Quan, X., & Yin, J. P. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3073227 · Cited by 13 · see citing works
KPQA: A Metric for Generative Question Answering Using Keyphrase Weights
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.170 · Cited by 14 · see citing works
Low-Resource Generation of Multi-hop Reasoning Questions
Yu, J., Liu, W., Qiu, S., Su, Q., Wang, K., Quan, X., & Yin, J. P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.601 · Cited by 13 · see citing works
Zeng, H., Zhi, Z., Liu, J., & Wei, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.026 · Cited by 14 · see citing works
QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Ji, T., Lyu, C., Jones, G., Zhou, L., & Graham, Y. (2022) · Entropy · doi:10.3390/e24111514 · Cited by 12 · see citing works
Liao, L., Mo, W., Wen, Y., Liu, S., Zou, Y., Mo, L., Chengke, W., & Fan, C. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6315 · Cited by 13 · see citing works
Automatic Multilingual Question Generation for Health Data Using LLMs
Ackerman, R., & Balyan, R. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7587-7_1 · Cited by 12 · see citing works
Varifocal Question Generation for Fact-checking
Ousidhoum, N. D., Yuan, Z., & Vlachos, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.163 · Cited by 12 · see citing works
Unanswerable Question Correction in Question Answering over Personal Knowledge Base
Yen, A.-Z., Huang, H., & Chen, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17678 · Cited by 12 · see citing works
Socratic Question Generation: A Novel Dataset, Models, and Evaluation
Ang, B. H., Gollapalli, S. D., & Ng, S.-K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.12 · Cited by 10 · see citing works
Srivastava, A., & Memon, A. M. (2024) · IEEE Access · doi:10.1109/access.2024.3446854 · Cited by 11 · see citing works
Improving Neural Question Generation using Deep Linguistic Representation
Yuan, W., He, T., & Dai, X. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449975 · Cited by 10 · see citing works
Graph augmented sequence-to-sequence model for neural question generation
Ma, H., Wang, J., Lin, H., & Xu, B. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04260-2 · Cited by 9 · see citing works
Xiao, Z., Zhang, S., Lai, V. W.-M., & Liao, Q. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.676 · Cited by 17 · see citing works
AnswerQuest: A System for Generating Question-Answer Items from Multi-Paragraph Documents
Roemmele, M., Sidhpura, D., DeNeefe, S., & Tsou, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.6 · Cited by 12 · see citing works
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages
Ananya Sai, B., Dixit, T., Nagarajan, V., Kunchukuttan, A., Kumar, P., Khapra, M. M., & Dabre, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.795 · Cited by 10 · see citing works
Self-citationAccelerating Real-Time Question Answering via Question GenerationPreprint
Fang, Y., Wang, S., Gan, Z., Sun, S., Liu, J., & Zhu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05167 · Cited by 10 · see citing works
Wang, X., Liu, B., Tang, S., & Wu, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.37 · Cited by 11 · see citing works
Addressing Semantic Drift in Question Generation for Semi-Supervised Question AnsweringPreprint
Zhang, S., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06356 · Cited by 14 · see citing works
Distractor Generation Through Text-to-Text Transformer Models
de-Fitero-Dominguez, D., García‐López, E., García‐Cabot, A., del-Hoyo-Gabaldon, J.-A., & Moreno-Cediel, A. (2024) · IEEE Access · doi:10.1109/access.2024.3361673 · Cited by 9 · see citing works
A Tutorial on Evaluation Metrics used in Natural Language Generation
Khapra, M. M., & Sai, A. B. (2021) · doi:10.18653/v1/2021.naacl-tutorials.4 · Cited by 7 · see citing works
Self-citationRe-evaluating ADEM: A Deeper Look at Scoring Dialogue ResponsesPreprint
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08832 · Cited by 7 · see citing works
Self-citationCQuAE: A new Contextualized QUestion Answering corpus on Education domain
Gerald, T., Tamames, L., Ettayeb, S., Le, H.-Q., Paroubek, P., & Vilnat, A. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102305 · Cited by 6 · see citing works
Language Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 6 · see citing works
Evaluating the Knowledge Dependency of Questions
Moon, H., Yang, Y., Yu, H., Lee, S., Jeong, M., Park, J., Shin, J., Kim, M., & Choi, S. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.718 · Cited by 6 · see citing works
Modelling Text Similarity: A Survey
Mu, W., & Lim, K. H. (2023) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3625007.3627305 · Cited by 6 · see citing works
Maity, S., Deroy, A., & Sarkar, S. (2024) · SN Computer Science · doi:10.1007/s42979-024-03464-2 · Cited by 7 · see citing works
Kalpakchi, D., & Boye, J. (2021) · Natural Language Engineering · doi:10.1017/s1351324923000037 · Cited by 8 · see citing works
Towards Vietnamese Question and Answer Generation: An Empirical Study
Hùng, P. Q., Le, H.-L., Nhat, M. D., Khang Tran, T., Tran-Tien, M., Dang, V.-H., Vu, H.-T., Nguyen, M.-T., & Phan, X.-H. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3675781 · Cited by 4 · see citing works
Shen, K., Wu, L., Tang, S., Xu, F., Long, B., Zhuang, Y., & Pei, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3425222 · Cited by 4 · see citing works
Adaptive Cross-Lingual Question Generation with Minimal Resources
Yu, J., Wang, S., & Yin, J. P. (2021) · The Computer Journal · doi:10.1093/comjnl/bxab106 · Cited by 5 · see citing works
Sewunetie, W. T., & Kovács, L. (2022) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v28.i3.pp1738-1748 · Cited by 4 · see citing works
Zhang, C., Zhang, H., Sun, Y., & Wang, J. (2022) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3558100.3563846 · Cited by 6 · see citing works
LLM-Based Quiz Generation for Assessments in Learning Management System
Shirkar, S., Langhi, D., & Agrawal, S. (2025) · International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS) · doi:10.1109/icc-robins64345.2025.11086273 · Cited by 3 · see citing works
Feeding What You Need by Understanding What You Learned
Wang, X., Liu, B., Xu, F., Long, B., Tang, S., & Wu, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.403 · Cited by 3 · see citing works
“Questomatic”: Automated Question Formulation System
Jadhav, P., Deore, S., Aute, S., Gaikwad, R., Barphe, S. S., & Meshram, E. (2024) · International Conference on Sustainable Expert Systems (ICSES) · doi:10.1109/icses63445.2024.10763124 · Cited by 3 · see citing works
QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing
Wu, Y., Mangla, R., Durrett, G., & Li, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.325 · Cited by 3 · see citing works
The Eighth Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI’24)
Frummet, A., Papenmeier, A., Fröbe, M., & Kiesel, J. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638310 · Cited by 3 · see citing works
A Model-Based Evaluation Metric for Question Answering Systems
Bakır, D., Aktaş, M. S., & Yildiz, B. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500032 · Cited by 3 · see citing works
Generating Consistent and Diverse QA pairs from Contexts with BN Conditional VAE
Li, J., Qi, P., & Luo, H. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776055 · Cited by 3 · see citing works
On Generating Fact-Infused Question Variations
Deschamps, A., Gollapalli, S. D., & Ng, S.-K. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_039 · Cited by 3 · see citing works
Automatic Question Generation: A Comparative Analysis of Rule-Based and Neural Network-Based Models
Velmurugan, H., Pandiarajan, N., Ravi, L., Voleti, S., Sheik, R., & Nirmala, S. J. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-1724-8_21 · Cited by 2 · see citing works
DORB: Dynamically Optimizing Multiple Rewards with Bandits
Pasunuru, R., Guo, H., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.625 · Cited by 2 · see citing works
Studying the Role of Named Entities for Content Preservation in Text Style Transfer
Babakov, N., Dale, D., Logacheva, V., Krotova, I., & Panchenko, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_40 · Cited by 2 · see citing works
Ren, Z., Guo, J., Xu, Y., & Xiao, B. J. (2023) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc59918.2023.10390776 · Cited by 2 · see citing works
Controllable Open-ended Question Generation with A New Question Type OntologyPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00152 · Cited by 2 · see citing works
Talk2Data: High-Level Question Decomposition for Data-Oriented Question and AnsweringPreprint
Shi, D., Guo, Y., Guo, M., Wu, Y., Chen, Q., & Cao, N. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Phalak, A., Yevale, S., Muley, G., & Nerurkar, A. K. (2023) · Advances in intelligent systems and computing · doi:10.1007/978-981-99-0550-8_19 · Cited by 1 · see citing works
W iki H int : A Human-Annotated Dataset for Hint Ranking and Generation
Mozafari, J., Gerhold, F., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730284 · Cited by 3 · see citing works
A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering
Maufe, M., Ravenscroft, J., Procter, R., & Liakata, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.9 · Cited by 1 · see citing works
Towards Generating Contextual and Empathetic Response for Covid-related Queries
Rasipuram, S., Maitra, A., Shaw, B., & Saha, S. (2022) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc55712.2022.9910984 · Cited by 1 · see citing works
Rule-Based Automatic Question Generationusing Dependency ParsingPreprint
Sewunetie, W. T., & Kovács, L. (2023) · Research Square · doi:10.21203/rs.3.rs-2693893/v1 · Cited by 1 · see citing works
KPQA: A Metric for Generative Question Answering Using Word Weights.Preprint
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2020) · arXiv (Cornell University)
DORB: Dynamically Optimizing Multiple Rewards with BanditsPreprint
Pasunuru, R., Guo, H., & Bansal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07635
How Well Do You Know Your Audience? Reader-aware Question Generation.Preprint
Stewart, I., & Mihalcea, R. F. (2021) · arXiv (Cornell University)
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274
ConvRGX: Recognition, Generation, and Extraction for Self-trained Conversational Question Answering
Zhang, T., Tang, L., Fang, W., Luo, H., Wu, X., Meng, H. M. L., & Glass, J. R. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.10
First Question Generation Method via Summarization and Syntax for English Reading Tests
张, 懿. (2024) · Computer Science and Application · doi:10.12677/csa.2024.148178
Generating Vietnamese Questions and Answers for Subject Marxist-Leninist Philosophy
Nam, D. T., Hieu, N. V., Hung, N. M., & Hung, P. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06952-8_6
Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation
Ishikawa, R., Fujii, R., Saitô, H., & Hachiuma, R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00729
- Authors
- Nema, P., & Khapra, M. M.
- Affiliations
- Indian Institute of Technology Madras (IN)
- ORCID
- Khapra, Mitesh M.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 3950-3959
- OpenAlex
- W2888812214
- Cited by
- 106
- References
- 43
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 18 · 2023: 18 · 2022: 19 · 2021: 18 · 2020: 11 · 2019: 12
- FWCI
- 4.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- automatic question generation, evaluation metrics, n-gram similarity, BLEU, NIST, answerability, scoring functions, human judgment correlation, natural language generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
