The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Towards a Better Metric for Evaluating Question Generation Systems
Open accessAbstract
There has always been criticism for using ngram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems.However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc.Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task.In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc.In this work, we show that current automatic evaluation metrics based on n-gram similarity do not always correlate well with human judgments about answerability of a question.To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments.The scripts and data developed as a part of this work are made publicly available.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“1) focusing only on specific dimensions like answerability;”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Nema and Khapra, 2018
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Automatic evaluation of QG is still dominated by reference-based metrics such as BLEU (Papineni et al., 2002) and Q-Metric”
From QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024) · cites this paper as Nema and Khapra, 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Q-BLEU is a variant of BLEU with a higher weight for matching entities and has higher correlation with human judgment for question generation than standard metrics.”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [42]
In the citing paper
“One notable contribution is made in [105], where the authors first use the answerability of a question through human evaluation to modify the existing automatic evaluation metrics to include the influence of relevant words, question types, function words, and named entities.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [105]
In the citing paper
“They further proved that their proposed metrics had a better correlation with the human evaluation scores.”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [105]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
(2004) · Cited by 9,718
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,323
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
NLTK: The Natural Language ToolkitPreprint
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,893
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Research works printed but not linked: 3
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In International Conference on Computer Vision (ICCV).
- Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason We- ston. 2016. Key-value memory networks for directly reading documents. In EMNLP. The Association for Computational Linguistics, pages 1400-1409.
- Ananthakrishnan R, Pushpak Bhattacharyya, M Sasikumar, and Ritesh M Shah. 2007. Some issues in automatic evaluation of english-hindi mt: More blues for bleu. In International Conference on Natural Language Processing.
Cited by
Sorted by citations
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
Self-citationHandling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132
MCQGen: A Large Language Model-Driven MCQ Generator for Personalized Learning
Hang, C. N., Tan, C. W., & Yu, P.-D. (2024) · IEEE Access · doi:10.1109/access.2024.3420709 · Cited by 130
Reinforcement Learning Based Graph-to-Sequence Model for Natural Question GenerationPreprint
Chen, Y., Wu, L., & Zaki, M. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04942 · Cited by 135
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Zhang, S., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1253 · Cited by 121
Evaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105
Recent Advances in Neural Question GenerationPreprint
Pan, L., Lei, W., Chua, T., & Kan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.08949 · Cited by 124
Current and future state of evaluation of large language models for medical summarization tasks
Croxford, E., Gao, Y., Pellegrino, N., Wong, K. K. Y., Wills, G. J., First, E., Liao, F., Goswami, C., Patterson, B. W., & Afshar, M. (2025) · npj Health Systems · doi:10.1038/s44401-024-00011-2 · Cited by 81
Lee, D. B., Lee, S., Jeong, W. T., Kim, D., & Hwang, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.20 · Cited by 69
Semantic Answer Similarity for Evaluating Question Answering Models
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.15 · Cited by 66
To Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 68
Let’s Ask Again: Refine Network for Automatic Question Generation
Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1326 · Cited by 60
Self-citationLearning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT WorldsPreprint
Klein, T., & Nabi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02365 · Cited by 68
Multi-hop Question Generation with Graph Convolutional Network
Su, D., Xu, Y., Dai, W., Ji, Z., Yu, T., & Fung, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.416 · Cited by 38
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 41
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.575 · Cited by 39
Self-citationIt is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Yao, B., Wang, D., Wu, T., Zhang, Z., Li, T., Yu, M., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.54 · Cited by 51
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 119
Re-Evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016220 · Cited by 33
Self-citationControllable Open-ended Question Generation with A New Question Type Ontology
Cao, S., & Wang, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.502 · Cited by 42
MixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31
A Feasibility Study of Answer-Unaware Question Generation for Education
Dugan, L. D., Miltsakaki, E., Upadhyay, S., Ginsberg, E., Gonzalez, H., Choi, D., Yuan, C., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.151 · Cited by 27
Hwang, M.-H., Shin, J., Seo, H., Im, J.-S., Cho, H., & Lee, C.-K. (2023) · Applied Sciences · doi:10.3390/app13020903 · Cited by 28
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR
Opitz, J., & Frank, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.129 · Cited by 28
A question–answer generation system for an asynchronous distance learning platform
Wang, H., Maslim, M., & Kan, C.-H. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11675-y · Cited by 22
Evaluation metrics on text summarization: comprehensive survey
Davoodijam, E., & Meybodi, M. A. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02217-0 · Cited by 23
Gašpar, A., Grubišić, A., & Šarić‐Grgić, I. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09672-1 · Cited by 18
A Novel Multi-Stage Prompting Approach for Language Agnostic MCQ Generation Using GPT
Maity, S., Deroy, A., & Sarkar, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56063-7_18 · Cited by 26
Rule-Based Automatic Question Generation Using Semantic Role Labeling
Keklik, O., Tuğlular, T., & Tekır, S. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018edp7199 · Cited by 18
Ask to Learn: A Study on Curiosity-driven Question Generation
Scialom, T., & Staiano, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.202 · Cited by 24
Maity, S., Deroy, A., & Sarkar, S. (2024) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3734947.3734949 · Cited by 24
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 28
Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation
Laban, P., Wu, C.-S., Murakhovs’ka, L., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.9 · Cited by 20
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story GenerationPreprint
Chhun, C., Colombo, P., Clavel, C., & Suchanek, F. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11646 · Cited by 38
RNN based question answer generation and ranking for financial documents using financial NER
Jayakumar, H., Krishnakumar, M. S., Peddagopu, V. V. V., & Sridhar, R. (2020) · Sadhana · doi:10.1007/s12046-020-01501-3 · Cited by 15
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16
“For instance, Nema and Khapra (2018) generated monotonous questions using rule-based methods for evaluation and focused primarily on the answer-ability of questions, neglecting other dimensions.”
BackgroundMethodInfluentialMulti-hop Reasoning Question Generation and Its Application
Yu, J., Su, Q., Quan, X., & Yin, J. P. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3073227 · Cited by 14
KPQA: A Metric for Generative Question Answering Using Keyphrase Weights
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.170 · Cited by 18
Low-Resource Generation of Multi-hop Reasoning Questions
Yu, J., Liu, W., Qiu, S., Su, Q., Wang, K., Quan, X., & Yin, J. P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.601 · Cited by 13
Zeng, H., Zhi, Z., Liu, J., & Wei, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.026 · Cited by 14
QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Ji, T., Lyu, C., Jones, G., Zhou, L., & Graham, Y. (2022) · Entropy · doi:10.3390/e24111514 · Cited by 12
Liao, L., Mo, W., Wen, Y., Liu, S., Zou, Y., Mo, L., Chengke, W., & Fan, C. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6315 · Cited by 13
Automatic Multilingual Question Generation for Health Data Using LLMs
Ackerman, R., & Balyan, R. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7587-7_1 · Cited by 13
Varifocal Question Generation for Fact-checking
Ousidhoum, N. D., Yuan, Z., & Vlachos, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.163 · Cited by 14
Unanswerable Question Correction in Question Answering over Personal Knowledge Base
Yen, A.-Z., Huang, H., & Chen, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17678 · Cited by 12
Socratic Question Generation: A Novel Dataset, Models, and Evaluation
Ang, B. H., Gollapalli, S. D., & Ng, S.-K. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.12 · Cited by 13
Srivastava, A., & Memon, A. M. (2024) · IEEE Access · doi:10.1109/access.2024.3446854 · Cited by 14
Improving Neural Question Generation using Deep Linguistic Representation
Yuan, W., He, T., & Dai, X. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449975 · Cited by 10
Graph augmented sequence-to-sequence model for neural question generation
Ma, H., Wang, J., Lin, H., & Xu, B. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04260-2 · Cited by 9
Xiao, Z., Zhang, S., Lai, V. W.-M., & Liao, Q. V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.676 · Cited by 44
AnswerQuest: A System for Generating Question-Answer Items from Multi-Paragraph Documents
Roemmele, M., Sidhpura, D., DeNeefe, S., & Tsou, L. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.6 · Cited by 12
IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for Indian Languages
Ananya Sai, B., Dixit, T., Nagarajan, V., Kunchukuttan, A., Kumar, P., Khapra, M. M., & Dabre, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.795 · Cited by 15
Self-citationAccelerating Real-Time Question Answering via Question GenerationPreprint
Fang, Y., Wang, S., Gan, Z., Sun, S., Liu, J., & Zhu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05167 · Cited by 17
Wang, X., Liu, B., Tang, S., & Wu, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.37 · Cited by 12
Addressing Semantic Drift in Question Generation for Semi-Supervised Question AnsweringPreprint
Zhang, S., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06356 · Cited by 19
Distractor Generation Through Text-to-Text Transformer Models
de-Fitero-Dominguez, D., García‐López, E., García‐Cabot, A., del-Hoyo-Gabaldon, J.-A., & Moreno-Cediel, A. (2024) · IEEE Access · doi:10.1109/access.2024.3361673 · Cited by 11
A Tutorial on Evaluation Metrics used in Natural Language Generation
Khapra, M. M., & Sai, A. B. (2021) · doi:10.18653/v1/2021.naacl-tutorials.4 · Cited by 7
Self-citationRe-evaluating ADEM: A Deeper Look at Scoring Dialogue ResponsesPreprint
Sai, A. B., Gupta, M. D., Khapra, M. M., & Srinivasan, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08832 · Cited by 7
Self-citationCQuAE: A new Contextualized QUestion Answering corpus on Education domain
Gerald, T., Tamames, L., Ettayeb, S., Le, H.-Q., Paroubek, P., & Vilnat, A. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102305 · Cited by 6
Language Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 10
Evaluating the Knowledge Dependency of Questions
Moon, H., Yang, Y., Yu, H., Lee, S., Jeong, M., Park, J., Shin, J., Kim, M., & Choi, S. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.718 · Cited by 8
Modelling Text Similarity: A Survey
Mu, W., & Lim, K. H. (2023) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3625007.3627305 · Cited by 8
Maity, S., Deroy, A., & Sarkar, S. (2024) · SN Computer Science · doi:10.1007/s42979-024-03464-2 · Cited by 7
Kalpakchi, D., & Boye, J. (2023) · Natural Language Engineering · doi:10.1017/s1351324923000037 · Cited by 7
Towards Vietnamese Question and Answer Generation: An Empirical Study
Hùng, P. Q., Le, H.-L., Nhat, M. D., Khang Tran, T., Tran-Tien, M., Dang, V.-H., Vu, H.-T., Nguyen, M.-T., & Phan, X.-H. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3675781 · Cited by 4
Shen, K., Wu, L., Tang, S., Xu, F., Long, B., Zhuang, Y., & Pei, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3425222 · Cited by 4
Adaptive Cross-Lingual Question Generation with Minimal Resources
Yu, J., Wang, S., & Yin, J. P. (2021) · The Computer Journal · doi:10.1093/comjnl/bxab106 · Cited by 5
Sewunetie, W. T., & Kovács, L. (2022) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v28.i3.pp1738-1748 · Cited by 4
Zhang, C., Zhang, H., Sun, Y., & Wang, J. (2022) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3558100.3563846 · Cited by 6
LLM-Based Quiz Generation for Assessments in Learning Management System
Shirkar, S., Langhi, D., & Agrawal, S. (2025) · International Conference on Cognitive Robotics and Intelligent Systems (ICC-ROBINS) · doi:10.1109/icc-robins64345.2025.11086273 · Cited by 3
Feeding What You Need by Understanding What You Learned
Wang, X., Liu, B., Xu, F., Long, B., Tang, S., & Wu, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.403 · Cited by 3
“Questomatic”: Automated Question Formulation System
Jadhav, P., Deore, S., Aute, S., Gaikwad, R., Barphe, S. S., & Meshram, E. (2024) · International Conference on Sustainable Expert Systems (ICSES) · doi:10.1109/icses63445.2024.10763124 · Cited by 3
QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing
Wu, Y., Mangla, R., Durrett, G., & Li, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.325 · Cited by 7
The Eighth Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI’24)
Frummet, A., Papenmeier, A., Fröbe, M., & Kiesel, J. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638310 · Cited by 3
A Model-Based Evaluation Metric for Question Answering Systems
Bakır, D., Aktaş, M. S., & Yildiz, B. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500032 · Cited by 3
Generating Consistent and Diverse QA pairs from Contexts with BN Conditional VAE
Li, J., Qi, P., & Luo, H. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776055 · Cited by 3
On Generating Fact-Infused Question Variations
Deschamps, A., Gollapalli, S. D., & Ng, S.-K. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_039 · Cited by 3
Automatic Question Generation: A Comparative Analysis of Rule-Based and Neural Network-Based Models
Velmurugan, H., Pandiarajan, N., Ravi, L., Voleti, S., Sheik, R., & Nirmala, S. J. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-1724-8_21 · Cited by 2
DORB: Dynamically Optimizing Multiple Rewards with Bandits
Pasunuru, R., Guo, H., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.625 · Cited by 2
Studying the Role of Named Entities for Content Preservation in Text Style Transfer
Babakov, N., Dale, D., Logacheva, V., Krotova, I., & Panchenko, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_40 · Cited by 2
Ren, Z., Guo, J., Xu, Y., & Xiao, B. J. (2023) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc59918.2023.10390776 · Cited by 2
Controllable Open-ended Question Generation with A New Question Type OntologyPreprint
Cao, S., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00152 · Cited by 2
Talk2Data: High-Level Question Decomposition for Data-Oriented Question and AnsweringPreprint
Shi, D., Guo, Y., Guo, M., Wu, Y., Chen, Q., & Cao, N. (2021) · arXiv (Cornell University) · Cited by 2
Phalak, A., Yevale, S., Muley, G., & Nerurkar, A. K. (2023) · Advances in intelligent systems and computing · doi:10.1007/978-981-99-0550-8_19 · Cited by 1
W iki H int : A Human-Annotated Dataset for Hint Ranking and Generation
Mozafari, J., Gerhold, F., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730284 · Cited by 6
A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering
Maufe, M., Ravenscroft, J., Procter, R., & Liakata, M. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.9 · Cited by 1
Towards Generating Contextual and Empathetic Response for Covid-related Queries
Rasipuram, S., Maitra, A., Shaw, B., & Saha, S. (2022) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc55712.2022.9910984 · Cited by 1
Rule-Based Automatic Question Generationusing Dependency ParsingPreprint
Sewunetie, W. T., & Kovács, L. (2023) · Research Square · doi:10.21203/rs.3.rs-2693893/v1 · Cited by 1
KPQA: A Metric for Generative Question Answering Using Word Weights.Preprint
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., Shin, J., & Jung, K. (2020) · arXiv (Cornell University)
DORB: Dynamically Optimizing Multiple Rewards with BanditsPreprint
Pasunuru, R., Guo, H., & Bansal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07635
How Well Do You Know Your Audience? Reader-aware Question Generation.Preprint
Stewart, I., & Mihalcea, R. F. (2021) · arXiv (Cornell University)
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
Murugesan, K., Swaminathan, S., Dan, S., Chaudhury, S., Gunasekara, C., Crouse, M., Mahajan, D., Abdelaziz, I., Fokoue, A., Kapanipathi, P., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.274 · Cited by 1
ConvRGX: Recognition, Generation, and Extraction for Self-trained Conversational Question Answering
Zhang, T., Tang, L., Fang, W., Luo, H., Wu, X., Meng, H. M. L., & Glass, J. R. (2023) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2023.dialdoc-1.10
First Question Generation Method via Summarization and Syntax for English Reading Tests
张, 懿. (2024) · Computer Science and Application · doi:10.12677/csa.2024.148178
Generating Vietnamese Questions and Answers for Subject Marxist-Leninist Philosophy
Nam, D. T., Hieu, N. V., Hung, N. M., & Hung, P. D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06952-8_6
Metadata
- Authors
- Nema, P., & Khapra, M. M.
- Affiliations
- Indian Institute of Technology Madras (IN)
- ORCID
- Khapra, Mitesh M.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 3950-3959
- OpenAlex
- W2888812214
- Cited by
- 106
- References
- 43
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 18 · 2023: 18 · 2022: 19 · 2021: 18 · 2020: 11 · 2019: 12
- FWCI
- 4.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- automatic question generation, evaluation metrics, n-gram similarity, BLEU, NIST, answerability, scoring functions, human judgment correlation, natural language generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
