Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Open accessAbstract
The predictions of question answering (QA) systems are typically evaluated against manually annotated finite sets of one or more answers.This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with predefined rules or with the token-level F 1 measure.In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures.To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and publish over 23k human judgments for candidates produced by multiple QA systems on SQuAD. 1 Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F 1 measure, such as a false impression of graduality, or missing dependence on the question.Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task.Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F 1 , and to more accurately reflect the performance of systems.Finally, we demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to ×2.6.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In conversational memory settings, the assumption that correctness can be determined by surface-level overlap does not hold, because multiple valid responses may satisfy the same memory constraint while receiving different scores.
“In conversational memory settings, this assumption does not hold. Multiple valid responses may satisfy the same memory constraint while receiving different scores.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Bulian et al., 2022
Section 2.1 and Section 3.6 show that token-overlap scores can undervalue answers humans judge equivalent, supporting the general point that valid answers can receive different scores. But the Limitations section explicitly leaves conversational question answering for future work; the paper does not establish this finding for conversational memory constraints. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLoCoMo relies on string-matching-based metrics to compare model outputs with a reference answer, assuming that correctness can be determined by surface-level overlap.
“On the output side, LoCoMo relies on stringmatching-based metrics to compare model outputs with a reference answer. These metrics assume that correctness can be determined by surface-level overlap.”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Bulian et al., 2022
The Introduction defines Exact Match and Token F1 as comparisons between a candidate answer and reference answers, and Section 2.1 explains why surface overlap can misrepresent correctness. The paper does not discuss LoCoMo or establish which metrics LoCoMo uses, so it supports the general critique but not the LoCoMo-specific assertion. (LLM, conf=0.88)
Experimental — accuracy not yet measuredGeneration-based metrics such as BLEU, ROUGE, exact match, and token-level F1 poorly reflect semantic correctness and behavioral validity in open-ended generation.
“these metrics are known to poorly reflect semantic correctness and behavioral validity in open-ended generation”
From Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Bulian et al., 2022
Section 2.1 says exact match and token-level F1 can misjudge whether a QA answer is correct. Section 6 notes a need to go beyond lexical overlap when adapting BLEU and ROUGE for answer comparison. The paper does not establish the broader claim about all four metrics reflecting behavioral validity in open-ended generation; its analysis focuses on question-answer equivalence, chiefly for short answers. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMultiple valid responses may satisfy the same memory constraint while receiving different scores.
“Multiple valid responses may satisfy the same memory constraint while receiving different scores.”
From LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents (2026) · cites this paper as Bulian et al., 2022
Section 2.1 shows that a valid answer can receive an EM and F1 score of zero because of a minor wording difference, and Section 3.6 describes answers conveying the same core content despite low F1 scores. This supports the point that valid responses can receive different scores, but the paper evaluates answers to questions, not responses satisfying a memory constraint. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
(2004) · Cited by 9,718 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Pay Enough or Don't Pay at All*
Gneezy, U., & Rustichini, A. (2000) · Quarterly Journal of Economics · doi:10.1162/003355300554917 · Cited by 2,175 · see citing works
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695 · see citing works
A comparison of string distance metrics for name-matching tasks
Cohen, W. W., Ravikumar, P., & Fienberg, S. E. (2003) · Cited by 1,371 · see citing works
Algorithmic Learning in a Random World
(2005) · doi:10.1007/b106715 · Cited by 1,119 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
COMET: A Neural Framework for MT Evaluation
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.213 · Cited by 767 · see citing works
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · see citing works
A tutorial on conformal prediction
Shafer, G., & Vovk, V. (2007) · arXiv · doi:10.48550/arxiv.0706.3188 · Cited by 655 · see citing works
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
Yamada, I., Asai, A., Shindo, H., Takeda, H., & Matsumoto, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.523 · Cited by 647 · see citing works
Building a question answering test collection
Voorhees, E. M., & Tice, D. M. (2000) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/345508.345577 · Cited by 437 · see citing works
A gentle introduction to conformal prediction and distribution-free uncertainty quantification
Angelopoulos, A. N., & Bates, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07511 · Cited by 240 · see citing works
AmbigQA: Answering Ambiguous Open-domain Questions
Min, S., Michael, J., Hajishirzi, H., & Zettlemoyer, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.466 · Cited by 219 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Distribution-free, Risk-controlling Prediction Sets
Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021) · Journal of the ACM · doi:10.1145/3478535 · Cited by 144 · see citing works
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
Clark, E., Çelikyılmaz, A., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1264 · Cited by 138 · see citing works
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126 · see citing works
Self-citationQAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · cited 1× · see citing works
“Thanks to its simplicity, BEM can easily replace F 1 in many other applications that leverage QA predictions (e.g., Honovich et al., 2021, 2022; Eyal et al., 2019; Fabbri et al., 2021; Schuster et al., 2021a).”
BackgroundEvaluating Question Answering Evaluation
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5817 · Cited by 105 · see citing works
Question Answering as an Automatic Evaluation Metric for News Article Summarization
Eyal, M., Baumel, T., & Elhadad, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1395 · Cited by 92 · see citing works
Confident adaptive language modeling
Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., & Metzler, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1269 · Cited by 91 · see citing works
Self-citationTRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.287 · Cited by 75 · see citing works
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons LearnedPreprint
Min, S., Boyd-Graber, J. L., Alberti, C., Chen, D., Choi, E., Collins, M., Guu, K., Hajishirzi, H., Lee, K., Palomaki, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00133 · Cited by 63 · see citing works
Semantic Answer Similarity for Evaluating Question Answering Models
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.15 · Cited by 62 · see citing works
Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task
An, Y., Liu, K., Liu, J., Lyu, Y., & Li, S. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2611 · Cited by 56 · see citing works
Can NLI Models Verify QA Systems’ Predictions?
Chen, J., Choi, E., & Durrett, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.324 · Cited by 51 · see citing works
Consistent Accelerated Inference via Confident Adaptive Transformers
Schuster, T., Fisch, A., Jaakkola, T., & Barzilay, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.406 · Cited by 34 · see citing works
Self-citationMOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics
Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.528 · Cited by 33 · see citing works
Sellam, T., Pu, A., Chung, H. W., Gehrmann, S., Tan, Q., Freitag, M., Das, D., & Parikh, A. P. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.102 · Cited by 27 · see citing works
Elazar, Y., & Goldberg, Y. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00280 · Cited by 20 · see citing works
What’s in a Name? Answer Equivalence For Open-Domain Question Answering
Si, C., Zhao, C., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.757 · Cited by 17 · see citing works
Efficient Conformal Prediction via Cascaded Inference with Expanded AdmissionPreprint
Fisch, A., Schuster, T., Jaakkola, T., & Barzilay, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03114 · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Ar- bor, Michigan. Association for Computational Lin- guistics.
- Eric Breck, John Burger, Lisa Ferro, David House, Marc Light, and Inderjeet Mani. 1999. A sys called qanda. In Proceedings of The Eighth Text REtrieval Conference.
- Adam Fisch, Robin Jia, and Tal Schuster. 2022. Uncer- tainty estimation for natural language processing. In COLING.
- Adam Fisch, Tal Schuster, Tommi S. Jaakkola, and Regina Barzilay. 2021. Efficient conformal predic- tion via cascaded inference with expanded admis- sion. In International Conference on Learning Rep- resentations.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Cited by
Sorted by citations
Evaluating Open-Domain Question Answering in the Era of Large Language Models
Kamalloo, E., Dziri, N., Clarke, C. L. A., & Rafiei, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.307 · Cited by 144 · see citing works
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74 · see citing works
Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H. T., & Shi, G. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00147 · Cited by 72 · see citing works
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60 · see citing works
Chen, Z., Chen, H., Imani, M., Chen, R., & Imani, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125769 · Cited by 55 · see citing works
HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data
Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., & Zhuang, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01230 · Cited by 59 · see citing works
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y., Stanovsky, G., & Schwartz, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00247 · Cited by 37 · see citing works
Hybrid Retrieval-Augmented Generation Approach for LLMs Query Response Enhancement
Omrani, P., Hosseini, A., Hooshanfar, K., Ebrahimian, Z., Toosi, R., & Akhaee, M. A. (2024) · International Conference on Web Research (ICWR) · doi:10.1109/icwr61162.2024.10533345 · Cited by 44 · see citing works
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Mensink, T., Uijlings, J., Castrejón, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araújo, A. D. M., & Ferrari, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00289 · Cited by 35 · see citing works
Temporal Blind Spots in Large Language Models
Wallat, J., Jatowt, A., & Anand, A. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635818 · Cited by 33 · see citing works
Benchmark Evaluations, Applications, and Challenges of Large Vision Language Models: A SurveyPreprint
Li, Z., Wu, X., Du, H., Nghiem, H. T., & Shi, G. (2025) · Qeios · doi:10.32388/gxr68q · Cited by 30 · see citing works
Selectively Answering Ambiguous Questions
Cole, J., Zhang, M., Gillick, D., Eisenschlos, J., Dhingra, B., & Eisenstein, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.35 · Cited by 19 · see citing works
MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance
Xu, J., Wei, T., Hou, B., Orzechowski, P., Yang, S., Jin, R., Paulbeck, R., Wagenaar, J., Demiris, G., & Shen, L. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737393 · Cited by 21 · see citing works
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Schuster, T., Chen, S., Buthpitiya, S., Fabrikant, A., & Metzler, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.28 · Cited by 17 · see citing works
Self-citationTowards Robust QA Evaluation via Open LLMs
Kamalloo, E., Upadhyay, S., & Lin, J. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657675 · Cited by 15 · see citing works
Cross-Modal Retrieval for Knowledge-Based Visual Question Answering
Lerner, P., Ferret, O., & Guinaudeau, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_26 · Cited by 17 · see citing works
Conversational Gold: Evaluating Personalized Conversational Search System Using Gold Nuggets
Abbasiantaeb, Z., Lupart, S., Azzopardi, L. A., Dalton, J., & Aliannejadi, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730316 · Cited by 15 · see citing works
Navigating the Landscape of Hint Generation Research: From the Past to the Future
Jangra, A., Mozafari, J., Jatowt, A., & Muresan, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00751 · Cited by 10 · see citing works
Learning Answer Generation using Supervision from Automatic Question Answering Evaluators
Gabburo, M., Garg, S., Koncel-Kedziorski, R., & Moschitti, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.467 · Cited by 6 · see citing works
Continually Improving Extractive QA via Human Feedback
Gao, G., Chen, H., Artzi, Y., & Choi, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.27 · Cited by 5 · see citing works
Evaluation of Semantic Answer Similarity Metrics
Mustafazade, F., & Ebbinghaus, P. F. (2022) · International Conference on Machine Learning & Applications (CMLA) · doi:10.5121/csit.2022.121109 · Cited by 4 · see citing works
Assessing Distractors in Multiple-Choice Tests
Raina, V., Liusie, A., & Gales, M. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.2 · Cited by 4 · see citing works
Detrimental Contexts in Open-Domain Question Answering
Oh, P., & Thorne, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.776 · Cited by 4 · see citing works
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction
Milbauer, J., Louis, A. P., Hosseini, M. J., Fabrikant, A., Metzler, D., & Schuster, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.571 · Cited by 4 · see citing works
Self-citationDiscord Questions: A Computational Approach To Diversity Analysis in News Coverage
Laban, P., Wu, C.-S., Murakhovs’ka, L., Chen, X., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.380 · Cited by 5 · see citing works
Entity-Aware Cross-Modal Pretraining for Knowledge-Based Visual Question Answering
Adjali, O., Ferret, O., Ghannay, S., & Borgne, H. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_38 · Cited by 1 · see citing works
W iki H int : A Human-Annotated Dataset for Hint Ranking and Generation
Mozafari, J., Gerhold, F., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730284 · Cited by 3 · see citing works
Mozafari, J., Abdallah, A., Piryani, B., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730299 · Cited by 1 · see citing works
Exploring question answering: metric analysis and evaluation framework for enhanced interpretability
Navarro, L. C., Mucciaccia, S. S., Mutz, F. W., Paixão, T. M., Badue, C. S., Souza, A. F. D., & Oliveira-Santos, T. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12053-8 · Cited by 1 · see citing works
Evaluation of Semantic Answer Similarity Metrics
Mustafazade, F., & Ebbinghaus, P. F. (2022) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2022.11305 · Cited by 2 · see citing works
SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References
Gabburo, M., Garg, S., Koncel-Kedziorski, R., & Moschitti, A. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.3 · Cited by 1 · see citing works
Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1150
LoCoMo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
Li, Y., Guo, W., Zhang, L., Xu, R., Huang, M., Liu, H., Xu, L., Xu, Y., & Liu, J. (2026) · arXiv · doi:10.48550/arxiv.2602.10715
DiffQG: Generating Questions to Summarize Factual Changes
Cole, J. R., Jain, P., Eisenschlos, J. M., Zhang, M. J. Q., Choi, E., & Dhingra, B. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.225
Predicting Uncertainty of Generative LLMs with MARS: Meaning-Aware Response Scoring
Bakman, Y. F., Yaldiz, D. N., Buyukates, B., Avestimehr, S., Tao, C., & Dimitriadis, D. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619136
Ren, T., Zhang, Z., Wang, H., & Rafferty, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730047
Retrieving and Reading Multimodal Documents for Knowledge-Based VQA
Xie, W., Zhang, X., Zhang, M., & Zhang, M. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3349-7_7
Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair
Han, W., Kwak, Y., Yu, M., Kim, K., Lee, Y., Moon, H., & Paek, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773167
Describe Anything Model for Visual Question Answering on Text-Rich Images
Vu, Y.-L., Duong, D.-T., Duong, T.-B., Nguyen, A.-K., Nguyen, T.-H., Nguyen, L. T., Xing, J., Li, X. S., Wang, T., Bağcı, U., et al. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00776
MG$$^2$$-RAG: Multi-granularity Graph for Multimodal Retrieval-Augmented Generation
Dai, S., Huang, Q., You, X., & Yu, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37167-6_32
MMAgent-R$$^2$$: Learning to Rerank and Reject for Agentic mRAG
Zhang, T., Zhang, Z., Ma, Z., Yang, Y., Li, B., Yuan, C., Rong, K., Rao, F., Lyu, J., & Hu, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37447-9_11
Abbasiantaeb, Z., Lupart, S., Gohsen, M., Mirzakhmedova, N., Kiesel, J., Dalton, J., & Aliannejadi, M. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808591
Kim, S.-J., Kim, H. R., Kim, S., Kim, K., & Ko, B. C. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01753-5
Yılmaz, M. B. (2026) · Research Square · doi:10.21203/rs.3.rs-11074536/v1
MapQA: A Map-Question-Answering Benchmark for Visual Language Model Reasoning
Arnold, C., Alini, A., Alabdulkareem, A., Wang, J., Feenstra, P., Arnold, C., DeWitt, J. K., Ritsema, N. C., Yae, J. H., Katz, B., et al. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831218
Kemelbek, M. N., Nurbekova, Z. K., & Sagimbayeva, A. (2026) · Information · doi:10.3390/info17101014
- Authors
- Bulian, J., Buck, C., Gajewski, W., Börschinger, B., & Schuster, T.
- Affiliations
- Google (United States) (US)
- ORCID
- Schuster, Tal
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 291-305
- OpenAlex
- W4385572714
- Cited by
- 44
- References
- 41
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 8 · 2023: 12 · 2022: 4
- FWCI
- 4.84
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering evaluation, exact match, SQuAD, human judgment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
