Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 701–800 of 1,158
ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models
Newman, B., Lee, Y., Naik, A., Siangliulue, P., Fok, R., Kim, J., Weld, D. S., Chang, J. C., & Lo, K. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.538 · Cited by 19
“In particular, the large body of literature on document-grounded question-answering (Kwiatkowski et al., 2019; Dasigi et al., 2021; Lee et al., 2023), information extraction (Luan et al., 2018), and query (Zhong et al., 2021; Xu and Lapata, 2020) or aspect-based summarization (Yang et al., 2023;…”
BackgroundSelectively Answering Ambiguous Questions
Cole, J. R., Zhang, M. J. Q., Gillick, D., Eisenschlos, J. M., Dhingra, B., & Eisenstein, J. (2023) · arXiv · doi:10.48550/arxiv.2305.14613 · Cited by 19
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.776 · Cited by 18
Comparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18
Plug-and-Play Adaptation for Continuously-updated QA
Lee, K., Han, W., Hwang, S.-W., Lee, H., Park, J., & Lee, S.-W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.37 · Cited by 31
Self-citationCREPE: Open-Domain Question Answering with False Presuppositions
Yu, X., Min, S., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.583 · Cited by 90
A Causal View of Entity Bias in (Large) Language Models
Wang, F., Mo, W., Wang, Y., Zhou, W., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1013 · Cited by 77
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Shi, C., Su, Y., Yang, C. H., Yang, Y., & Cai, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.947 · Cited by 45
Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation
Yang, E., Lawrie, D., Mayfield, J., Oard, D. W., & Miller, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_4 · Cited by 40
Integration of cognitive tasks into artificial general intelligence test for large models
Qu, Y., Wei, C., Du, P., Che, W., Zhang, C., Ouyang, W., Bian, Y., Xu, F., Hu, B., Du, K., et al. (2024) · iScience · doi:10.1016/j.isci.2024.109550 · Cited by 25
Text Embedding Inversion Security for Multilingual Language Models
Chen, Y., Lent, H., & Bjerva, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.422 · Cited by 64
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
Zhang, Q., Wang, L., Yu, S., Wang, S., Wang, Y., Jiang, J., & Lim, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.350 · Cited by 28
Garg, S., & Moschitti, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.583 · Cited by 32
SD-QA: Spoken Dialectal Question Answering for the Real World
Faisal, F., Keshava, S., Alam, M. M. I., & Anastasopoulos, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.281 · Cited by 61
Gokhale, T., Mishra, S. R., Luo, M., Sachdeva, B., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.213 · Cited by 34
Fan, A., & Gardent, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.586 · Cited by 38
STARC: Structured Annotations for Reading Comprehension
Berzak, Y., Malmaud, J., & Lévy, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.507 · Cited by 46
Question Answering for Privacy Policies: Combining Computational and Legal PerspectivesPreprint
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00841 · Cited by 71
End-to-End QA on COVID-19: Domain Adaptation with Synthetic TrainingPreprint
Reddy, R. G., Iyer, B., Sultan, M. A., Zhang, R., Sil, A., Castelli, V., Florian, R., & Roukos, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01414 · Cited by 20
Active Retrieval Augmented Generation
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2305.06983 · Cited by 18
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.19234 · Cited by 17
“We use two datasets commonly employed in RAG attack research: Natural Questions [24] and Trivia-QA [25], in accordance with existing works [7], [26], [27].”
Physics of language models: Part 3.3, knowledge capacity scaling laws
Allen-Zhu, Z., & Li, Y. (2025) · doi:10.2139/ssrn.5250617 · Cited by 17
Huang, J., Wang, M., Cui, Y., Liu, J., Chen, L., Wang, T., Li, H., & Wu, J. (2024) · Applied Sciences · doi:10.3390/app142311014 · Cited by 17
Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review
Kuznetsov, I., Buchmann, J., Eichler, M., & Gurevych, I. (2022) · Computational Linguistics · doi:10.1162/coli_a_00455 · Cited by 17
An Argumentative Dialogue System for COVID-19 Vaccine Information
Fazzinga, B., Galassi, A., & Torroni, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89391-0_27 · Cited by 31
Evidence Selection as a Token-Level Prediction Task
Stammbach, D. (2021) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/2021.fever-1.2 · Cited by 19
Document Summarization with Latent Queries
Xu, Y., & Lapata, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00480 · Cited by 30
SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
Choi, E., Lee, S., Choi, M., Ko, H., Song, Y.-I., & Lee, J. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557456 · Cited by 28
Open-WikiTable : Dataset for Open Domain Question Answering with Complex Reasoning over Table
Kweon, S., Kwon, Y., Cho, S., Jo, Y., & Choi, E. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.526 · Cited by 70
Saving Dense Retriever from Shortcut Dependency in Conversational Search
Kim, S., & Kim, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.701 · Cited by 21
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Liello, L. D., Garg, S., Soldaini, L., & Moschitti, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.810 · Cited by 22
Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., & Ravfogel, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.220 · Cited by 143
Experimental Analysis of Large-Scale Learnable Vector Storage Compression
Zhang, H., Zhao, P., Miao, X., Shao, Y., Liu, Z., Yang, T., & Cui, B. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3636218.3636234 · Cited by 12
Yu, Q., Xiao, Z., Li, B., Wang, Z., Chen, C., & Zhang, W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730288 · Cited by 48
Novel regularization method for the class imbalance problem
Kim, B., Ko, Y., & Seo, J. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115974 · Cited by 18
What’s in a Name? Answer Equivalence For Open-Domain Question Answering
Si, C., Zhao, C., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.757 · Cited by 17
Towards Understanding and Answering Comparative Questions
Bondarenko, A., Ajjour, Y., Dittmar, V., Homann, N., Braslavski, P., & Hagen, M. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498534 · Cited by 19
Provably Confidential Language Modelling
Zhao, X., Li, L., & Wang, Y.-X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.69 · Cited by 26
Tevatron: An Efficient and Flexible Toolkit for Neural Retrieval
Gao, L., Ma, X., Lin, J., & Callan, J. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591805 · Cited by 39
Re-Examining Calibration: The Case of Question Answering
Si, C., Zhao, C., Min, S., & Boyd-Graber, J. L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.204 · Cited by 52
The First Wikipedia Questions and Factoid Answers Corpus in the Thai Language
Trakultaweekoon, K., Thaiprayoon, S., Palingoon, P., & Rugchatjaroen, A. (2019) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp48611.2019.9045143 · Cited by 17
Unsupervised Domain Adaptation of Language Models for Reading ComprehensionPreprint
Nishida, K., Nishida, K., Saito, I., Asano, H., & Tomita, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10768 · Cited by 25
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?Preprint
Chen, X., Lakhotia, K., Oğuz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06918 · Cited by 51
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 43
PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
Zou, W., Geng, R., Wang, B., & Jia, J. (2024) · arXiv · doi:10.48550/arxiv.2402.07867 · Cited by 16
“Datasets: We use three benchmark question-answering datasets in our evaluation: Natural Questions (NQ) [36], HotpotQA [37], and MS-MARCO [38], where each dataset has a knowledge database.”
MethodREIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
Chen, K., Nadig, R., Frouzakis, M., Ghiasi, N. M., Liang, Y., Mao, H., Park, J., Sadrosadati, M., & Mutlu, O. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731116 · Cited by 25
Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models
Kostić, B., Risch, J., & Möller, T. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.8 · Cited by 20
Li, Z., Sharma, P., Lù, X. H., Cheung, J. C. K., & Reddy, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.75 · Cited by 39
Kia, M. A., Garifullina, A., Kern, M., Chamberlain, J., & Jameel, S. (2022) · IEEE Access · doi:10.1109/access.2022.3170466 · Cited by 18
How to Quantify the Degree of Explainability: Experiments and Practical Implications
Sovrano, F., & Vitali, F. (2022) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee55066.2022.9882574 · Cited by 22
Hybrid Hierarchical Retrieval for Open-Domain Question Answering
Arivazhagan, M. G., Liu, L., Qi, P., Chen, X., Wang, W. Y., & Huang, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.679 · Cited by 56
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
Li, X., Liu, Z., Xiong, C., Shi, Y., Gu, Y., Liu, Z., & Yu, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.734 · Cited by 52
Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems
Shi, T., Li, L., Lin, Z., Yang, T., Quan, X., & Wang, Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.405 · Cited by 33
RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
Hoshi, Y., Miyashita, D., Ng, Y., Tatsuno, K., Morioka, Y., Torii, O., & Deguchi, J. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.4 · Cited by 24
Xu, S., Jin, X., Qu, H., & Yan, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713496 · Cited by 24
Shaikh, T. A., Rasool, T., & Mir, W. A. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104005 · Cited by 22
Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019
Webster, K., Costa‐jussà, M. R., Hardmeier, C., & Radford, W. (2019) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/w19-3801 · Cited by 31
Generating Biomedical Question Answering Corpora From Q&A Forums
Lamúrias, A., Sousa, D., & Couto, F. M. (2020) · IEEE Access · doi:10.1109/access.2020.3020868 · Cited by 27
Improving Spoken Question Answering Using Contextualized Word Representation
Su, D., & Fung, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053979 · Cited by 17
Answer Sentence Selection Using Local and Global Context in Transformer Models
Lauriola, I., & Moschitti, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_20 · Cited by 18
You Don’t Need Labeled Data for Open-Book Question Answering
Gholami, S., & Noori, M. (2021) · Applied Sciences · doi:10.3390/app12010111 · Cited by 22
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V. A., Lamm, M., Aroyo, L. M., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00490 · Cited by 69
Self-citationQuery Refinement Prompts for Closed-Book Long-Form QA
Amplayo, R. K., Webster, K., Collins, M., Das, D., & Narayan, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.444 · Cited by 31
Self-citationDuReader-Retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine
Qiu, Y., Li, H., Qu, Y., Chen, Y., She, Q., Liu, J., Wu, H. Y., & Wang, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.357 · Cited by 48
NLQuAD: A Non-Factoid Long Question Answering Data Set
Soleimani, A., Monz, C., & Worring, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.106 · Cited by 30
ContextCite: Attributing Model Generation to Context
Cohen-Wang, B., Shah, H., Georgiev, K., & Madry, A. (2024) · arXiv · doi:10.48550/arxiv.2409.00729 · Cited by 16
Generative Retrieval as Multi-Vector Dense Retrieval
Wu, S., Wei, W., Zhang, M., Chen, Z., Ma, J., Ren, Z., de Rijke, M., & Ren, P. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657697 · Cited by 15
Getting MoRE out of Mixture of Language Model Reasoning Experts
Si, C., Shi, W., Zhao, C., Zettlemoyer, L., & Boyd-Graber, J. L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.552 · Cited by 67
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA
Lee, D., Kim, S., Lee, M., Lee, H., Park, J., Lee, S., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.772 · Cited by 58
BehnamGhader, P., Miret, S., & Reddy, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1036 · Cited by 46
Development of system for generating questions, answers, distractors using transformers
Barlybayev, A., & Matkarimov, B. T. (2024) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v14i2.pp1851-1863 · Cited by 17
Retrieval Augmented Generation with Rich Answer Encoding
Huang, W., Lapata, M., Vougiouklis, P., Papasarantopoulos, N., & Pan, J. Z. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.65 · Cited by 36
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
Wang, B., Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6450 · Cited by 26
Multi-Domain Multilingual Question Answering
Ruder, S., & Sil, A. (2021) · doi:10.18653/v1/2021.emnlp-tutorials.4 · Cited by 23
Multi-task joint training model for machine reading comprehension
Li, F., Shan, Y., Mao, X., Ren, X., Liu, X., & Zhang, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.082 · Cited by 17
MetaQA: Combining Expert Agents for Multi-Skill Question Answering
Puerto, H., Şahin, G. G., & Gurevych, I. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.259 · Cited by 23
Data Augmentation for Biomedical Factoid Question Answering
Pappas, D., Malakasiotis, P., & Androutsopoulos, I. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.6 · Cited by 24
Multi-Hop Open-Domain Question Answering over Structured and Unstructured Knowledge
Feng, Y., Han, Z. Y., Sun, M., & Li, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.12 · Cited by 15
Question Answering with BERT: designing a 3D virtual avatar for Cultural Heritage exploration
Farella, M., Chiazzese, G., & Bosco, G. L. (2022) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melecon53508.2022.9843028 · Cited by 18
Knowledge Bases and Language Models: Complementing Forces
Suchanek, F. M., & Tuan, L. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45072-3_1 · Cited by 23
TOME: A Two-stage Approach for Model-based Retrieval
Ren, R., Zhao, W. X., Liu, J., Wu, H., Wen, J.-R., & Wang, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.336 · Cited by 14
Kedia, A., Zaidi, M. A., & Lee, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.285 · Cited by 14
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143
Measuring short-form factuality in large language models
Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024) · arXiv · doi:10.48550/arxiv.2411.04368 · Cited by 14
“Compared to older benchmarks such as TriviaQA (Joshi et al., 2017) or Natural Questions (Kwiatkowski et al., 2019) that are now saturated, SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
BackgroundQuery Rewriting for Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2305.14283 · Cited by 14
“(ii) AmbigNQ (Min et al., 2020) provides a disambiguated version of Natural Questions (NQ) (Kwiatkowski et al., 2019).”
BackgroundCompAct: Compressing Retrieved Documents Actively for Question Answering
Yoon, C., Lee, T., Hwang, H., Jeong, M., & Kang, J. (2024) · arXiv · doi:10.48550/arxiv.2407.09014 · Cited by 14
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Huber, P., Aghajanyan, A., Oğuz, B., Okhonko, D., Yih, S., Gupta, S., & Chen, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.184 · Cited by 20
HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data
Nakamura, K., Levy, S., Tuan, Y.-L., Chen, W., & Wang, W. Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.41 · Cited by 39
DuReadervis: A : A Chinese Dataset for Open-domain Document Visual Question Answering
Qi, L., Lv, S., Li, H., Liu, J., Zhang, Y., She, Q., Wu, H. J., Wang, H., & Liu, T. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.105 · Cited by 29
DocFlow: A Visual Analytics System for Question-Based Document Retrieval and Categorization
Qiu, R., Tu, Y., Wang, Y.-S., Yen, P., & Shen, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3219762 · Cited by 16
Decomposing Complex Queries for Tip-of-the-tongue Retrieval
Lin, K., Lo, K., Gonzalez, J. E., & Klein, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.367 · Cited by 39
Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback
Zhou, Y., Dou, Z., & Wen, J.-R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.768 · Cited by 46
Listwise Generative Retrieval Models via a Sequential Learning Process
Tang, Y., Zhang, R., Guo, J., de Rijke, M., Chen, W., & Cheng, X. (2024) · ACM Transactions on Information Systems · doi:10.1145/3653712 · Cited by 37
Event extraction as machine reading comprehension with question-context bridging
Liu, L., Liu, M., Liu, S., & Ding, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112041 · Cited by 16
Sun, J., Pan, Y., & Yan, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129219 · Cited by 23
Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering
Keluskar, A., Bhattacharjee, A., & Liu, H. (2024) · BigData Congress [Services Society] · doi:10.1109/bigdata62323.2024.10825265 · Cited by 33
Is Relevance Propagated from Retriever to Generator in RAG?
Tian, F., Ganguly, D., & Macdonald, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_3 · Cited by 33
Robust Neural Information Retrieval: An Adversarial and Out-of-Distribution Perspective
Liu, Y.-A., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768153 · Cited by 40
QuGAN: Quasi Generative Adversarial Network for Tibetan Question Answering Corpus Generation
Sun, Y., Chen, C., Xia, T., & Zhao, X. (2019) · IEEE Access · doi:10.1109/access.2019.2934581 · Cited by 15
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.