Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 201–300 of 1,158
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 134
Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation
Ma, J., Korotkov, I., Yang, Y., Hall, K., & McDonald, R. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.92 · Cited by 80
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T., & Sadeh, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1500 · Cited by 99
ASQA: Factoid Questions Meet Long-Form Answers
Stelmakh, I., Luan, Y., Dhingra, B., & Chang, M.-W. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.566 · Cited by 63
Self-citationDistilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, E. (2020) · arXiv · doi:10.48550/arxiv.2012.04584 · Cited by 102
“Lee et al. (2019) introduced the inverse cloze task for pre-training retrievers, which can then be fine-tuned end-to-end on question-answering tasks.”
BackgroundAsai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101
“We also sample instances from a couple of knowledge-intensive datasets, Natural Questions (Kwiatkowski et al., 2019), Wizard of Wikipedia (Dinan et al., 2019) and FEVER (Thorne et al., 2018) from the KILT benchmark (Petroni et al., 2021), ASQA (Stel-makh et al., 2022) and multiple QA datasets including ARC-Easy and Op…”
MethodEntity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101
A Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 143
Few-Shot Conversational Dense Retrieval
Yu, S., Liu, Z., Xiong, C., Feng, T., & Liu, Z. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462856 · Cited by 180
Ensemble Multifeatured Deep Learning Models and Applications: A Survey
Abimannan, S., El-Alfy, E.-S. M., Chang, Y., Hussain, S., Shukla, S., & Satheesh, D. (2023) · IEEE Access · doi:10.1109/access.2023.3320042 · Cited by 119
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“Closed-book question answering datasets including TriviaQA (Joshi et al., 2017) and Natu-ralQuestions (Kwiatkowski et al., 2019).”
MethodSimple Entity-Centric Questions Challenge Dense Retrievers
Sciavolino, C., Zhong, Z., Lee, J., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.496 · Cited by 98
Tu, F., Wu, Z., Wang, Y., Liang, L., Liu, L., Ding, Y., Liu, L., Wei, S., Xie, Y., & Yin, S. (2022) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2022.3213542 · Cited by 113
AI-Generated Content (AIGC) for Various Data Modalities: A Survey
Foo, L. G., Rahmani, H., & Liu, J. (2025) · ACM Computing Surveys · doi:10.1145/3728633 · Cited by 138
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 400
Open Domain Question Answering over Tables via Dense Retrieval
Herzig, J., Müller, T., Krichene, S., & Eisenschlos, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.43 · Cited by 78
Transformers for Tabular Data Representation: A Survey of Models and Applications
Badaro, G., Saeed, M., & Papotti, P. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00544 · Cited by 96
“Natural Questions (NQ) (Kwiatkowski et al., 2019) 4 169,898 4 Questions: 320K.”
RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses
Zhuang, H., Qin, Z., Jagerman, R., Hui, K., Ma, J., Lu, J. R., Ni, J., Wang, X., & Bendersky, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592047 · Cited by 300
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 173
VisualMRC: Machine Reading Comprehension on Document Images
Tanaka, R., Nishida, K., & Yoshida, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17635 · Cited by 232
TREC CAsT 2019: The Conversational Assistance Track OverviewPreprint
Dalton, J., Xiong, C., & Callan, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.13624 · Cited by 93
Reasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P., & Gardner, M. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5808 · Cited by 92
Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E., & Fung, P. (2020) · Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020 · doi:10.18653/v1/2020.nlpcovid19-2.14 · Cited by 88
A Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 153
MultiModalQA: Complex Question Answering over Text, Tables and ImagesPreprint
Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06039 · Cited by 276
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
Improving Passage Retrieval with Zero-Shot Question Generation
Sachan, D., Lewis, M., Joshi, M., Aghajanyan, A., Yih, W.-T., Pineau, J., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.249 · Cited by 90
A Survey of Knowledge Enhanced Pre-trained Language Models
Yang, J., Hu, X., Xiao, G., & Shen, Y. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3631392 · Cited by 47
Retrieval Augmented Visual Question Answering with Outside Knowledge
Lin, W., & Byrne, B. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.772 · Cited by 61
Hindi, M., Mohammed, L., Maaz, O., & Alwarafy, A. (2025) · IEEE Access · doi:10.1109/access.2025.3550145 · Cited by 140
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · arXiv · doi:10.48550/arxiv.2305.13281 · Cited by 85
Abbasiantaeb, Z., & Momtazi, S. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1412 · Cited by 101
GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
Wang, K., Thakur, N., Reimers, N., & Gurevych, I. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.168 · Cited by 76
TopiOCQA: Open-domain Conversational Question Answering with Topic Switching
Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00471 · Cited by 74
Learning Dense Representations of Phrases at Scale
Lee, J., Sung, M., Kang, J., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.518 · Cited by 68
Oguz, B., Chen, X., Karpukhin, V., Peshterliev, S., Okhonko, D., Schlichtkrull, M., Gupta, S., Mehdad, Y., & Yih, S. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.115 · Cited by 41
Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., & Hang, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103809 · Cited by 128
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
Min, S., Chen, D., Zettlemoyer, L., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03868 · Cited by 75
Fabbri, A. R., Ng, P., Wang, Z., Nallapati, R. M., & Xiang, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.413 · Cited by 89
Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Gao, L., Zhang, Y., Han, J., & Callan, J. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.31 · Cited by 230
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79
Few-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 98
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts
Asai, A., Salehi, M., Peters, M. E., & Hajishirzi, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.446 · Cited by 178
Shuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.27 · Cited by 72
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., & Chang, M.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.925 · Cited by 294
Self-citationLearning to Extract Attribute Value from Product via Question Answering: A Multi-task Approach
Wang, Q., Li, Y., Kanagal, B., Sanghai, S. K., Sivakumar, D., Shu, B., Yu, Z., & Elsas, J. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403047 · Cited by 125
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., & Burtsev, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.367 · Cited by 157
A Simple and Effective Model for Answering Multi-span Questions
Segal, E., Efrat, A., Shoham, M., Globerson, A., & Berant, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.248 · Cited by 73
The Effect of Natural Distribution Shift on Question Answering Models
Miller, J., Krauth, K., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2004.14444 · Cited by 74
“Beyond SQuAD, a number of works have proposed datasets for question answering (Richardson et al., 2013; Berant et al., 2014; Joshi et al., 2017; Trischler et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019).”
BackgroundEvaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74
End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems
Shakeri, S., Nogueira dos Santos, C., Zhu, H., Ng, P., Nan, F., Wang, Z., Nallapati, R., & Xiang, B. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.439 · Cited by 59
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
Sachan, D., Patwary, M., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., & Catanzaro, B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.519 · Cited by 62
Dense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.845 · Cited by 74
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2023) · arXiv · doi:10.48550/arxiv.2307.16877 · Cited by 74
Geographic Question Answering: Challenges, Uniqueness, Classification, and Future Directions
Mai, G., Janowicz, K., Zhu, R., Cai, L., & Lao, N. (2021) · AGILE GIScience Series · doi:10.5194/agile-giss-2-8-2021 · Cited by 120
Universal Information Extraction as Unified Semantic Matching
Lou, J., Lu, Y., Dai, D., Jia, W., Lin, H., Han, X., Sun, L., & Wu, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26563 · Cited by 132
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · arXiv · doi:10.48550/arxiv.2305.14938 · Cited by 73
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00494 · Cited by 72
Reducing conversational agents' overconfidence through linguistic calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2020) · arXiv · doi:10.48550/arxiv.2012.14983 · Cited by 72
Relevance-guided Supervision for OpenQA with ColBERT
Khattab, O., Potts, C., & Zaharia, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00405 · Cited by 71
SituatedQA: Incorporating Extra-Linguistic Contexts into QA
Zhang, M., & Choi, E. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.586 · Cited by 28
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5827 · Cited by 69
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71
Self-citationSKILL: Structured Knowledge Infusion for Large Language Models
Moiseev, F., Dong, Z., Alfonseca, E., & Jaggi, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.113 · Cited by 71
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.125 · Cited by 42
On the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 210
SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Efimov, P., Chertok, A., Boytsov, L., & Braslavski, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58219-7_1 · Cited by 106
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2021) · ACM Transactions on Information Systems · doi:10.1145/3446426 · Cited by 98
Context-faithful Prompting for Large Language Models
Zhou, W., Zhang, S., Poon, H., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.968 · Cited by 46
Span Selection Pre-training for Question Answering
Glass, M., Gliozzo, A., Chakravarti, R., Ferritto, A., Pan, L., Bhargav, G. P. S., Garg, D., & Sil, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.247 · Cited by 62
MATE: Multi-view Attention for Table Transformer Efficiency
Eisenschlos, J., Gor, M., Müller, T., & Cohen, W. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.600 · Cited by 67
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671620 · Cited by 274
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 56
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 63
(2022) · doi:10.1145/3511808.3557325 · Cited by 67
Open-Domain Question Answering
Chen, D., & Yih, W.-T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts · doi:10.18653/v1/2020.acl-tutorials.8 · Cited by 64
Long-Short Transformer: Efficient Transformers for Language and Vision
Zhu, C., Ping, W., Xiao, C., Shoeybi, M., Goldstein, T., Anandkumar, A., & Catanzaro, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02192 · Cited by 55
Document Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 216
LLM-Based Agents for Tool Learning: A Survey
Xu, W., Huang, C., Gao, S., & Shang, S. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00296-9 · Cited by 166
ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)Preprint
Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., & Burtsev, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11352 · Cited by 67
Explanations for CommonsenseQA: New Dataset and Models
Aggarwal, S., Mandowara, D., Agrawal, V., Khandelwal, D., Singla, P., & Garg, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.238 · Cited by 200
REPLUG: Retrieval-augmented black-box language models
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2023) · arXiv · doi:10.48550/arxiv.2301.12652 · Cited by 65
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.14249 · Cited by 65
“This level of expert disa-greement is in line with what is observed in other well-known machine learning benchmarks 59–62 .”
Self-citationMr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
Zhang, X., Ma, X., Shi, P., & Lin, J. (2021) · Proceedings of the 1st Workshop on Multilingual Representation Learning · doi:10.18653/v1/2021.mrl-1.12 · Cited by 51
Laskar, M. T. R., Huang, J. X., & Hoque, E. (2020) · Language Resources and Evaluation · Cited by 133
Lee, D. B., Lee, S., Jeong, W. T., Kim, D., & Hwang, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.20 · Cited by 69
Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
Yu, H., Xiong, C., & Callan, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482124 · Cited by 119
Pre-training Methods in Information Retrieval
Fan, Y., Xie, X., Cai, Y., Chen, J., Ma, X., Li, X., Zhang, R., & Guo, J. (2022) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000100 · Cited by 98
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences
Liu, X., Lai, H., Yu, H., Xu, Y., Zeng, A., Du, Z., Zhang, P., Dong, Y., & Tang, J. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599931 · Cited by 45
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 84
MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00595 · Cited by 274
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
Min, S., Boyd-Graber, J. L., Alberti, C., Chen, D., Choi, E., Collins, M., Guu, K., Hajishirzi, H., Lee, K., Palomaki, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00133 · Cited by 59
Self-citationGEO: Generative Engine Optimization
Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671900 · Cited by 62
“Natural Questions: (Kwiatkowski et al., 2019; Alexander et al., 2022; Craswell et al., 2021) These datasets contain real anonymized user queries from Bing and Google Search Engines.”
BackgroundQuestion Answering with Long Multiple-Span Answers
Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., & Reddy, C. K. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.342 · Cited by 57
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 121
Self-citationPAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval
Ren, R., Lv, S., Qu, Y., Liu, J., Zhao, W. X., She, Q., Wu, H., Wang, H., & Wen, J.-R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.191 · Cited by 110
Semantic Answer Similarity for Evaluating Question Answering Models
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.15 · Cited by 83
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 32
Poisoning Retrieval Corpora by Injecting Adversarial Passages
Zhong, Z., Huang, Z., Wettig, A., & Chen, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.849 · Cited by 253
Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models
Zhang, X., Zhang, Y., Xie, W., Li, M., Dai, Z., Long, D., Xie, P., Zhang, M., Li, W., & Zhang, M. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00866 · Cited by 296
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.