Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 301–400 of 1,158
Table Search Using a Deep Contextualized Language Model
Chen, Z., Trabelsi, M., Heflin, J., Xu, Y., & Davison, B. D. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401044 · Cited by 89
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question AnsweringPreprint
Sachan, D. S., Reddy, S., Hamilton, W. L., Dyer, C., & Yogatama, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05346 · Cited by 78
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., & Jin, X. (2024) · arXiv · doi:10.48550/arxiv.2404.12457 · Cited by 61
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, S., Liu, X., & Jin, X. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3768628 · Cited by 61
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Kassner, N., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.307 · Cited by 63
CAsT-19: A Dataset for Conversational Information Seeking
Dalton, J., Xiong, C., Kumar, V., & Callan, J. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401206 · Cited by 62
Pilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 95
Adversarial Retriever-Ranker for dense text retrievalPreprint
Zhang, H., Gong, Y., Shen, Y., Lv, J., Duan, N., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03611 · Cited by 98
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
More Bang for Your Buck: Natural Perturbation for Robust Question Answering
Khashabi, D., Khot, T., & Sabharwal, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.12 · Cited by 56
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering
Yu, D., Zhu, C., Fang, Y., Yu, W., Wang, S., Xu, Y., Ren, X., Yang, Y., & Zeng, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.340 · Cited by 46
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Jang, J., Ye, S., Lee, C.-H., Yang, S., Shin, J., Han, J., Kim, G., & Seo, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.418 · Cited by 173
Large language models present new questions for decision support
Handler, A., Larsen, K. R. T., & Hackathorn, R. D. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102811 · Cited by 107
Improving Subject-Area Question Answering with External Knowledge
Pan, X., Sun, K., Yu, D., Chen, J., Ji, H., Cardie, C., & Yu, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5804 · Cited by 72
Look at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 50
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?
Pruthi, D., Bansal, R., Dhingra, B., Soares, L. B., Collins, M., Lipton, Z. C., Neubig, G., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00465 · Cited by 55
Self-citationKnowledge-Aware Language Model Pretraining
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 40
Retrieval-augmented Generation across Heterogeneous Knowledge
Yu, W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.7 · Cited by 97
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117
“As a first step toward investigating this question, we apply ITI—using the activation shift directions and hyperparameters learned from TruthfulQA—to two different data sets that relate to real-world truth: Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
MethodCONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
Wu, Z., Luan, Y., Rashkin, H., Reitter, D., Hajishirzi, H., Ostendorf, M., & Tomar, G. S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.679 · Cited by 136
CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
Chen, J., Zhang, R., Guo, J., Liu, Y., Fan, Y., & Cheng, X. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557271 · Cited by 57
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
“We examine CRITIC across diverse tasks: free-form question answering concentrates on truthfulness related to open-ended general factual knowledge (Kwiatkowski et al., 2019; Min et al., 2020; Joshi et al., 2017) and multi-hop reasoning (Yang et al.”
BackgroundInfluentialTeach Me to Explain: A Review of Datasets for Explainable NLP.Preprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · Cited by 64
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.463 · Cited by 47
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?
Wang, C., Liu, P., & Zhang, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.251 · Cited by 55
Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Zheng, B., Wen, H., Liang, Y., Duan, N., Che, W., Jiang, D., Zhou, M., & Liu, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.599 · Cited by 55
RikiNet: Reading Wikipedia Pages for Natural Question Answering
Liu, D., Gong, Y., Fu, J., Yan, Y., Chen, J., Jiang, D., Lv, J., & Duan, N. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.604 · Cited by 57
All You May Need for VQA are Image Captions
Changpinyo, S., Kukliansy, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.142 · Cited by 101
Decontextualization: Making Sentences Stand-Alone
Choi, E., Palomaki, J., Lamm, M., Kwiatkowski, T., Das, D., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00377 · Cited by 54
Self-citationAn objective metric for Explainable AI: How and why to estimate the degree of explainability
Sovrano, F., & Vitali, F. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110866 · Cited by 79
Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1221 · Cited by 72
Machine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 79
Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization
Laskar, M. T. R., Hoque, E., & Huang, J. X. (2022) · Computational Linguistics · doi:10.1162/coli_a_00434 · Cited by 81
The Cascade Transformer: an Application for Efficient Answer Sentence Selection
Soldaini, L., & Moschitti, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.504 · Cited by 68
Karamcheti, S., Krishna, R., Fei-Fei, L., & Manning, C. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.564 · Cited by 84
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 57
Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering
Banerjee, P., & Baral, C. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.11 · Cited by 76
The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes
Reimers, N., & Gurevych, I. (2021) · doi:10.18653/v1/2021.acl-short.77 · Cited by 106
Answering Open-Domain Questions of Varying Reasoning Steps from Text
Qi, P., Lee, H., Sido, T., & Manning, C. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.292 · Cited by 38
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591687 · Cited by 51
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 93
Can NLI Models Verify QA Systems’ Predictions?
Chen, J., Choi, E., & Durrett, G. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.324 · Cited by 47
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Ye, F., Fang, M., Li, S., & Yılmaz, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.398 · Cited by 147
Towards Human-centered Proactive Conversational Agents
Deng, Y., Liao, L., Zheng, Z., Yang, G. H., & Chua, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657843 · Cited by 141
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 158
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation
Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2022) · arXiv · doi:10.48550/arxiv.2209.14290 · Cited by 51
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
DeepSeek-AI, Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.11931 · Cited by 51
ReQA: An Evaluation for End-to-End Answer Retrieval Models
Ahmad, A., Constant, N., Yang, Y., & Cer, D. M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5819 · Cited by 63
Sharpness-Aware Minimization Improves Language Model Generalization
Bahri, D., Mobahi, H., & Tay, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.508 · Cited by 139
Improved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News
Deka, P., Jurek-Loughrey, A., & P., D. (2022) · Journal of Data Intelligence · doi:10.26421/jdi3.4-5 · Cited by 49
Source-Free Unsupervised Domain Adaptation: Current research and future directions
Zhang, N., Lü, J., Li, K., Fang, Z., & Zhang, G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126921 · Cited by 57
Retrieval-Generation Synergy Augmented Large Language Models
Feng, Z., Feng, X., Zhao, D., Yang, M., & Qin, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448015 · Cited by 117
Results of the Seventh Edition of the BioASQ Challenge
Nentidis, A., Bougiatiotis, K., Krithara, A., & Παλιούρας, Γ. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43887-6_51 · Cited by 79
Unsupervised Multi-hop Question Answering by Question Generation
Pan, L., Chen, W., Xiong, W., Kan, M., & Wang, W. Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.469 · Cited by 76
Deep learning based question answering system in Bengali
Mayeesha, T. T., Sarwar, A. M., & Rahman, R. M. (2020) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1833136 · Cited by 66
Towards Continual Knowledge Learning of Language Models
Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S. J., & Seo, M. (2021) · arXiv · doi:10.48550/arxiv.2110.03215 · Cited by 49
“Configuration KILT (Petroni et al., 2021) consists of 5 different tasks and 11 datasets: OpenDomain Question Answering (Joshi et al., 2017; Kwiatkowski et al., 2019; Fan et al., 2019; Yang et al., 2018), Fact Checking (Thorne et al., 2018), Entity Linking (Hoffart et al., 2011; Guo & Barbosa, 2018), Slot-filling (Levy…”
BackgroundLess is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T.-Y., & Overwijk, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.220 · Cited by 49
Multi-Task Retrieval for Knowledge-Intensive Tasks
Maillard, J., Karpukhin, V., Petroni, F., Yih, W.-T., Oguz, B., Stoyanov, V., & Ghosh, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.89 · Cited by 47
Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models
Kim, G., Kim, S., Jeon, B., Park, J., & Kang, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.63 · Cited by 35
Chakraborty, N., Ornik, M., & Driggs-Campbell, K. (2025) · ACM Computing Surveys · doi:10.1145/3716846 · Cited by 97
Towards Ecologically Valid Research on Language User Interfaces
de Vries, H., Bahdanau, D., & Manning, C. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14435 · Cited by 40
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 377
“For example, training on 100k ChatGPT outputs from broad-coverage user inputs provides no benefits to Natural Questions accuracy (e.g., Figure 1, center), but training exclusively on ChatGPT responses for Natural-Questions-like queries drastically improves task accuracy.”
MethodInfluentialAnswering Questions on COVID-19 in Real-Time
Lee, J., Yi, S. S., Jeong, M., Sung, M., Yoon, W., Choi, Y., Ko, M., & Kang, J. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.1 · Cited by 58
Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
Orr, L., Leszczynski, M., Arora, S., Wu, S., Guha, N., Xiao, L., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10363 · Cited by 40
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47
Web Table Retrieval using Multimodal Deep Learning
Shraga, R., Roitman, H., Feigenblat, G., & Canim, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401120 · Cited by 53
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 41
Quiz-Style Question Generation for News Stories
Lelkes, A. D., Tran, V. Q., & Yu, C. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449892 · Cited by 43
Task-aware Retrieval with Instructions
Asai, A., Schick, T., Lewis, P., Chen, X., Izacard, G., Riedel, S., Hajishirzi, H., & Yih, W.-T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.225 · Cited by 46
Capturing Row and Column Semantics in Transformer Based Question Answering over Tables
Glass, M., Canim, M., Gliozzo, A., Chemmengath, S., Kumar, V., Chakravarti, R., Sil, A., Pan, F., Bharadwaj, S., & Fauceglia, N. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.96 · Cited by 68
Domain-matched Pre-training Tasks for Dense Retrieval
Oğuz, B., Lakhotia, K., Gupta, A., Lewis, P., Karpukhin, V., Piktus, A., Chen, X., Riedel, S., Yih, S., Gupta, S., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.114 · Cited by 65
It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story Books
Yao, B., Wang, D., Wu, T., Zhang, Z., Li, T., Yu, M., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.54 · Cited by 85
A Survey for Efficient Open Domain Question Answering
Zhang, Q., Chen, S., Xu, D., Cao, Q., Chen, X., Cohn, T., & Fang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.808 · Cited by 89
QED: A Framework and Dataset for Explanations in Question Answering
Lamm, M., Palomaki, J., Alberti, C., Andor, D., Choi, E., Soares, L. B., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00398 · Cited by 43
Self-citationWang, N. X. R., Mahajan, D., Danilevsky, M., & Rosenthal, S. (2021) · Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) · doi:10.18653/v1/2021.semeval-1.39 · Cited by 36
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021) · arXiv · doi:10.48550/arxiv.2104.08663 · Cited by 45
Multi-View Document Representation Learning for Open-Domain Dense Retrieval
Zhang, S., Liang, Y., Gong, M., Jiang, D., & Duan, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.414 · Cited by 93
Yu, Y., Xiong, C., Sun, S., Zhang, C., & Overwijk, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.95 · Cited by 92
Truth-O-Meter: Collaborating with LLM in fighting its hallucinations
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00004-3 · Cited by 50
Mitigating privacy risks in Retrieval-Augmented Generation via locally private entity perturbation
He, L., Tang, P., Zhang, Y., Zhou, P., & Su, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104150 · Cited by 63
Survey of Neural Text Representation Models
Babić, K., Martinčić-Ipšić, S., & Meštrović, A. (2020) · Information · doi:10.3390/info11110511 · Cited by 54
General-Purpose Question-Answering with MacawPreprint
Tafjord, O., & Clark, P. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02593 · Cited by 57
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Jang, J., Ye, S., Lee, C., Yang, S., Shin, J., Han, J., Kim, G., & Seo, M. (2022) · arXiv · doi:10.48550/arxiv.2204.14211 · Cited by 45
Accelerating Retrieval-Augmented Generation
Quinn, D., Nouri, M., Patel, N., Salihu, J., Salemi, A., Lee, S., Zamani, H., & Alian, M. (2025) · Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 · doi:10.1145/3669940.3707264 · Cited by 48
MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents
Feng, S., Patel, S. S., Wan, H., & Joshi, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.498 · Cited by 44
English Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 69
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 124
A survey of deep learning techniques for machine reading comprehension
Kazi, S., Khoja, S. A., & Daud, A. Y. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10583-4 · Cited by 52
Xu, H., Zhang, N., Yin, Z., & Njock, P. G. A. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106257 · Cited by 51
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension
Xu, Y., Zhong, X., Yepes, A. J., & Lau, J. H. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206891 · Cited by 62
ABNIRML: Analyzing the Behavior of Neural IR Models
MacAvaney, S., Feldman, S., Goharian, N., Downey, D., & Cohan, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00457 · Cited by 78
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Cheng, H., Shen, Y., Liu, X., He, P., Chen, W., & Gao, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.240 · Cited by 35
Promptagator: Few-shot Dense Retrieval From 8 Examples
Dai, Z., Zhao, V. Y., Ma, J., Luan, Y., Ni, J., Lu, J., Bakalov, A., Guu, K., Hall, K. B., & Chang, M.-W. (2022) · arXiv · doi:10.48550/arxiv.2209.11755 · Cited by 43
“This model was trained on a collection of datasets including question-answer datasets; specifically, it includes Natural Questions (NQ) and Quora.”
MethodInfluentialSelf-citationGenerating fact checking briefs
Fan, A., Piktus, A., Petroni, F., Wenzek, G., Saeidi, M., Vlachos, A., Bordes, A., & Riedel, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.580 · Cited by 42
Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering
Hu, Z., Xu, Y., Yu, W., Wang, S., Yang, Z., Zhu, C., Chang, K.-W., & Sun, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.650 · Cited by 23
Metacognitive Retrieval-Augmented Large Language Models
Zhou, Y., Liu, Z., Jin, J., Nie, J.-Y., & Dou, Z. (2024) · Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589334.3645481 · Cited by 38
Prompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 107
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Dong, G., Zhu, Y., Zhang, C.-H., Wang, Z., Wen, J.-R., & Dou, Z. (2024) · The Web Conference · doi:10.1145/3696410.3714717 · Cited by 59
HalluLens: LLM Hallucination Benchmark
Bang, Y., Ji, Z., Schelten, A., Hartshorn, A., Fowler, T., Zhang, C., Cancedda, N., & Fung, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1176 · Cited by 41
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., & Abend, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.559 · Cited by 135
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.