Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 401–500 of 1,158
How Does Generative Retrieval Scale to Millions of Passages?
Pradeep, R., Hui, K., Gupta, J., Lelkes, A., Zhuang, H., Lin, J., Metzler, D., & Tran, V. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.83 · Cited by 28
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H., Tran, V., So, D., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 26
Self-citationAQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization
Kulkarni, S., Chammas, S., Zhu, W., Sha, F., & Ie, E. (2020) · arXiv · doi:10.48550/arxiv.2010.12694
Continual Learning for Generative Retrieval over Dynamic Corpora
Chen, J., Zhang, R., Guo, J., de Rijke, M., Chen, W., Fan, Y., & Cheng, X. (2023) · Proceedings of the 32nd ACM International Conference on Information and Knowledge Management · doi:10.1145/3583780.3614821 · Cited by 40
“To facilitate the study of continual document learning for GR, we build two benchmark datasets, i.e., CDI-MS and CDI-NQ, from MS MARCO 300k [33, 36, 54] and Natural Questions (NQ) [25], respectively.”
InfluentialA Dataset for Answering Time-Sensitive Questions
Chen, W., Wang, X., & Wang, W. Y. (2021) · arXiv · doi:10.48550/arxiv.2108.06314 · Cited by 40
“There are a few major differences: 1) SituatedQA contains more realistic queries selected from NQ dataset [Kwiatkowski et al., 2019] while TimeQA contains mostly synthesized queries (except human-paraphrased subset).”
BackgroundMethodResultInfluentialSCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40
Knowledge graph-extended retrieval augmented generation for question answering
Linders, J., & Tomczak, J. M. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06885-5 · Cited by 40
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Chen, X., Lakhotia, K., Oğuz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.19 · Cited by 66
Reranking for Efficient Transformer-based Answer Selection
Matsubara, Y., Vu, T., & Moschitti, A. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401266 · Cited by 56
DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding
Nie, P., Zhang, Y., Geng, X., Ramamurthy, A., Song, L., & Jiang, D. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401271 · Cited by 68
Differentiable Open-Ended Commonsense Reasoning
Lin, B. Y., Sun, H., Dhingra, B., Zaheer, M., Ren, X., & Cohen, W. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.366 · Cited by 64
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models
Yin, D., Bansal, H., Monajatipoor, M., Li, L. H., & Chang, K.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.132 · Cited by 123
Large Language Models are Built-in Autoregressive Search Engines
Ziems, N., Yu, W., Zhang, Z., & Jiang, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.167 · Cited by 31
Three Bricks to Consolidate Watermarks for Large Language Models
Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374576 · Cited by 168
Mitigating Entity-Level Hallucination in Large Language Models
Su, W., Tang, Y., Ai, Q., Wang, C., Wu, Z., & Liu, Y. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698403 · Cited by 65
DynamicRetriever: A Pre-trained Model-based IR System Without an Explicit Index
Zhou, Y., Yao, J., Dou, Z., Wu, L., & Wen, J.-R. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1373-9 · Cited by 52
ConTextual Masked Auto-Encoder for Dense Passage Retrieval
Wu, D. D., Ma, G., Lin, M. S., Lin, Z., Wang, Z., & Hu, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25598 · Cited by 63
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks
Wu, Y., Zhao, Y., Hu, B., Minervini, P., Stenetorp, P., & Riedel, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.346 · Cited by 34
Embedding-based Zero-shot Retrieval through Query GenerationPreprint
Liang, D., Xu, P., Shakeri, S., dos Santos, C. N., Nallapati, R. M., Huang, Z., & Xiang, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10270 · Cited by 63
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 39
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.699 · Cited by 38
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Zhou, K., Jurafsky, D., & Hashimoto, T. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.335 · Cited by 38
GDPR and Large Language Models: Technical and Legal Obstacles
Feretzakis, G., Vagena, E., Kalodanis, K., Peristera, P., Kalles, D., & Anastasiou, A. (2025) · Future Internet · doi:10.3390/fi17040151 · Cited by 38
Testing Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 63
Learning Cross-Lingual IR from an English Retriever
Li, Y., Franz, M., Sultan, M. A., Iyer, B., Lee, Y., & Sil, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.329 · Cited by 46
Automating Question Generation From Educational Text
Bhowmick, A. K., Jagmohan, A., Vempaty, A., Dey, P., Hall, L., Hartman, J., Kokku, R., & Maheshwari, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47994-6_38 · Cited by 46
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Saad-Falcon, J., Khattab, O., Santhanam, K., Florian, R., Franz, M., Roukos, S., Sil, A., Sultan, M., & Potts, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.693 · Cited by 27
Thanasi-Boçe, M., & Hoxha, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12775-z · Cited by 46
Phrase Retrieval Learns Passage Retrieval, Too
Lee, J., Wettig, A., & Chen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.297 · Cited by 58
ParsiNLU : A Suite of Language Understanding Challenges for Persian
Khashabi, D., Cohan, A., Shakeri, S., Hosseini, P., Pezeshkpour, P., Alikhani, M., Aminnaseri, M., Bitaab, M., Brahman, F., Ghazarian, S., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00419 · Cited by 95
CoLT5: Faster Long-Range Transformers with Conditional Computation
Ainslie, J., Lei, T., de Jong, M., Ontanon, S., Brahma, S., Zemlyanskiy, Y., Uthus, D., Guo, M., Lee-Thorp, J., Tay, Y., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.309 · Cited by 31
Multiview Identifiers Enhanced Generative Retrieval
Li, Y., Yang, N., Wang, L., Wei, F., & Li, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.366 · Cited by 37
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.61 · Cited by 37
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2022) · arXiv · doi:10.48550/arxiv.2208.02814 · Cited by 37
Open Domain Question Answering with A Unified Knowledge Interface
Ma, K., Cheng, H., Liu, X., Nyberg, E., & Gao, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.113 · Cited by 30
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 59
Complex Factoid Question Answering with a Free-Text Knowledge Graph
Zhao, C., Xiong, C., Qian, X., & Boyd-Graber, J. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380197 · Cited by 48
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
Li, D., Hu, B., Chen, Q., Peng, W., & Wang, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.111 · Cited by 46
GooAQ: Open Question Answering with Diverse Answer Types
Khashabi, D., Ng, A., Khot, T., Sabharwal, A., Hajishirzi, H., & Callison-Burch, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.38 · Cited by 31
AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry
Katsis, Y., Chemmengath, S., Kumar, V., Bharadwaj, S., Canim, M., Glass, M., Gliozzo, A., Pan, F., Sen, J., Sankaranarayanan, K., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.34 · Cited by 43
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Lyu, C., Shang, L., Graham, Y., Foster, J., Jiang, X., & Liu, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.340 · Cited by 39
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks
Asai, A., Gardner, M., & Hajishirzi, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.162 · Cited by 71
Read before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · arXiv · doi:10.48550/arxiv.2203.00343 · Cited by 37
R2-D2: A Modular Baseline for Open-Domain Question Answering
Fajcik, M., Docekal, M., Ondrej, K., & Smrz, P. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.73 · Cited by 36
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 36
“The problems in this dataset are very hard as they come from mathematics competitions written in L A TEX. • NQ (Kwiatkowski et al., 2019) were collected from real Google search queries and the answers are one or multiple spans in Wikipedia articles identified by human annotators.”
BackgroundInfluentialLee, Y., Son, K., Kim, T. S., Kim, J., Chung, J. J. Y., Adar, E., & Kim, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3662681 · Cited by 36
“We chose TruthfulQA because LLMs may also have been trained on these false beliefs so the model may be induced to answer incorrectly (i.e., produce hallucinations)—allowing us to observe more diverse types of inconsistencies compared to adopting other QA benchmarks [14, 24, 29, 34].”
MethodLearning to break: Knowledge-enhanced reasoning in multi-agent debate system
Wang, H., Du, X., Yu, W., Chen, Q., Zhu, K., Chu, Z., Yan, L., & Guan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2024.129063 · Cited by 36
“…et al., 2019), multi-hop reasoning: HotpotQA (Yang et al., 2018) and 2WikiMQA (Ho et al., 2020), fact verification: FEVER (Thorne et al., 2018 NQ (Kwiatkowski et al., 2019) Open Single-Hop Reasoning Generative 500 HotpotQA (Yang et al., 2018) Open Multi-Hop Reasoning Generative 500 2WikiMQA (Ho…”
MethodInfluentialmt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Katsis, Y., Rosenthal, S., Fadnis, K., Gunasekara, C., Lee, Y., Popa, L., Shah, V., Zhu, H., Contractor, D., & Danilevsky, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.19 · Cited by 111
Context-Aware Answer Extraction in Question Answering
Seonwoo, Y., Kim, J., Ha, J.-W., & Oh, A. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.189 · Cited by 42
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
Liu, D., Gong, Y., Fu, J., Yan, Y., Chen, J., Lv, J., Duan, N., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.467 · Cited by 44
D2S: Document-to-Slide Generation Via Query-Based Text Summarization
Sun, E., Hou, Y., Wang, D., Zhang, Y., & Wang, N. X. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.111 · Cited by 85
Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
Asai, A., & Choi, E. (2021) · doi:10.18653/v1/2021.acl-long.118 · Cited by 67
A review of public datasets in question answering research
Cambazoglu, B. B., Sanderson, M., Scholer, F., & Croft, B. W. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483389 · Cited by 36
One Question Answering Model for Many Languages with Cross-lingual Dense Passage RetrievalPreprint
Asai, A., Yu, X., Kasai, J., & Hajishirzi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11976 · Cited by 74
DAPIE: Interactive Step-by-Step Explanatory Dialogues to Answer Children’s Why and How Questions
Lee, Y., Kim, T. S., Kim, S., Yun, Y., & Kim, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581369 · Cited by 35
Neural Retrievers are Biased Towards LLM-Generated Content
Dai, S., Zhou, Y., Pang, L., Liu, W., Hu, X., Liu, Y., Zhang, X., Wang, G., & Xu, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671882 · Cited by 35
“NQ320K 2 Kwiatkowski et al. (2019) is based on the Natural Questions (NQ) dataset from Google, where the documents are gathered from Wikipedia pages, and the queries are natural language questions.”
Learning to retrieve passages without supervision
Ram, O., Shachaf, G., Levy, O., Berant, J., & Globerson, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.193 · Cited by 35
Varshney, N., Mishra, S., & Baral, C. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.158 · Cited by 32
Xiao, S., Liu, Z., Han, W., Zhang, J., Lian, D., Gong, Y., Chen, Q., Yang, F., Sun, H., Shao, Y., et al. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531799 · Cited by 49
Questions Are All You Need to Train a Dense Passage Retriever
Sachan, D. S., Lewis, M., Yogatama, D., Zettlemoyer, L., Pineau, J., & Zaheer, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00564 · Cited by 76
C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
Yang, E., Nair, S., Chandradevan, R., Iglesias-Flores, R., & Oard, D. W. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531886 · Cited by 44
An Analysis of Negation in Natural Language Understanding Corpora
Hossain, M. M., Chinnappa, D., & Blanco, E. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.81 · Cited by 67
Resources for Turkish natural language processing: A critical survey
Çöltekin, Ç., Doğruöz, A. S., & Çetinoğlu, Ö. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09605-4 · Cited by 49
A Thorough Examination on Zero-shot Dense Retrieval
Ren, R., Qu, Y., Liu, J., Zhao, X., Wu, Q., Ding, Y., Wu, H. Y., Wang, H., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1057 · Cited by 58
Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators
Liang, C., Deng, Y., Bian, Y., Qin, Z., Wu, B., Chua, T., & Wong, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.390 · Cited by 93
SLM: Bridge the Thin Gap Between Speech and Text Foundation Models
Wang, M., Han, W., Shafran, I., Wu, Z., Chiu, C., Cao, Y. J., Chen, N., Zhang, Y., Soltau, H., Rubenstein, P. K., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389703 · Cited by 102
MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA
Yu, L., Chen, Q., Zhou, J., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29916 · Cited by 144
RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice
Yang, R., Fu, M. C., Tantithamthavorn, C. K., Arora, C., Vandenhurk, L., & Chua, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112436 · Cited by 47
Domain-agnostic Question-Answering with Adversarial Training
Lee, S., Kim, D., & Park, J. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5826 · Cited by 50
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
Longpre, S., Lu, Y., Tu, Z., & DuBois, C. (2019) · Proceedings of the 2nd Workshop on Machine Reading for Question Answering · doi:10.18653/v1/d19-5829 · Cited by 30
Representations for Question Answering from Documents with Tables and Text
Zayats, V., Toutanova, K., & Ostendorf, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.253 · Cited by 45
Self-citationQuestion Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 46
Semantic matching in machine reading comprehension: An empirical study
Liu, Q., Mao, R., Geng, X., & Cambria, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103145 · Cited by 41
Inquisitive Question Generation for High Level Text Comprehension
Ko, W.-J., Chen, T.-Y., Huang, Y., Durrett, G., & Li, J. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.530 · Cited by 61
Exploring Neural Models for Query-Focused Summarization
Vig, J., Fabbri, A. R., Kryściński, W., Wu, C.-S., & Liu, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.109 · Cited by 69
ArabicaQA: A Comprehensive Dataset for Arabic Question Answering
Abdallah, A., Kasem, M. S., Abdalla, M. I., Mahmoud, M., Elkasaby, M., Elbendary, Y., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657889 · Cited by 67
Xu, K., Zhang, K., Li, J., Huang, W. D., & Wang, Y. (2024) · Electronics · doi:10.3390/electronics14010047 · Cited by 51
Interactive Language Learning by Question Answering
Yuan, X., Côté, M.-A., Fu, J., Lin, Z., Pal, C., Bengio, Y., & Trischler, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1280 · Cited by 45
BioMRC: A Dataset for Biomedical Machine Reading Comprehension
Pappas, D., Stavropoulos, P., Androutsopoulos, I., & McDonald, R. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.15 · Cited by 29
ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
Han, R., Ren, X., & Peng, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.436 · Cited by 71
LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval
Xu, C., Guo, D., Duan, N., & McAuley, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.281 · Cited by 53
Chandra, R., & Ranjan, M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273476 · Cited by 50
AI-based Question Answering Assistance for Analyzing Natural-language Requirements
Ezzini, S., Abualhaija, S., Arora, C., & Sabetzadeh, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00113 · Cited by 49
Aligning Large Language Models through Synthetic Feedback
Kim, S., Bae, S., Shin, J., Kang, S., Kwak, D., Yoo, K. M., & Seo, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.844 · Cited by 109
Can We Edit Multimodal Large Language Models?
Cheng, S., Tian, B., Liu, Q., Chen, X., Wang, Y., Chen, H., & Zhang, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.856 · Cited by 89
Is Retriever Merely an Approximator of Reader?Preprint
Yang, S., & Seo, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10999 · Cited by 51
A Memory Efficient Baseline for Open Domain Question AnsweringPreprint
Izacard, G., Petroni, F., Hosseini, L., Cao, N. D., Riedel, S., & Grave, É. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15156 · Cited by 47
Can AI Assistants Know What They Don't Know?
Cheng, Q., Sun, T., Liu, X., Zhang, W., Yin, Z., Li, S., Li, L., He, Z., Chen, K., & Qiu, X. (2024) · arXiv · doi:10.48550/arxiv.2401.13275 · Cited by 33
A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning
Chen, J., Zhang, R., Guo, J., de Rijke, M., Liu, Y., Fan, Y., & Cheng, X. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591631 · Cited by 32
“Existing methods for KILT typically contain a search component and a reader component [4, 22, 26, 41, 46].”
BackgroundMethodInfluentialWhy Machine Reading Comprehension Models Learn Shortcuts?
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.85 · Cited by 32
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32
Scalable and Effective Generative Information Retrieval
Zeng, H., Luo, C., Jin, B., Sarwar, S. M., Wei, T., & Zamani, H. (2024) · Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589334.3645477 · Cited by 32
Emerging trends: a gentle introduction to RAG
Church, K., Sun, J., Yue, R., Vickers, P., Saba, W., & Chandrasekar, R. (2024) · Natural Language Engineering · doi:10.1017/s1351324924000044 · Cited by 45
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Zhao, S., Shao, Y., Huang, Y., Song, J., Wan, C., Wan, C., & Ма, Л. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802824 · Cited by 49
Conversations with Documents: An Exploration of Document-Centered Assistance
ter Hoeve, M., Sim, R., Nouri, E., Fourney, A., de Rijke, M., & White, R. W. (2020) · Proceedings of the 2020 Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377971 · Cited by 25
Castelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.117 · Cited by 76
A Technical Question Answering System with Transfer Learning
Yu, W., Wu, L., Deng, Y., Mahindru, R., Zeng, Q., Güven, S., & Jiang, M. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.13 · Cited by 32
GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval
Möller, T., Risch, J., & Pietsch, M. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.4 · Cited by 36
Kim, G., Kim, H., Park, J., & Kang, J. (2021) · doi:10.18653/v1/2021.acl-long.478 · Cited by 45
Uniqorn: Unified question answering over RDF knowledge graphs and natural language text
Pramanik, S., Alabi, J. O., Roy, R. S., & Weikum, G. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100833 · Cited by 55
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.