Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 1,101–1,158 of 1,158
Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
Mekonnen, K. A., Tang, Y., & de Rijke, M. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730023 · Cited by 3
“We conduct our experiments using two widely recognized benchmarks: the MS MARCO Document Ranking dataset [3]andthe NaturalQuestions(NQ) dataset 2 [23].”
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7
Probabilistic Consensus through Ensemble Validation: A Framework for LLM Reliability
Naik, N. (2024) · arXiv · doi:10.48550/arxiv.2411.06535 · Cited by 3
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Liu, A., Green, J., Patel, K., Meng, R., Su, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.06600 · Cited by 3
“BrowseComp-Plus retains the challenging nature of the original BrowseComp while providing a more controlled and transparent experimental setup similar to early pivotal evaluation benchmarks like Natural Question (NQ) [27] and HotpotQA [30].”
Attribute or Abstain: Large Language Models as Long Document Assistants
Buchmann, J., Liu, X., & Gurevych, I. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.463 · Cited by 3
Answer Bubbles: Information Exposure in AI-Mediated Search
Huang, M., Goyal, A., Saha, K., & Chandrasekharan, E. (2026) · Findings of the Association for Computational Linguistics: EMNLP 2026 · doi:10.18653/2026.findings-emnlp.8 · Cited by 3
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Aryabumi, V., Su, Y., Ma, R., Morisot, A., Zhang, I., Locatelli, A., Fadaee, M., Üstün, A., & Hooker, S. (2024) · arXiv · doi:10.48550/arxiv.2408.10914 · Cited by 3
“We include Natural Questions Open [Kwiatkowski et al., 2019], and TriviaQA [Joshi et al., 2017] as the datasets.”
BackgroundIs It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP
Goldman, O., Jacovi, A., Slobodkin, A., Maimon, A., Dagan, I., & Tsarfaty, R. (2024) · arXiv · doi:10.48550/arxiv.2407.00402 · Cited by 3
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Akhtar, M., Reuel, A., Soni, P., Ahuja, S., Ammanamanchi, P. S., Rawal, R., Zouhar, V., Yadav, S., Whitehouse, C., Ki, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.16763 · Cited by 3
Instruction-tuned Language Models are Better Knowledge Learners
Jiang, Z., Sun, Z., Shi, W., Rodriguez, P., Zhou, C., Neubig, G., Lin, X. V., Yih, W.-T., & Iyer, S. (2024) · arXiv · doi:10.48550/arxiv.2402.12847 · Cited by 3
MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory
Modarressi, A., Köksal, A., Imani, A., Fayyaz, M., & Schütze, H. (2024) · arXiv · doi:10.48550/arxiv.2404.11672 · Cited by 3
Think before you speak: Training Language Models With Pause Tokens
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., & Nagarajan, V. (2023) · arXiv · doi:10.48550/arxiv.2310.02226 · Cited by 2
“We consider nine varied downstream tasks: (a) reasoning (GSM8k (Cobbe et al., 2021)), (b) ex-tractive question answering (SQuAD (Rajpurkar et al., 2016), CoQA (Reddy et al., 2019)), (c) general understanding (CommonSenseQA (Talmor et al., 2019), PhysicalIQA (Bisk et al., 2020)), (d) (Berant et al., 2013), Natural Ques…”
BackgroundQuantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2
“…al., 2011), GSM8k (Cobbe et al., 2021), Hellaswag (Zellers et al., 2019), HumanEval (Chen et al., 2021), MATH (Hendrycks et al., 2021), MMLU (Hendrycks et al., 2020), Natural Questions (Kwiatkowski et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and TriviaQA (Joshi et al., 2017).”
BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Sharifymoghaddam, S., Liu, A., Green, J., Patel, K., Meng, R., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1023 · Cited by 2
“Traditional benchmarks such as NaturalQues-tions (Kwiatkowski et al., 2019) and Trivi-aQA (Joshi et al., 2017) have significantly contributed to evaluating retrieval and retrieval-augmented generation systems (Lewis et al., 2020; Karpukhin et al., 2020; Lin et al., 2024).”
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00656 · Cited by 2
Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.16153 · Cited by 2
“…(Press et al., 2022), and PopQA (Mallen et al., 2022) are selected as OOD datasets due to their distinct question styles and information distributions, while NQ (Kwiatkowski et al., 2019), TQ (Joshi et al., 2017), HotpotQA (Yang et al., 2018), and 2Wiki (Ho et al., 2020) are used for training.”
InfluentialSelf-citationEliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Qiu, Y., Embar, V., Zhang, Y., Jaitly, N., Cohen, S. B., & Han, B. (2025) · arXiv · doi:10.48550/arxiv.2501.08248 · Cited by 2
“LOFT tests retrieval, single-and multi-hop question answering, and reasoning using NaturalQuestions (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), and MuSiQue (Trivedi et al., 2022).”
InfluentialRecent Advances in Generative Information Retrieval
Tang, Y., Zhang, R., Ren, Z., Guo, J., & de Rijke, M. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-56069-9_48 · Cited by 2
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
“Our evaluations cover factual question answering (NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017)), multi-hop question answering (HotpotQA (Yang et al., 2018)), scientific and common sense world knowledge (MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019),…”
MiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019).”
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2
“We use Natural Questions (NQ; Kwiatkowski et al., 2019), TriviaQA (TQA; Joshi et al., 2017), HotpotQA (Yang et al., 2018), and PopQA (Mallen et al., 2023).”
MethodSimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1
“SimpleQA (Wei et al., 2024a) was created as a challenging factuality benchmark in response to older short-answer factuality benchmarks like TriviaQA (Joshi et al., 2017) and Natural Questions (Kwiatkowski et al., 2019) becoming saturated.”
Mamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“…( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐵 , U ) C = contract ( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐶 , U ) , Intermediate output Y ′ is obtained by the residual function 𝜙 , Y ′ ← 𝜙 ( Y 𝑡 , Z ) , where 𝜙 ( Y 𝑡 , Z ) : = Y ⊙ SiLU ( Z 𝑡 ) in Finally, the layer output O ∈ R is computed via the following down projections:”
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1
Nested Learning: The Illusion of Deep Learning Architectures
Behrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2512.24695 · Cited by 37
Gated Delta Networks: Improving Mamba2 with Delta Rule
Yang, S., Kautz, J., & Hatamizadeh, A. (2024) · arXiv · doi:10.48550/arxiv.2412.06464 · Cited by 1
ACON: Optimizing Context Compression for Long-horizon LLM Agents
Kang, M., Chen, W.-N., Han, D., Inan, H. A., Wutschitz, L., Chen, Y., Sim, R., & Rajmohan, S. (2025) · arXiv · doi:10.48550/arxiv.2510.00615 · Cited by 1
Chen, X., He, B., Lin, H., Han, X., Wang, T., Cao, B., Sun, L., & Sun, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.10496 · Cited by 1
SEDM: Selective Self-Evolving Distributed Memory for Agents
Liu, Y., Min, A., Wang, Y., Yao, Y., Han, Y., Zhang, R., He, J., Li, Z., Qi, Y., Xu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.09498 · Cited by 20
“FEVER is a fact-checking dataset that provides human-written claims about Wikipedia entities, each labeled as Supported , Refuted , or NotEnoughInfo .”
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“The suite includes SWDE [46] for structured relation extraction from HTML, FDA [47] for key-value retrieval from PDFs, and question-answering datasets including SQuAD [48], TriviaQA [49], DROP [50], and Natural Questions [51].”
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“To gauge the ability to recall in-context information, we follow Arora et al. [4] and Yang et al. [111] and evaluate models on SWDE [65], SQUAD [82], FDA [3], TQA [55], NQ [58] and DROP [29].”
Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., & Zharov, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.21433
“A recent exception, MEM1 [41], explores dynamic state management for multi-hop QA [37, 13, 12] and web navigation [38] tasks.”
Gopal, A. M., Pirbadian, A., Carlson, K. D., Lewis, M. A., & Tapson, J. (2026) · arXiv
“Natural Questions [35] admits the ablation: NQ’s gold long-answer spans range from ∼ 50 to 5 , 000+ tokens after HTML stripping (Appendix F).”
Cao, S., Zou, K., & Wang, L. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1707
“Specifically, we include Natural Questions (Kwiatkowski et al., 2019) and Hot-potQA (Yang et al., 2018) in the retrieval-augmented generation setup, single-document QA from InfiniteBench (Zhang et al., 2024b) and Nar-rativeQA (Koˇciský et al., 2018), as well as single-document summarization using InfiniteBench.”
Continual Learning via Sparse Memory Finetuning
Lin, J., Zettlemoyer, L., Ghosh, G., Yih, W.-T., Markosyan, A., Berges, V.-P., & Oğuz, B. (2025) · arXiv · doi:10.48550/arxiv.2510.15103 · Cited by 33
“Using the set of indices accessed on a particular set that we want to preserve performance on (NaturalQuestions) leads to similar learning and forgetting performance as using DCLM batches as a representative sample of pretraining data.”
InfluentialFound in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“Our experiments demonstrate improvements over standard model generation by up to 15 percentage point on NaturalQuestion dataset (Kwiatkowski et al., 2019).”
InfluentialMEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
Zhou, Z., Qu, A., Wu, Z., Kim, S., Prakash, A., Rus, D., Zhao, J., Low, B. K. H., & Liang, P. P. (2025) · arXiv · doi:10.48550/arxiv.2506.15841 · Cited by 218
“The first environment is question answering with retrieval-augmented generation (RAG) [26, 59], where the agent must answer queries by retrieving relevant information from an external knowledge store (either a database or an online search engine).”
InfluentialATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…(Ma et al., 2025)); (b) language understanding (race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019) and wino-grande (Sakaguchi et al., 2021)); (c) professional knowledge (e.g., MMLU (Hendrycks et al., 2021a), CMMLU (Li et…”
InfluentialWideSearch: Benchmarking Agentic Broad Info-Seeking
Wong, R., Wang, J., Zhao, J., Chen, L., Gao, Y., Zhang, L., Zhou, X., Wang, Z., Xiang, K., Zhang, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07999
“Early benchmarks such as Natural Questions [11] and TriviaQA [10] established a foundation for question answering, but often tested information that could be retrieved with a single query or was already contained within a model’s parametric knowledge.”
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
TaSR-RAG: Taxonomy-guided Structured Reasoning for Retrieval-Augmented Generation
Sun, J., Xie, Y., Shi, J., Wang, S., & Han, J. (2026) · arXiv · doi:10.48550/arxiv.2603.09341
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
“These consisted of commonly used datasets focused on: entities (Min et al., 2023; Zhao et al., 2024), QA (Yang et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017), and complex reasoning (Su et al., 2024).”
C-SEO Bench: Does Conversational SEO Work?
Puerto, H., Gubri, M., Green, T., Oh, S. J., & Yun, S. (2025) · arXiv · doi:10.48550/arxiv.2506.11097
Mitigating Context Interference for Reliable and Efficient Search Agents
Xue, B., Wu, B., Qiao, S., Wang, S., Wang, R., Du, Y., Wang, H., Pan, J. Z., Yilmaz, E., Wong, K.-F., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.160
“Single-hop QA includes: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023).”
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Tian, C., Wang, J., Zhao, Q., Chen, K., Liu, J., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2025) · arXiv · doi:10.48550/arxiv.2507.17634
Efficient Benchmarking (of Language Models)
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L., Shnarch, E., Slonim, N., Shmueli-Scheuer, M., & Choshen, L. (2023) · arXiv · doi:10.48550/arxiv.2308.11696
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · arXiv · doi:10.48550/arxiv.2105.03011
“Unlike Q ASPER , Natural Questions and TyDiQA questions are not grounded in any contexts, and the associated documents are linked to the questions after they are written.”
BackgroundMethodInfluentialEliciting In-context Retrieval and Reasoning for Long-context Large Language Models
Qiu, Y., Embar, V. R., Zhang, Y., Jaitly, N., Cohen, S. B., & Han, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.165
MatKV: Trading Compute for Flash Storage in LLM Inference
Shin, K.-W., Park, J. H., Oh, M., Jo, Y., Do, J., & Lee, S.-W. (2026) · 2026 IEEE 42nd International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00071
“…easily contain thousands of tokens per query, the KV cache size can quickly grow to several GBs for a single sequence, leading to 60%- 15.56 11.17 Trivia QA [18] 18.16 4.05 Google NQ [19] 10.09 5.77 HotpotQA NQ [20] 23.11 3.53 80% of memory being wasted due to fragmentation and over-allocation [1].”
InfluentialFreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Thakur, N., Lin, J., Havens, S., Carbin, M., Khattab, O., & Drozdov, A. (2025) · arXiv · doi:10.48550/arxiv.2504.13128
Deletion Isn't Enough: Auditing RAG for Selective Forgetting
Tavakoli, L., & Sanderson, M. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808545
Memory-augmented agent training for business document understanding
Liu, J., Zeng, Y., Højmark-Bertelsen, M., Gadeberg, M. N., Wang, H., & Wu, Q. (2024) · arXiv · doi:10.48550/arxiv.2412.15274
Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
Wang, R., Wang, H., Xue, B., Li, Y., Pang, J., Liu, S., Chen, Y., Qiu, J., Wong, D. F., Chen, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.24377
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.