Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 501–600 of 1,158
Pitfalls of Static Language Modelling.Preprint
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Giménez, M., d’Autume, C. D. M., Ruder, S., Yogatama, D., et al. (2021) · arXiv (Cornell University) · Cited by 41
MixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31
Generation-Augmented Retrieval for Open-domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2020) · arXiv · doi:10.48550/arxiv.2009.08553 · Cited by 31
Chen, H.-T., Zhang, M. J. Q., & Choi, E. (2022) · arXiv · doi:10.48550/arxiv.2210.13701 · Cited by 31
Chen, H.-T., Zhang, M., & Choi, E. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.146 · Cited by 31
Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization
Zamani, H., & Bendersky, M. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657923 · Cited by 31
Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
Li, Y., Liu, G., Wang, C., & Yang, Y. (2025) · ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49660.2025.10889013 · Cited by 15
Multilingual Transfer Learning for QA using Translation as Data Augmentation
Bornea, M. A., Pan, L., Rosenthal, S., Florian, R., & Sil, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17491 · Cited by 46
Aligning Generative Language Models with Human Values
Liu, R., Zhang, G., Feng, X., & Vosoughi, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.18 · Cited by 82
Lin, S.-C., Li, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00556 · Cited by 50
WebArena: A Realistic Web Environment for Building Autonomous Agents
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.13854 · Cited by 31
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.12022 · Cited by 30
“This approach has been used to create benchmarks of various capabilities, including basic reading comprehension (Rajpurkar et al., 2016; Kwiatkowski et al., 2019), assembling information across multiple contexts (Yang et al., 2018), and coreference resolution and numerical reasoning (Dua et al.,…”
Ke, W., Zheng, Y., Li, Y., Xu, H., Nie, D., Wang, P., & He, Y. (2025) · ACM Transactions on Information Systems · doi:10.1145/3768156 · Cited by 73
What Question Answering can Learn from Trivia Nerds
Boyd-Graber, J. L., & Börschinger, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.662 · Cited by 31
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
Riabi, A., Scialom, T., Keraron, R., Sagot, B., Seddah, D., & Staiano, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.562 · Cited by 67
Reader-Guided Passage Reranking for Open-Domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.29 · Cited by 28
External features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 35
Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
Kim, N., Pavlick, E., Ayan, B. K., & Ramachandran, D. (2021) · doi:10.18653/v1/2021.acl-long.304 · Cited by 35
MultiSpanQA: A Dataset for Multi-Span Question Answering
Li, H., Tomko, M., Vasardani, M., & Baldwin, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.90 · Cited by 72
EduQG: A Multi-Format Multiple-Choice Dataset for the Educational Domain
Hadifar, A., Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2023) · IEEE Access · doi:10.1109/access.2023.3248790 · Cited by 48
Strategic Deductive Reasoning in Large Language Models: A Dual-Agent Approach
Li, S., Zhou, X., Wu, Z., Long, Y., & Shen, Y. (2024) · IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) · doi:10.1109/icpics62053.2024.10797012 · Cited by 36
Quizbowl: The Case for Incremental Question AnsweringPreprint
Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 63
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 41
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training
Lei, Y., Ding, L., Cao, Y. H., Zan, C., Yates, A., & Tao, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.695 · Cited by 83
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 33
Generating Query Focused Summaries from Query-Free Resources
Xu, Y., & Lapata, M. (2021) · doi:10.18653/v1/2021.acl-long.475 · Cited by 40
ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities
Lerner, P. J., Ferret, O., Guinaudeau, C., Borgne, H. L., Besançon, R., Moreno, J. G., & Lovón-Melgarejo, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531753 · Cited by 109
Natural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 53
Nonparametric masked language modeling
Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-T., Hajishirzi, H., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.132 · Cited by 16
FIT-RAG: Black-Box RAG with Factual Information and Token Reduction
Mao, Y., Dong, X., Xu, W., Gao, Y., Wei, B., & Zhang, Y. (2025) · ACM Transactions on Information Systems · doi:10.1145/3676957 · Cited by 49
Church, K. W., & Kordoni, V. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000043 · Cited by 28
Zero-Shot Cross-lingual Semantic Parsing
Sherborne, T., & Lapata, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.285 · Cited by 30
Improving Biomedical Information Retrieval with Neural Retrievers
Luo, M., Mitra, A., Gokhale, T., & Baral, C. R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21352 · Cited by 50
Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models
Liu, J., Kang, Y., Tang, D., Song, K., Sun, C., Wang, X., Lu, W. P., & Liu, X. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560683 · Cited by 68
Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering
Ju, M., Yu, W., Zhao, T., Zhang, C., & Ye, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.13 · Cited by 8
FaVIQ: FAct Verification from Information-seeking Questions
Park, J., Min, S., Kang, J., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.354 · Cited by 49
Transformer-Based Models for Question Answering on COVID19Preprint
Ngai, H., Park, Y., Chen, J., & Parsapoor, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11432 · Cited by 33
Muppet: Massive Multi-task Representations with Pre-FinetuningPreprint
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11038 · Cited by 109
Xu, K., Zhang, K., Li, J., Huang, W., & Wang, Y. (2024) · doi:10.20944/preprints202411.1648.v1 · Cited by 28
Learning to Rank in Generative Retrieval
Li, Y., Yang, N., Wang, L., Wei, F., & Li, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28717 · Cited by 27
Performance analysis of large language models in the domain of legal argument mining
Zubaer, A. A., Granitzer, M., & Mitrović, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1278796 · Cited by 60
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Ravichander, A., Dalmia, S., Ryskina, M., Metze, F., Hovy, E. H., & Black, A. W. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.259 · Cited by 49
Thorne, J., Yazdani, M., Saeidi, M., Silvestri, F., Riedel, S., & Halevy, A. Y. (2021) · doi:10.18653/v1/2021.acl-long.241 · Cited by 44
On the Diversity and Limits of Human Explanations
Tan, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.158 · Cited by 44
Exemplars-Guided Empathetic Response Generation Controlled by the Elements of Human Communication
Majumder, N., Ghosal, D., Hazarika, D., Gelbukh, A. F., Mihalcea, R. F., & Poria, S. (2022) · IEEE Access · doi:10.1109/access.2022.3193159 · Cited by 37
Contrastive Domain Adaptation for Question Answering using Limited Text Corpora
Yue, Z., Kratzwald, B., & Feuerriegel, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.754 · Cited by 29
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Zhou, K., Gong, Y., Liu, X., Zhao, W. X., Shen, Y., Dong, A., Lü, J., Majumder, R., Wen, J.-R., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.56 · Cited by 51
Generative Retrieval via Term Set Generation
Zhang, P., Liu, Z., Zhou, Y., Dou, Z., Liu, F., & Cao, Z. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657797 · Cited by 66
Contextualized Representations Using Textual Encyclopedic KnowledgePreprint
Joshi, M., Lee, K., Luan, Y., & Toutanova, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12006 · Cited by 34
Self-citationLearning to Rank in Generative Retrieval
Li, Y., Yang, N., Wang, L., Wei, F., & Li, W. (2023) · arXiv · doi:10.48550/arxiv.2306.15222 · Cited by 27
“, 2020) setting on two widely-used open-domain QA datasets: NQ (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al.”
MethodAre Large Language Models Good at Utility Judgments?
Zhang, H., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657784 · Cited by 55
Huang, W., Zhou, G., Lapata, M., Vougiouklis, P., Montella, S., & Pan, J. Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113648 · Cited by 44
Graph-Based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey
Zhu, Z., Huang, T., Wang, K., Ye, J., Chen, X., & Luo, S. (2026) · ACM Computing Surveys · doi:10.1145/3795880 · Cited by 60
QAConv: Question Answering on Informative Conversations
Wu, C.-S., Madotto, A., Liu, W., Fung, P., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.370 · Cited by 42
Improving Evidence Retrieval for Automated Explainable Fact-Checking
Samarinas, C., Hsu, W., & Lee, M. L. (2021) · doi:10.18653/v1/2021.naacl-demos.10 · Cited by 51
Kaushik, D., Kiela, D., Lipton, Z. C., & Yih, W.-T. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.517 · Cited by 40
Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering
Li, A. H., Ng, P., Xu, P., Zhu, H., Wang, Z., & Xiang, B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.315 · Cited by 12
Knowing More About Questions Can Help: Improving Calibration in Question Answering
Zhang, S., Gong, C., & Choi, E. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.172 · Cited by 60
xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering
Yang, N., Wei, F., Jiao, B., Jiang, D., & Yang, L. (2021) · doi:10.18653/v1/2021.acl-long.477 · Cited by 27
Challenges in Generalization in Open Domain Question Answering
Liu, L., Lewis, P., Riedel, S., & Stenetorp, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.155 · Cited by 53
ICDAR 2021 Competition on Document Visual Question Answering
Tito, R., Mathew, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86337-1_42 · Cited by 38
Dense Hierarchical Retrieval for Open-domain Question Answering
Liu, Y., Hashimoto, K., Zhou, Y., Yavuz, S., Xiong, C., & Yu, P. S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.19 · Cited by 43
A Free Format Legal Question Answering System
Khazaeli, S., Punuru, J. R., Morris, C., Sharma, S., Staub, B., Cole, M., Chiu-Webster, S., & Sakalley, D. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.11 · Cited by 29
ABLE: Meta-Path Prediction in Heterogeneous Information Networks
Huang, C., Fang, Y., Lin, X., Cao, X., & Zhang, W. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3494558 · Cited by 29
PROD: Progressive Distillation for Dense Retrieval
Lin, Z., Gong, Y., Liu, X., Zhang, H., Lin, C., Dong, A., Jiao, J., Lu, J., Jiang, D., Majumder, R., et al. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583421 · Cited by 45
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference
de Jong, M., Zemlyanskiy, Y., Ainslie, J., FitzGerald, N., Sanghai, S. K., Sha, F., & Cohen, W. W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.732 · Cited by 56
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
Chuang, Y.-S., Fang, W., Li, S.-W. D., Yih, W.-T., & Glass, J. R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.768 · Cited by 65
SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
Shon, S., Arora, S., Lin, C.-J., Pasad, A., Wu, F. F., Sharma, R. S., Wu, W.-L., Lee, H.-Y., Livescu, K., & Watanabe, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.496 · Cited by 76
MaXM: Towards Multilingual Visual Question Answering
Changpinyo, S., Xue, L., Yarom, M., Thapliyal, A. V., Szpektor, I., Amelot, J., Chen, X., & Soricut, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.176 · Cited by 76
Li, X., Zhou, Y., & Dou, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28714 · Cited by 63
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Min, S., Lee, K., Chang, M.-W., Toutanova, K., & Hajishirzi, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.560 · Cited by 58
Self-citationSynthetic QA Corpora Generation with Roundtrip ConsistencyPreprint
Alberti, C., Andor, D., Pitler, E., Devlin, J., & Collins, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05416 · Cited by 45
Self-citationReason first, then respond: Modular Generation for Knowledge-infused Dialogue
Adolphs, L., Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.527 · Cited by 25
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments
Zheng, Y., Fu, D., Hu, X., Cai, X., Ye, L., Lu, P., & Liu, P. (2025) · arXiv · doi:10.48550/arxiv.2504.03160 · Cited by 25
“For in-domain evaluation, we include the dev sets of NQ (Kwiatkowski et To ensure a fair and balanced evaluation, we randomly sample 512 examples from the development sets of NQ, TQ, HotpotQA, 2Wiki, MuSiQue, and PopQA as well as all 125 samples from Bamboogle’s development set.”
HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems
Tan, J., Dou, Z., Wang, W., Wang, M., Chen, W., & Wen, J.-R. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714546 · Cited by 73
Transforming Wikipedia Into Augmented Data for Query-Focused Summarization
Zhu, H., Dong, L., Wei, F., Qin, B., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3171963 · Cited by 38
Exploring Automated Question Answering Methods for Teaching Assistance
Zylich, B., Viola, A., Toggerson, B., Al-Hariri, L. A., & Lan, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52237-7_49 · Cited by 38
Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance
Su, D., Yu, T., & Fung, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.275 · Cited by 36
Reinforced History Backtracking for Conversational Question Answering
Qiu, M., Huang, X., Chen, C., Ji, F., Qu, C., Wei, W., Huang, J., & Zhang⋆, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17617 · Cited by 35
Peng, W., Hu, Y., Yu, J., Xing, L., & Xie, Y. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107364 · Cited by 30
Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations
Xin, J., Xiong, C., Srinivasan, A., Sharma, A., Jose, D., & Bennett, P. N. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.316 · Cited by 51
E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning
Chen, J., Xu, R., Fu, Z., Shi, W., Li, Z., Zhang, X., Sun, C., Li, L., Xiao, Y., & Zhou, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.311 · Cited by 61
Reddy, R. G., Rui, X., Li, M., Lin, X., Wen, H., Cho, J., Huang, L., Bansal, M., Sil, A., Chang, S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21370 · Cited by 38
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
Wang, Z., Cuenca, G., Zhou, S., Xu, F. F., & Neubig, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.20 · Cited by 65
Casanueva, I., Vulić, I., Spithourakis, G., & Budzianowski, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.154 · Cited by 52
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A. P., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00583 · Cited by 67
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Pradeep, R., Thakur, N., Sharifymoghaddam, S., Zhang, E., Nguyen, R., Campos, D., Craswell, N., & Lin, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88708-6_9 · Cited by 63
Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Education
Zhao, C., Agrawal, G., Zhang, F., Kumarage, T., Tan, Z., Deng, Y., Chen, Y.-C., & Liu, H. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401740 · Cited by 41
Teaching Machine Comprehension with Compositional Explanations
Ye, Q., Huang, X., Boschee, E., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.145 · Cited by 37
Cluster-Former: Clustering-based Sparse Transformer for Question Answering
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.346 · Cited by 37
A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models
Mokrii, I., Boytsov, L., & Braslavski, P. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463093 · Cited by 35
Sequence tagging for biomedical extractive question answering
Yoon, W., Jackson, R., Lagerberg, A., & Kang, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac397 · Cited by 29
Iyer, S., Min, S., Mehdad, Y., & Yih, W.-T. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.100 · Cited by 23
Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLP
Chen, A., Gudipati, P., Longpre, S., Ling, X., & Singh, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.345 · Cited by 20
Modeling Multi-hop Question Answering as Single Sequence Prediction
Yavuz, S., Hashimoto, K., Zhou, Y., Keskar, N. S., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.69 · Cited by 28
Dealing with textual noise for robust and effective BERT re-ranking
Chen, X., He, B., Hui, K., Sun, L., & Sun, Y. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103135 · Cited by 27
Zhu, H., Peng, H. M., Lyu, Z., Hou, L., Li, J., & Xiao, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119369 · Cited by 41
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 53
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.