Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 601–700 of 1,158
Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU
Koto, F., Aisyah, N., Li, H., & Baldwin, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.760 · Cited by 101
A Survey on Knowledge Editing of Neural Networks
Mazzia, V., Pedrani, A., Caciolai, A., Rottmann, K., & Bernardi, D. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3498935 · Cited by 52
Crossing Variational Autoencoders for Answer Retrieval
Yu, W., Wu, L., Zeng, Q., Tao, S., Deng, Y., & Jiang, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.498 · Cited by 25
Self-Supervised Test-Time Learning for Reading Comprehension
Banerjee, P., Gokhale, T., & Baral, C. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.95 · Cited by 42
Baradaran, R., & Amirkhani, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.095 · Cited by 25
Block-Skim: Efficient Question Answering for Transformer
Guan, Y., Li, Z., Lin, Z., Zhu, Y., Leng, J., & Guo, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21316 · Cited by 35
Conditional Generation with a Question-Answering Blueprint
Narayan, S., Maynez, J., Amplayo, R. K., Ganchev, K., Louis, A., Huot, F., Sandholm, A., Das, D., & Lapata, M. (2022) · arXiv · doi:10.48550/arxiv.2207.00397 · Cited by 24
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
Jia, R., Lewis, M., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.59 · Cited by 37
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.357 · Cited by 51
MGRC: An End-to-End Multigranularity Reading Comprehension Model for Question Answering
Liu, Q., Geng, X., Huang, H., Qin, T., Lü, J., & Jiang, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107029 · Cited by 25
Unsupervised Question Clarity Prediction through Retrieved Item Coherency
Arabzadeh, N., Seifikar, M., & Clarke, C. L. A. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557719 · Cited by 24
FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answering
Rani, A., Tonmoy, S. M. T. I., Dalal, D., Gautam, S., Chakraborty, M., Chadha, A. R., Sheth, A., & Das, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.581 · Cited by 49
Maity, S., Deroy, A., & Sarkar, S. (2023) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3632754.3632755 · Cited by 44
Fluent Response Generation for Conversational Question Answering
Baheti, A., Ritter, A., & Small, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.19 · Cited by 32
MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering ModelsPreprint
Guo, M., Yang, Y., Cer, D. M., Shen, Q., & Constant, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02507 · Cited by 26
Zeng, H., Luo, C., & Zamani, H. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657746 · Cited by 22
“…Neural IR Models : With the emergence of large language models (LLMs) [14, 17, 40, 45, 52] and large-scale information retrieval datasets [4, 33], neural-based IR models have demonstrated superior results over the traditional lexical-matching models, such as BM25 [57]. o This model is…”
ResultRecent Advances in Generative Information Retrieval
Tang, Y., Zhang, R., Guo, J., & de Rijke, M. (2023) · Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region · doi:10.1145/3624918.3629547 · Cited by 22
A BP Neural Network-Assisted Smart Decision Method for Education Quality
Zhang, D., & Chen, L. (2023) · IEEE Access · doi:10.1109/access.2023.3294804 · Cited by 22
Search Augmented Instruction Learning
Luo, H., Zhang, T., Chuang, Y.-S., Gong, Y., Kim, Y., Wu, X., Meng, H. M. L., & Glass, J. R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.242 · Cited by 57
RuBQ 2.0: An Innovated Russian Question Answering Dataset
Rybin, I., Korablinov, V., Efimov, P., & Braslavski, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77385-4_32 · Cited by 23
Robust Retrieval Augmented Generation for Zero-shot Slot Filling
Glass, M., Rossiello, G., Chowdhury, M. F. M., & Gliozzo, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.148 · Cited by 38
Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
Paschoal, A. F. A., Pirozelli, P., Freire, V., Delgado, K. V., Peres, S. M., José, M. M., Cação, F. N., Oliveira, A. S., Brandão, A. A. F., Costa, A. H. R., et al. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482012 · Cited by 38
Deep learning-based approach for Arabic open domain question answering
Alsubhi, K., Jamal, A. T., & Alhothali, A. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.952 · Cited by 24
Wang, J., Jatowt, A., & Yoshikawa, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531734 · Cited by 52
Chain-of-Skills: A Configurable Model for Open-Domain Question Answering
Ma, K., Cheng, H., Zhang, Y., Liu, X., Nyberg, E., & Gao, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.89 · Cited by 47
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer
Jiang, Z., Gao, L., Wang, Z., Araki, J., Ding, H., Callan, J., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.149 · Cited by 62
Toward Best Practices for Training Multilingual Dense Retrieval Models
Zhang, X., Ogueji, K., Ma, X., & Lin, J. (2023) · ACM Transactions on Information Systems · doi:10.1145/3613447 · Cited by 64
Zhang, J., Li, Z., Das, K., Malin, B., & Kumar, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1032 · Cited by 94
QAmeleon : Multilingual QA with Only 5 Examples
Agrawal, P., Alberti, C., Huot, F., Maynez, J., Ma, J., Ruder, S., Ganchev, K., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00625 · Cited by 43
Self-citationLangTest: A comprehensive evaluation library for custom LLM and NLP models
Nazir, A., Chakravarthy, T. K., Cecchini, D., Khajuria, R., Sharma, P., Mirik, A. T., Kocaman, V., & Talby, D. (2024) · Software Impacts · doi:10.1016/j.simpa.2024.100619 · Cited by 31
Salemi, A., & Zamani, H. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657733 · Cited by 54
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 31
CLAPnq : Cohesive Long-form Answers from Passages in Natural Questions for RAG systems
Rosenthal, S., Sil, A., Florian, R., & Roukos, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00729 · Cited by 52
Luo, M., Hashimoto, K., Yavuz, S., Liu, Z., Baral, C. R., & Zhou, Y. (2022) · doi:10.18653/v1/2022.spanlp-1.2 · Cited by 27
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 34
Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellings
Sidiropoulos, G., & Kanoulas, E. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531818 · Cited by 54
Evaluating the generalisability of neural rumour verification models
Kochkina, E., Hossain, T., Logan, R. L., Arana‐Catania, M., Procter, R., Zubiaga, A., Singh, S., He, Y., & Liakata, M. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103116 · Cited by 43
A Discrete Hard EM Approach for Weakly Supervised Question AnsweringPreprint
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04849 · Cited by 30
Hindsight: Posterior-guided training of retrievers for improved open-ended generationPreprint
Paranjape, A., Khattab, O., Potts, C., Zaharia, M., & Manning, C. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07752 · Cited by 48
Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Question Answering Task
Ranaldi, L. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.35 · Cited by 22
DSI++: Updating Transformer Memory with New Documents
Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.510 · Cited by 21
Correctness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21
Recap: Retrieval-Augmented Audio Captioning
Ghosh, S., Kumar, S., Reddy Evuru, C. K., Duraiswami, R., & Manocha, D. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10448030 · Cited by 21
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 29
Answer Generation for Retrieval-based Question Answering Systems
Hsu, C.-C., Lind, E., Soldaini, L., & Moschitti, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.374 · Cited by 34
Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models
Cai, Y., Guo, J., Fan, Y., Ai, Q., Zhang, R., & Cheng, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557343 · Cited by 23
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary
Ram, O., Bezalel, L., Zicher, A., Belinkov, Y., Berant, J., & Globerson, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.140 · Cited by 53
Mitchell, E., Noh, J. J., Li, S., Armstrong, W., Agarwal, A., Liu, P., Finn, C., & Manning, C. D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.115 · Cited by 87
Knowledge-Augmented Language Model Verification
Baek, J., Jeong, S., Kang, M., Park, J. C., & Hwang, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.107 · Cited by 76
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
Ruder, S., Clark, J. H., Gutkin, A., Kale, M., Ma, M., Nicosia, M., Rijhwani, S., Riley, P., Sarr, J.-M., Wang, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.125 · Cited by 68
Evaluation of Question Answering Systems: Complexity of Judging a Natural Language
Farea, A., Yang, Z., Duong, K., Perera, N., & Emmert‐Streib, F. (2025) · ACM Computing Surveys · doi:10.1145/3744663 · Cited by 40
TuringAdvice: A Generative and Dynamic Evaluation of Language Use
Zellers, R., Holtzman, A., Clark, E., Qin, L., Farhadi, A., & Choi, Y. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.386 · Cited by 13
RoR: Read-over-Read for Long Document Machine Reading Comprehension
Zhao, J., Bao, J., Wang, Y., Zhou, Y., Wu, Y., He, X., & Zhou, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.160 · Cited by 22
Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
Jin, Z., von Kügelgen, J., Ni, J., Vaidhya, T., Kaushal, A., Sachan, M., & Schölkopf, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.748 · Cited by 40
You Only Need One Model for Open-domain Question Answering
Lee, H., Kedia, A., Lee, J., Paranjape, A., Manning, C. D., & Woo, K.-G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.198 · Cited by 39
Neural Passage Retrieval with Improved Negative ContrastPreprint
Lu, J. R., Ábrego, G. H., Ma, J., Ni, J., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12523 · Cited by 24
Cai, T., Tan, Z., Song, X., Sun, T., Jiang, J., Xu, Y., Zhang, Y., & Gu, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672065 · Cited by 44
Coordinated LLM multi-agent systems for collaborative question-answer generation
Saadaoui, S., & Alonso, E. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114627 · Cited by 32
Single-dataset Experts for Multi-dataset Question Answering
Friedman, D., Dodge, B., & Chen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.495 · Cited by 22
Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval
Lu, J. R., Ábrego, G. H., Ma, J., Ni, J., & Yang, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.492 · Cited by 34
Implicit Estimation of Paragraph Relevance From Eye Movements
Barz, M., Bhatti, O. S., & Sonntag, D. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.808507 · Cited by 21
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Malaviya, C., Shaw, P., Chang, M.-W., Lee, K., & Toutanova, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.784 · Cited by 50
Self-citationKILM: Knowledge Injection into Encoder-Decoder Language Models
Xu, Y., Namazifar, M., Hazarika, D., Padmakumar, A., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.275 · Cited by 37
Wei, Y., Su, Y., Ma, H., Yu, X., Lei, F., Zhang, Y., Zhao, J., & Liu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.100 · Cited by 71
Merging Generated and Retrieved Knowledge for Open-Domain QA
Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., & Wang, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.286 · Cited by 62
Are Large Language Models a Good Replacement of Taxonomies?
Sun, Y., Xin, H., Sun, K., Xu, Y., Yang, X., Dong, X., Tang, N., & Chen, L. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681973 · Cited by 35
Pal, K. K., Bajaj, A. P., Banerjee, P., Dutcher, A., Nakamura, M., Basque, Z. L., Gupta, H., Sawant, S. A., Anantheswaran, U., Shoshitaishvili, Y., et al. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00152 · Cited by 24
MIMOQA: Multimodal Input Multimodal Output Question Answering
Singh, H., Nasery, A., Mehta, D., Agarwal, A., Lamba, J., & Srinivasan, B. V. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.418 · Cited by 39
Learning to Generate Task-Specific Adapters from Task Description
Ye, Q., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.82 · Cited by 42
A survey on complex factual question answering
Zhang, L., Zhang, J., Ke, X., Li, H., Huang, X., Shao, Z., Cao, S., & Lv, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.12.003 · Cited by 35
Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
Chen, W., Verga, P., de Jong, M., Wieting, J., & Cohen, W. W. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.117 · Cited by 37
Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question AnsweringPreprint
Banerjee, P., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03101 · Cited by 24
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information
Aly, R., Guo, Z., Schlichtkrull, M., Thorne, J. P., Vlachos, A., Christodoulopoulos, C., Cocarascu, O., & Mittal, A. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4911507 · Cited by 20
A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation
Chen, Z., Min, E., Zhao, X., Li, Y., Jia, X., Liao, J., Li, J., Wang, S., Hu, B., & Yin, D. (2025) · Scientific Data · doi:10.1038/s41597-025-06098-y · Cited by 20
Selectively Answering Ambiguous Questions
Cole, J., Zhang, M., Gillick, D., Eisenschlos, J., Dhingra, B., & Eisenstein, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.35 · Cited by 19
Robust Fine-tuning for Retrieval Augmented Generation against Retrieval Defects
Tu, Y. L., Su, W., Zhou, Y., Liu, Y., & Ai, Q. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730078 · Cited by 50
Guo, X., Yu, M., Gao, Y., Gan, C., Campbell, M. S., & Chang, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.624 · Cited by 27
Pan, F., Canim, M., Glass, M., Gliozzo, A., & Fox, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.acl-demo.24 · Cited by 17
González, A. V., Bansal, G., Fan, A., Mehdad, Y., Jia, R., & Iyer, S. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.95 · Cited by 20
ReasonBERT: Pre-trained to Reason with Distant Supervision
Deng, X., Su, Y., Lees, A., Wu, Y., Yu, C., & Sun, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.494 · Cited by 35
PersianQuAD: The Native Question Answering Dataset for the Persian Language
Kazemi, A., Mozafari, J., & Nematbakhsh, M. A. (2022) · IEEE Access · doi:10.1109/access.2022.3157289 · Cited by 25
PTAU: Prompt Tuning for Attributing Unanswerable Questions
Liao, J., Zhao, X. H., Zheng, J., Li, X., Cai, F., & Tang, J. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532048 · Cited by 26
Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval
Hong, W., Zhang, Z., Wang, J., & Zhao, H. Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.76 · Cited by 30
Xu, S., Pang, L., Shen, H., & Cheng, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557388 · Cited by 25
Understanding Differential Search Index for Text Retrieval
Chen, X., Liu, Y., He, B., Sun, L., & Sun, Y. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.681 · Cited by 13
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Ma, K., Cheng, H., Liu, X., Nyberg, E., & Gao, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.392 · Cited by 11
Towards Imperceptible Document Manipulations against Neural Ranking Models
Chen, X., He, B., Ye, Z., Sun, L., & Sun, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.416 · Cited by 39
BLUEX: A Benchmark Based on Brazilian Leading Universities Entrance eXams
Almeida, T. S., Laitz, T., Bonás, G. K., & Nogueira, R. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45368-7_22 · Cited by 24
CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks
Li, X., Dou, Z., Zhou, Y., & Liu, F. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657778 · Cited by 38
Multi-granular Adversarial Attacks against Black-box Neural Ranking Models
Liu, Y.-A., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657704 · Cited by 35
Comparative Web Search Questions
Bondarenko, A., Braslavski, P., Völske, M., Aly, R., Fröbe, M., Panchenko, A., Biemann, C., Stein, B., & Hagen, M. (2020) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3336191.3371848 · Cited by 28
FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation
Lakhotia, K., Paranjape, B., Ghoshal, A., Yih, S., Mehdad, Y., & Iyer, S. V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.301 · Cited by 38
Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction
Gao, Y., Zhu, H., Ng, P., dos Santos, C. N., Wang, Z., Nan, F., Zhang, D., Nallapati, R. M., Arnold, A. O., & Xiang, B. (2021) · doi:10.18653/v1/2021.acl-long.253 · Cited by 21
Retrieval-guided Counterfactual Generation for QA
Paranjape, B., Lamm, M., & Tenney, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.117 · Cited by 46
Li, M., Lin, S.-C., Oğuz, B., Ghoshal, A., Lin, J., Mehdad, Y., Yih, W.-T., & Chen, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.663 · Cited by 51
Query Reformulation using Query History for Passage Retrieval in Conversational SearchPreprint
Lin, S.-C., Yang, J.-H., Nogueira, R. F., Tsai, M.-F., Wang, C., & Lin, J. (2020) · arXiv (Cornell University) · Cited by 20
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency EncodingPreprint
Wang, S., Zhou, L., Gan, Z., Chen, Y.-C., Fang, Y., Sun, S., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.06097 · Cited by 36
An Open-Source Data Contamination Report for Large Language Models
Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.30 · Cited by 19
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.