Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 801–900 of 1,158
An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension
Liu, X., Liu, K., Li, X., Su, J., Ge, Y., Wang, B., & Luo, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/525 · Cited by 15
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
Xiong, W., Wang, H., & Wang, W. Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.244 · Cited by 20
Suissa, O., Zhitomirsky‐Geffet, M., & Elmalech, A. (2022) · Semantic Web · doi:10.3233/sw-222925 · Cited by 17
Table Retrieval May Not Necessitate Table-specific Model Design
Wang, Z., Jiang, Z., Nyberg, E., & Neubig, G. (2022) · doi:10.18653/v1/2022.suki-1.5 · Cited by 17
LIQUID: A Framework for List Question Answering Dataset Generation
Lee, S., Kim, H., & Kang, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26529 · Cited by 37
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering
Bolotova-Baranova, V., Blinov, V., Филиппова, С. В., Scholer, F., & Sanderson, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.290 · Cited by 39
Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA
Huang, J., Zhong, W., Liu, Q., Gong, M., Jiang, D., & Duan, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.303 · Cited by 19
Reasoning over Public and Private Data in Retrieval-Based Systems
Arora, S., Lewis, P., Fan, A., Kahn, J., & Ré, C. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00580 · Cited by 39
AVA: an Automatic eValuation Approach for Question Answering Systems
Vu, T., & Moschitti, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.412 · Cited by 14
AmazonQA: A Review-Based Question Answering TaskPreprint
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04364 · Cited by 38
Frustratingly Easy Natural Question AnsweringPreprint
Pan, L., Chakravarti, R., Ferritto, A., Glass, M., Gliozzo, A., Roukos, S., Florian, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05286 · Cited by 16
Efficient Conformal Prediction via Cascaded Inference with Expanded AdmissionPreprint
Fisch, A., Schuster, T., Jaakkola, T., & Barzilay, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03114 · Cited by 37
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
Sun, H., Verga, P., Dhingra, B., Salakhutdinov, R., & Cohen, W. W. (2021) · arXiv · doi:10.48550/arxiv.2102.07043 · Cited by 14
Attention-guided Generative Models for Extractive Question AnsweringPreprint
Xu, P., Liang, D., Huang, Z., & Xiang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06393 · Cited by 25
MATE: Multi-view Attention for Table Transformer EfficiencyPreprint
Eisenschlos, J. M., Gor, M., Müller, T., & Cohen, W. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04312 · Cited by 32
TrustRAG: Enhancing Robustness and Trustworthiness in RAGPreprint
Zhou, H., Zhan, Z., Li, Z., Haddadi, H., & Yılmaz, E. (2025) · Qeios · doi:10.32388/z4dwhq · Cited by 23
Investigating Educational and Noneducational Answer Selection for Educational Question Generation
Steuer, T., Filighera, A., & Tregel, T. (2022) · IEEE Access · doi:10.1109/access.2022.3180838 · Cited by 13
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · arXiv · doi:10.48550/arxiv.2407.18940 · Cited by 13
Sun, K., Xu, Y. E., Zha, H., Liu, Y., & Dong, X. L. (2023) · arXiv · doi:10.48550/arxiv.2308.10168 · Cited by 13
OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG
Mo, F., Su, Z., Hui, Y., Zhang, J., Sun, J. A., Liu, Z., Zhang, C., Sakai, T., & Nie, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792524 · Cited by 29
SentiMedQAer: A Transfer Learning-Based Sentiment-Aware Model for Biomedical Question Answering
Zhu, X., Chen, Y., Gu, Y., & Xiao, Z. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.773329 · Cited by 15
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering
Zhou, J., Li, X., Shang, L., Luo, L., Zhan, K., Hu, E., Zhang, X., Jiang, H., Cao, Z., Yu, F., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.493 · Cited by 34
Visual Explanation for Open-Domain Question Answering With BERT
Shao, Z., Sun, S., Zhao, Y., Wang, S., Wei, Z., Gui, T., Turkay, Ç., & Chen, S. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3243676 · Cited by 13
MSQ-BioBERT: Ambiguity Resolution to Enhance BioBERT Medical Question-Answering
Guo, M., Guo, M., Dougherty, E. T., & Jin, F. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583878 · Cited by 22
End-to-end Knowledge Retrieval with Multi-modal Queries
Luo, M., Fang, Z., Gokhale, T., Yang, Y., & Baral, C. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.478 · Cited by 40
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Youssef, P., Koraş, O. A., Li, M., Schlötterer, J., & Seifert, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1043 · Cited by 52
Translucent Answer Predictions in Multi-Hop Reading Comprehension
Bhargav, G. P. S., Glass, M., Garg, D., Shevade, S., Dana, S., Khandelwal, D., Subramaniam, L. V., & Gliozzo, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6272 · Cited by 16
Tempura: Query Analysis with Structural Templates
Wu, T., Wongsuphasawat, K., Ren, D., Patel, K., & DuBois, C. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376451 · Cited by 15
Asking questions on handwritten document collections
Mathew, M., Gómez, L., Karatzas, D., & Jawahar, C. V. (2021) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-021-00383-3 · Cited by 31
Automatic question-answer pair generation using Deep Learning
Kumar, A., Kharadi, A., Singh, D., & Kumari, M. (2021) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca51532.2021.9544654 · Cited by 13
Alkhaldi, T., Chu, C., & Kurohashi, S. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3120643 · Cited by 15
Another Look at DPR: Reproduction of Training and Replication of Retrieval
Ma, X., Sun, K., Pradeep, R., Li, M., & Lin, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_41 · Cited by 30
CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course
Lee, C., Seonwoo, Y., & Oh, A. H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.148 · Cited by 32
Full-Memory Transformer for Image Captioning
Lu, T., Wang, J., & Min, F. (2023) · Symmetry · doi:10.3390/sym15010190 · Cited by 13
KINet: Incorporating Relevant Facts Into Knowledge-Grounded Dialog Generation
Bai, J., Yang, Z., Yang, J., Guo, H., & Li, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3240654 · Cited by 18
Combo of Thinking and Observing for Outside-Knowledge VQA
Si, Q., Mo, Y., Lin, Z., Ji, H., & Wang, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.614 · Cited by 31
Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker
Cho, S., Jeong, S., Seo, J. Y., & Park, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.61 · Cited by 32
Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
Liu, Z., Zhang, H., Xiong, C., Liu, Z., Gu, Y., & Li, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.384 · Cited by 29
MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders Are Better Dense Retrievers
Zhou, K., Liu, X., Gong, Y., Zhao, W. X., Jiang, D., Duan, N., & Wen, J.-R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43415-0_37 · Cited by 19
Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
Wang, Y., Li, J., Naumann, T., Xiong, C., Cheng, H., Tinn, R., Wong, C., Usuyama, N., Rogahn, R., Shen, Z., et al. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3469053 · Cited by 20
ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers
Sun, H., Cohen, W. W., & Salakhutdinov, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.253 · Cited by 52
Learning a Cost-Effective Annotation Policy for Question Answering
Kratzwald, B., Feuerriegel, S., & Sun, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.246 · Cited by 13
RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question AnsweringPreprint
Iyer, S. V., Min, S., Mehdad, Y., & Yih, W.-T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10757 · Cited by 14
Studying Strategically: Learning to Mask for Closed-book QAPreprint
Ye, Q., Li, B. Z., Wang, S., Bolte, B., Ma, H., Yih, W.-T., Ren, X., & Khabsa, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15856 · Cited by 16
Evidence-based Verification for Real World Information Needs.Preprint
Thorne, J., Glockner, M., Vallejo, G., Vlachos, A., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 18
Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak DecoderPreprint
Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T., & Overwijk, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.09206 · Cited by 21
Semantic Answer Similarity for Evaluating Question Answering ModelsPreprint
Risch, J., Möller, T., Gutsch, J., & Pietsch, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06130 · Cited by 51
Is Semantic Chunking Worth the Computational Cost?Preprint
Qu, R., Bao, F. S., & Tu, R. (2024) · Qeios · doi:10.32388/m7ykdz · Cited by 19
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2310.01801 · Cited by 13
“For Llama 1, we choose 4 different tasks, including HumanEval (Chen et al., 2021), GSM8k (Cobbe et al., 2021), NQ (Kwiatkowski et al., 2019) and TQA (Kembhavi et al., 2017) to evaluate models’ abilities on different domains (code, math, question answering and reading comprehension).”
MethodLongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12
“The specific details of the dataset are as follows: NaturalQuestions multi-document QA A multi-document question-answering dataset, comprising 2,655 problems, was built by (Liu et al., 2024) based on the NaturalQuestions dataset (Kwiatkowski et al., 2019).”
Methodde Oliveira, R. S., & Nascimento, E. G. S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104477 · Cited by 14
Kamalipour, A. A., Asadi, S., & Chimeh, M. M. A. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100925 · Cited by 18
Liu, P., Wang, X., Wang, L., Ye, W., Xi, X., & Zhang, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481909 · Cited by 20
RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media
Wadhwa, S., Khetan, V., Amir, S., & Wallace, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.61 · Cited by 25
Optimizing Retrieval-augmented Reader Models via Token Elimination
Berchansky, M., Izsak, P., Caciularu, A., Dagan, I., & Wasserblat, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.93 · Cited by 23
Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering
Wang, D., Huang, Q., Jackson, M., & Gao, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00646 · Cited by 26
MS MARCO Web Search: A Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Chen, Q., Geng, X., Rosset, C., Buractaon, C., Lu, J., Shen, T., Zhou, K., Xiong, C., Gong, Y., Bennett, P., et al. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3648327 · Cited by 34
Kostric, I., & Balog, K. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657933 · Cited by 33
Understanding and Mitigating the Threat of Vec2Text to Dense Retrieval Systems
Zhuang, S., Koopman, B., Chu, X., & Zuccon, G. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698414 · Cited by 24
W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering
Nian, J., Peng, Z., Wang, Q., & Fang, Y. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744578 · Cited by 27
Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study
Yu, S., Cheng, M., Liu, Q., Wang, D., Yang, J., Ouyang, J., Luo, Y., Lei, C., & Chen, E. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761340 · Cited by 24
Unsupervised Adaptation of Question Answering Systems via Generative Self-training
Rennie, S. J., Marcheret, E., Mallinar, N., Nahamoo, D., & Goel, V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.87 · Cited by 12
Weakly-Supervised Open-Retrieval Conversational Question Answering
Qu, C., Liu, Y., Chen, C., Croft, W. B., Krishna, K., & Iyyer, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_35 · Cited by 13
Kulshreshtha, D., Belfer, R., Serban, I. V., & Reddy, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.566 · Cited by 19
Understanding Questions that Arise When Working with Business Documents
Jahanbakhsh, F., Nouri, E., Sim, R., White, R. W., & Fourney, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555761 · Cited by 15
Adversarial Knowledge Distillation Based Biomedical Factoid Question Answering
Bai, J., Yin, C., Zhang, J., Wang, Y., Dong, Y., Rong, W., & Xiong, Z. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3161032 · Cited by 13
Constructing Arabic Reading Comprehension Datasets: Arabic WikiReading and KaifLematha
Albilali, E., Al-Twairesh, N., & Hosny, M. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09577-5 · Cited by 15
How Do We Answer Complex Questions: Discourse Structure of Long-form Answers
Xu, F., Li, J. J., & Choi, E. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.249 · Cited by 28
Entity-aware answer sentence selection for question answering with transformer-based language models
Abbasiantaeb, Z., & Momtazi, S. (2022) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00724-6 · Cited by 12
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?
Qin, C., Joty, S., Li, Q., & Zhao, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.659 · Cited by 21
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
Yu, W., Jiang, M., Clark, P., & Sabharwal, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.515 · Cited by 42
Mitigating Knowledge Conflicts in Language Model-Driven Question Answering
Cao, H., Zhang, Z., Li, X., Wu, C., Zhang, H., & Zhang, W. (2024) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst64597.2024.11118151 · Cited by 12
Talk to Papers: Bringing Neural Question Answering to Academic Search
Zhao, T., & Lee, K. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.5 · Cited by 13
Self-citationDaNetQA: A Yes/No Question Answering Dataset for the Russian Language
Glushkova, T., Machnev, A., Fenogenova, A., Shavrina, T., Artemova, E., & Ignatov, D. I. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72610-2_4 · Cited by 13
Improving QA Generalization by Concurrent Modeling of Multiple Biases
Wu, M., Moosavi, N. S., Rücklé, A., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.74 · Cited by 18
Biomedical Question Answering: A Survey of Methods and Datasets
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds50568.2020.9268742 · Cited by 18
Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval
Wang, Y., Bai, J., Wang, Y., Zhang, J., Rong, W., Ji, Z., Wang, S., & Xiao, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.198 · Cited by 13
Hyperdecoders: Instance-specific decoders for multi-task NLP
Ivison, H., & Peters, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.124 · Cited by 41
Dataset Geography: Mapping Language Data to Language Users
Faisal, F., Wang, Y., & Anastasopoulos, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.239 · Cited by 23
Extracting Contextualized Quantity Facts from Web Tables
Ho, V. T., Pal, K., Razniewski, S., Berberich, K., & Weikum, G. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450072 · Cited by 12
The TechQA DatasetPreprint
Castelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02984 · Cited by 30
QURIOUS: Question Generation Pretraining for Text GenerationPreprint
Narayan, S., Simões, G., Ma, J., Craighead, H., & McDonald, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11026 · Cited by 14
A Survey of Document Grounded Dialogue Systems (DGDS)Preprint
Ma, L., Zhang, W., Li, M., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13818 · Cited by 23
Towards Robust Neural Retrieval Models with Synthetic Pre-TrainingPreprint
Reddy, R. G., Yadav, V., Sultan, M. A., Franz, M., Castelli, V., Ji, H., & Sil, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07800 · Cited by 19
Fine-tune the Entire RAG Architecture (including DPR retriever) for Question-AnsweringPreprint
Siriwardhana, S., Weerasekera, R., Wen, E., & Nanayakkara, S. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11517 · Cited by 18
More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question AnsweringPreprint
Bai, Y., & Wang, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12264 · Cited by 22
ECONET: Effective Continual Pretraining of Language Models for Event Temporal ReasoningPreprint
Han, R., Ren, X., & Peng, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15283 · Cited by 28
Learning Fine-Grained Grounded Citations for Attributed Large Language Models
Huang, L., Feng, X., Ma, W., Gu, Y., Zhong, W., Feng, X., Yu, W., Peng, W., Tang, D., Tu, D., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.838 · Cited by 12
Dual-Feedback Knowledge Retrieval for Task-Oriented Dialogue Systems
Shi, T., Li, L., Lin, Z., Yang, T., Quan, X., & Wang, Q. (2023) · arXiv · doi:10.48550/arxiv.2310.14528 · Cited by 12
“Previous research has extensively explored meth-ods for knowledge retrieval across various tasks, including question answering (Chen et al., 2017; Kwiatkowski et al., 2019), fact checking (Thorne et al., 2018), and dialogue systems (Dinan et al., 2019).”
BackgroundRAG vs. GraphRAG: A Systematic Evaluation and Key Insights
Han, H., Ma, L., Wang, Y., Shomer, H., Guo, K., Lei, Y., Qi, Z., Hua, Z., Long, B., Liu, H., et al. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817575 · Cited by 12
“For the single-hop QA task, we select the Natural Questions (NQ) dataset [20].”
Dehghan, M., Alomrani, M., Bagga, S., Alfonso-Hermelo, D., Bibi, K., Ghaddar, A., Zhang, Y., Li, X., Hao, J., Liu, Q., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.764 · Cited by 12
Lin, W., Blloshmi, R., Byrne, B., de Gispert, A., & Iglesias, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.133 · Cited by 15
A Unified Encoder-Decoder Framework with Entity Memory
Zhang, Z., Yu, W., Zhu, C., & Jiang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.43 · Cited by 15
Prompting with Pseudo-Code Instructions
Mishra, M., Kumar, P., Bhat, R. A., Murthy, R., Contractor, D., & Tamilselvam, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.939 · Cited by 25
Piryani, B., Mozafari, J., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657891 · Cited by 24
"Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time
Rome, S., Chen, T., Tang, R., Zhou, L., & Türe, F. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661345 · Cited by 17
Lehmann, J., Bhandiwad, D., Gattogi, P., & Vahdati, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00671 · Cited by 24
Legate, A. E., Nimon, K., & Noblin, A. (2024) · F1000Research · doi:10.12688/f1000research.151493.1 · Cited by 15
A Knowledge Graph Embedding Model for Answering Factoid Entity Questions
Jafarzadeh, P., Ensan, F., Alavi, M. A. A., & Zarrinkalam, F. (2024) · ACM Transactions on Information Systems · doi:10.1145/3678003 · Cited by 12
Srivastava, A., & Memon, A. M. (2024) · IEEE Access · doi:10.1109/access.2024.3446854 · Cited by 22
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.