Natural Questions: A Benchmark for Question Answering Research
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 1,001–1,100 of 1,158
Chen, Z., Sun, X., Jiao, X., Lian, F., Kang, Z., Wang, D., & Xu, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30087 · Cited by 69
Hoque, M. N., Mahfuz, A. A., Kindi, M. S., & Hassan, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642007 · Cited by 17
Fairness Definitions in Language Models Explained
Yin, Z., Wang, Z., Palikhe, A., & Zhang, W. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70063 · Cited by 29
Survey of Large Language Models for Answering Questions Across Various Fields
Shailendra, P., Ghosh, R. C., Kumar, R., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717078 · Cited by 24
Online Safety Analysis for LLMs: A Benchmark, an Assessment, and a Path Forward
Xie, X., Song, J.-Y., Zhou, Z., Huang, Y.-H., Song, D., & Ma, L. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3603547 · Cited by 15
Open Korean Corpora: A Practical Report
Cho, W. I., Moon, S., & Song, Y. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.12 · Cited by 13
Shinoda, K., Sugawara, S., & Aizawa, A. (2021) · doi:10.18653/v1/2021.acl-srw.21 · Cited by 10
KerasBERT: Modeling the Keras Language
Shorten, C., & Khoshgoftaar, T. M. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00041 · Cited by 9
Can Question Rewriting Help Conversational Question Answering?
Ishii, E., Xu, Y., Cahyawijaya, S., & Wilie, B. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.13 · Cited by 11
Lim, J., Kang, M., Hur, Y., Jeong, S. W., Kim, J., Jang, Y., Lee, D., Ji, H., Shin, D.-H., Kim, S., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.75 · Cited by 15
Bai, J., Guo, H., Liu, J., Yang, J., Liang, X., Yan, Z., & Li, Z. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614901 · Cited by 19
DG2: Data Augmentation Through Document Grounded Dialogue Generation
Wu, Q., Feng, S., Chen, D., Joshi, S., Lastras, L., & Zhou, Y. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.21 · Cited by 20
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 11
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8
“…are OpenbookQA (Mihaylov et al., 2018), MMLU (Hendrycks et al., 2020), NarrativeQA (Koˇcisk`y et al., 2018), NaturalQuestions (closed-book) (Kwiatkowski et al., 2019), NaturalQuestions (open-book), Math (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), LegalBench (Guha et al., 2024),…”
Wang, S., Zhang, Y., & Nguyen, C.-T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29885 · Cited by 21
List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation
Xu, S., Pang, L., Xu, J., Shen, H., & Cheng, X. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645336 · Cited by 24
Hyperbolic Pre-Trained Language Model
Chen, W., Han, X., Lin, Y., He, K., Xie, R., Zhou, J., Liu, Z., & Sun, M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3407575 · Cited by 15
Handover QG: Question Generation by Decoder Fusion and Reinforcement Learning
Chung, H.-L., Chan, Y.-H., & Fan, Y.-C. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426292 · Cited by 8
Zhang, C., Wang, B., & Song, D. (2024) · ACM Transactions on Information Systems · doi:10.1145/3677375 · Cited by 8
Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation
Wu, C., Shao, N., Liu, Z., Xiao, S., Li, C., Zhang, C., Wang, S., & Lian, D. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703580 · Cited by 18
Aligning large language models with human preferences using historical text edits
Majkutewicz, J., & Szymański, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113566 · Cited by 9
Jiao, Y., Wang, X., & Yang, K. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730058 · Cited by 52
EDCEW-LLM: Error detection and correction in English writing: A large language model-based approach
Wang, J. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.08.005 · Cited by 8
Performance Evaluation of Modern GPU Accelerator-Based Edge Systems: A Holistic Approach
Lee, H., & Kang, P. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3616166 · Cited by 11
Privacy-Aware Distributed RAG with Confidence-Aware Retrieval Routing
Huang, W., Xu, S., & Jiang, Z. (2026) · International Conference on Image Processing and Artificial Intelligence (ICIPAI) · doi:10.1109/icipai70034.2026.11605698 · Cited by 9
Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering
Bachina, S., Balumuri, S., & Kamath, S. S. (2021) · doi:10.18653/v1/2021.dialdoc-1.9 · Cited by 8
Evaluation Paradigms in Question Answering
Rodríguez, P., & Boyd-Graber, J. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.758 · Cited by 21
semiPQA: A Study on Product Question Answering over Semi-structured Data
Shen, X., Barlacchi, G., Tredici, M. D., Cheng, W., & de Gispert, A. (2022) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2022.ecnlp-1.14 · Cited by 15
OpenMatch-v2: An All-in-one Multi-Modality PLM-based Information Retrieval Toolkit
Yu, S., Liu, Z., Xiong, C., & Liu, Z. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591813 · Cited by 19
Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya
Gaim, F., Yang, W., Park, H., & Park, J. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.661 · Cited by 26
Generating Information-Seeking Conversations from Unlabeled Documents
Kim, G., Kim, S., Yoo, K. M., & Kang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.151 · Cited by 28
Liu, J., Liu, J., Yang, Y., Wang, J., Wu, W., Zhao, D., & Yan, R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.39 · Cited by 10
VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Zhang, Y., Li, J., & Li, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.203 · Cited by 25
Assessment of adults with low literacy skills: a review of methods
Chyl, K., Dębska, A., Pokropek, A., Szczerbiński, M., Tanaś, Ł., & Sitek, M. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1346073 · Cited by 13
Development of a Geographical Question- Answering System in the Kazakh Language
Mukanova, A. S., Barlybayev, A., Nazyrova, A., Kussepova, L., Matkarimov, B. T., & Abdikalyk, G. (2024) · IEEE Access · doi:10.1109/access.2024.3433426 · Cited by 10
Li, Z., Prabhu, S., Popp, Z., Jain, S., Balakundi, V., Ang, T. F. A., Au, R., & Chen, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0328262 · Cited by 11
DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task
Liu, W., Zhu, Y., Dou, Z., & Zhou, Y. (2026) · ACM Transactions on Information Systems · doi:10.1145/3788867 · Cited by 22
Question Answering over Curated and Open Web Sources
Roy, R. S., & Anand, A. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401421 · Cited by 16
Towards building a Robust Industry-scale Question Answering System
Chakravarti, R., Ferritto, A., Iyer, B., Pan, L., Florian, R., Roukos, S., & Sil, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-industry.9 · Cited by 12
Improving Embedding-based Large-scale Retrieval via Label Enhancement
Liu, P., Wang, X., Wang, S., Ye, W., Xi, X., & Zhang, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.13 · Cited by 8
WDRASS: A Web-scale Dataset for Document Retrieval and Answer Sentence Selection
Zhang, Z., Vu, T., Gandhi, S., Chadha, A., & Moschitti, A. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557678 · Cited by 8
The Web Can Be Your Oyster for Improving Language Models
Li, J., Tang, T., Zhao, W. X., Wang, J., Nie, J., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.46 · Cited by 20
In-Context Demonstration Selection with Cross Entropy Difference
Iter, D., Pryzant, R., Xu, R., Wang, S., Liu, Y., Yi‐chong, X., & Zhu, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.81 · Cited by 29
Mitigating Temporal Misalignment by Discarding Outdated Facts
Zhang, M., & Choi, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.879 · Cited by 36
Who are you referring to? Coreference resolution in image narrations
Goel, A., Fernando, B., Keller, F., & Bilen, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01399 · Cited by 18
Evaluation of Medium-large Language Models at Zero-shot Closed Book Generative Question Answering
Peinl, R., & Wirth, J. (2024) · doi:10.5121/csit.2024.140106 · Cited by 9
Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot
Koyanagi, K., Wang, D., Noguchi, K., Kondo, M., Serebrenik, A., Kamei, Y., & Ubayashi, N. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644917 · Cited by 18
Soto-Jiménez, F., Martínez-Velásquez, M., Chicaiza, J. A., Naranjo, P. G. V., & Nadjet Bouayad-agha, A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66329-1_37 · Cited by 8
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding
Takahashi, T., Taniguchi, M., Taniguchi, T., & Ohkuma, T. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5824 · Cited by 9
Kwame: A Bilingual AI Teaching Assistant for Online SuaCode Courses
Boateng, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78270-2_16 · Cited by 8
Wang, X., Shou, L., Gong, M., Duan, N., & Jiang, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.370 · Cited by 8
Recursive Tree-Structured Self-Attention for Answer Sentence Selection
Mrini, K., Farcas, E., & Nakashole, N. (2021) · doi:10.18653/v1/2021.acl-long.358 · Cited by 12
Towards a more Robust Evaluation for Conversational Question Answering
Siblini, W., Sayil, B., & Kessaci, Y. (2021) · doi:10.18653/v1/2021.acl-short.130 · Cited by 8
Ofoghi, B., Mahdiloo, M., & Yearwood, J. L. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108574 · Cited by 8
Synthetic Question Value Estimation for Domain Adaptation of Question Answering
Yue, X., Yao, Z., & Sun, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.95 · Cited by 26
A privacy-preserving dialogue system based on argumentation
Fazzinga, B., Galassi, A., & Torroni, P. (2022) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2022.200113 · Cited by 15
Text Summarization with Latent Queries
Xu, Y., & Lapata, M. (2021) · arXiv · doi:10.48550/arxiv.2106.00104
MeDiaQA: A Question Answering Dataset on Medical DialoguesPreprint
Suri, H., Zhang, Q., Huo, W., Liu, Y., & Guan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08074 · Cited by 17
Contrastive Domain Adaptation for Question Answering using Limited Text CorporaPreprint
Yue, Z., Kratzwald, B., & Feuerriegel, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13854 · Cited by 10
GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage RetrievalPreprint
Möller, T., Risch, J., & Pietsch, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.12741 · Cited by 24
Beamaggr: Beam aggregation reasoning over multi-source knowledge for multi-hop question answering
Chu, Z., Chen, J., Chen, Q., Wang, H., Zhu, K., Du, X., Yu, W., Liu, M., & Qin, B. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.67 · Cited by 11
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
Gonzalez, A. V., Bansal, G., Fan, A., Jia, R., Mehdad, Y., & Iyer, S. (2020) · arXiv · doi:10.48550/arxiv.2012.15075 · Cited by 7
Bejan, I., Sokolov, A., & Filippova, K. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.625 · Cited by 7
ROGER: Ranking-Oriented Generative Retrieval
Zhou, Y., Yao, J., Dou, Z., Tu, Y., Wu, L., Chua, T., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3603167 · Cited by 16
Steering Large Language Models for Cross-lingual Information Retrieval
Guo, P., Ren, Y., Hu, Y., Cao, Y., Li, Y., & Huang, H. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657819 · Cited by 25
Perceived Text Relevance Estimation Using Scanpaths and GNNs
Selim, A. M., Bhatti, O. S., Barz, M., & Sonntag, D. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685736 · Cited by 7
SUGAR: Leveraging Contextual Confidence for Smarter Retrieval
Zubkova, H., Park, J.-H., & Lee, S.-W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890064 · Cited by 15
Hard label adversarial attack with high query efficiency against NLP models
Qiu, S., Liu, Q., Zhou, S., Gou, M., Zeng, Y., Zhang, Z., & Wu, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-93566-5 · Cited by 10
Evaluating Sparse and Dense Retrieval in Retrieval-Augmented Generation Systems: A Study
Wang, Y., Dai, G., Ke, S., & Zheng, C. (2024) · International Conference on Communication and Information Processing (ICCIP) · doi:10.1145/3708657.3708747 · Cited by 7
Education Research: Can Large Language Models Match MS Specialist Training?
Inojosa, H., Ramezanzadeh, A., Gasparovic-Curtini, I., Wiest, I. C., Kather, J. N., Gilbert, S., & Ziemssen, T. (2025) · Neurology Education · doi:10.1212/ne9.0000000000200260 · Cited by 8
HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows
Hu, Z., Murthy, V., Pan, Z., Li, W., Fang, X., Ding, Y., & Wang, Y. (2025) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3731569.3764806 · Cited by 28
Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking
Ji, Y., Xu, Z., Liu, Z., Yan, Y., Yu, S., Li, Y., Liu, Z., Gu, Y., Yu, G., & Sun, M. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769491 · Cited by 20
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
Jiao, F., Guo, Y., Niu, Y., Ji, F., Li, F.-L., & Nie, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.13 · Cited by 10
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.756 · Cited by 15
G4: Grounding-guided Goal-oriented Dialogues Generation with Multiple Documents
Zhang, S., Du, Y., Liu, G., Zhao, Y., & Cao, Y. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.11 · Cited by 12
Cooperative Self-training of Machine Reading Comprehension
Luo, H., Li, S.-W. D., Gao, M., Yu, S., & Glass, J. R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.18 · Cited by 15
ELQA: A Corpus of Metalinguistic Questions and Answers about English
Behzad, S., Sakaguchi, K., Schneider, N., & Zeldes, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.113 · Cited by 19
On Complementarity Objectives for Hybrid Retrieval
Lee, D., Hwang, S.-W., Lee, K., Choi, S. L., & Park, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.746 · Cited by 22
Self-citationAnalyzing Multi-Task Learning for Abstractive Text Summarization
Kirstein, F., Wahle, J. P., Ruas, T., & Gipp, B. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.5 · Cited by 13
Matsubara, Y., Soldaini, L., Lind, E., & Moschitti, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.537 · Cited by 10
Retrieval-Based Unsupervised Noisy Label Detection on Text Data
Liu, P., Yang, J., Wang, L., Wang, S., Hao, Y., & Bai, H. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615146 · Cited by 27
SciPhyRAG - Retrieval Augmentation to Improve LLMs on Physics Q &A
Anand, A., Goel, A., Hira, M., Buldeo, S., Kumar, J., Verma, A., Gupta, R., & Shah, R. R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49601-1_4 · Cited by 28
Cross-lingual Open-Retrieval Question Answering for African Languages
Ogundepo, O., Gwadabe, T., Rivera, C., Clark, J. R., Ruder, S., Adelani, D. I., Dossou, B. F. P., Diop, A., Sikasote, C., Hacheme, G., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.997 · Cited by 35
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 20
Pirozelli, P., José, M. M., Silveira, I. C., Cação, F. N., Peres, S. M., Brandão, A. A. F., Costa, A. H. R., & Cozman, F. G. (2024) · Data Intelligence · doi:10.1162/dint_a_00245 · Cited by 15
ArQuAD: An Expert-Annotated Arabic Machine Reading Comprehension Dataset
Obeidat, R., Al-Harbi, M., Al‐Ayyoub, M., & Alawneh, L. M. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10248-6 · Cited by 13
Enhancing Open-Domain Table Question Answering via Syntax- and Structure-aware Dense Retrieval
Jin, N., Li, D., Chen, J., Siebert, J. I., & Chen, Q. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.18 · Cited by 12
TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions
Mozafari, J., Jangra, A., & Jatowt, A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657855 · Cited by 15
Source-Free Domain Adaptation for Question Answering with Masked Self-training
Yin, M. J., Wang, B., Dong, Y., & Ling, C. X. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00669 · Cited by 7
Advancing Faithfulness of Large Language Models in Goal-Oriented Dialogue Question Answering
Sticha, A., Braunschweiler, N., Doddipatla, R., & Knill, K. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665573 · Cited by 11
Measuring an LLM's Proficiency at using APIs: A Query Generation Strategy
Sheng, Y., Gandhe, S., Kanagal, B., Edmonds, N., Fisher, Z., Tata, S., & Selvan, A. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671592 · Cited by 7
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6
“…(GSM), code understanding (CodeU), goal-oriented dialogues (Multi-Doc2Dial (Feng et al., 2021)), extractive QA (CUAD (Hendrycks et al., 2021b), NQ (Kwiatkowski et al., 2019)), abstractive QA (LongFQA, NarrativeQA (Ko ˇ cisk ´ y et al., 2017), Qasper (Dasigi et al., 2021)), single document…”
BackgroundMethodTrusting Your Evidence: Hallucinate Less with Context-aware Decoding
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, S. W.-T. (2023) · arXiv · doi:10.48550/arxiv.2305.14739 · Cited by 6
“NQ-Swap is based on a QA dataset, natural questions (NQ) (Kwiatkowski et al., 2019), where the objective is to answer questions based on a reliable gold document.”
BackgroundMethodInfluentialThe Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models
Hagström, L., Saynova, D., Norlund, T., Johansson, M., & Johansson, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.332 · Cited by 6
Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework
Shao, Z., & Huang, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.128 · Cited by 6
How Does Knowledge Selection Help Retrieval Augmented Generation?
Li, X., & Ouyang, J. (2024) · arXiv · doi:10.48550/arxiv.2410.13258 · Cited by 6
Exploring The Landscape of Distributional Robustness for Question Answering Models
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., & Schmidt, L. (2022) · arXiv · doi:10.48550/arxiv.2210.12517 · Cited by 6
Large language models robustness against perturbation
Alahmari, S. S., Hall, L., Mouton, P. R., & Goldgof, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-29770-0 · Cited by 5
“Wang et al. assess LLMs robustness to perturbation, using a subset of the Natural Questions dataset 11 (1k questions) to assess robustness 5 .”
Farahani, M., & Johansson, R. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.943 · Cited by 5
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., & Chen, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14768 · Cited by 19
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.