The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Natural Questions: A Benchmark for Question Answering Research
Open accessAbstract
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We instantiate this task with data from NaturalQuestions-Open (Lee et al., 2019;Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine, coupled with human-annotated answers extracted from Wikipedia.”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“This level of expert disagreement is in line with what is observed in other well-known machine learning benchmarks59–62.”
From Humanity's Last Exam (2025) · cites this paper as [59]
In the citing paper
“NaturalQuestions contains historical queries issued to the Google search engine coupled with long and short answers extracted from Wikipedia.”
From Evaluating Verifiability in Generative Search Engines (2023) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Kwiatkowski et al., 2019
In the citing paper
“The widening gap (PRECOG loses 15 F1 on the ≥ 2 , 400 -token tail) is not a defect of the algorithm but a property of the underlying SSM's memory”
From Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection (2026) · cites this paper as [35]
18 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · cited 2×
“The Bayes optimal hypothesis (see Devroye et al., 1997) for hβ , defined asargmaxf Eq,d,a[[hβ(a, f(q, d))]], is a function of the posterior distribution p(·|q, d),11 and will generally exceed the performance of a single random annotation, Eq,d,a[[ ∑ l p(l|q, d) × hβ(a, l)]].”
BackgroundAutomatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · Proceedings of the 14th conference on Computational linguistics - · doi:10.3115/992133.992154 · Cited by 3,253
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Research works printed but not linked: 4
- Hearst Marti A., 1992, COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics
- Hill Felix, 2015, Proceedings of the International Conference on Learning Representations
- Lai Guokun, 2017, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 785
- Nguyen Tri, 2016, Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches
Cited by
Items 901–1,000 of 1,143
Disentangled Retrieval and Reasoning for Implicit Question Answering
Liu, Q., Geng, X., Wang, Y., Cambria, E., & Jiang, D. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3220933 · Cited by 17
Slovak Dataset for Multilingual Question Answering
Hládek, D., Staš, J., Juhár, J., & Koctúr, T. (2023) · IEEE Access · doi:10.1109/access.2023.3262308 · Cited by 11
Near-Duplicate Sequence Search at Scale for Large Language Model Memorization Evaluation
Peng, Z., Wang, Z., & Deng, D. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3589324 · Cited by 26
Xu, S., Pang, L., Shen, H., & Cheng, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.365 · Cited by 22
Kim, N., Htut, P. M., Bowman, S. R., & Petty, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.472 · Cited by 22
Ekram, S. M. S., Rahman, A. A., Altaf, M. S., Islam, M. S., Rahman, M. M., Rahman, M. M., Hossain, M. A., & Kamal, A. R. M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.186 · Cited by 13
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts
Ye, Q., Zha, J., & Ren, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.189 · Cited by 26
Universal Self-Adaptive Prompting
Wan, X., Sun, R., Nakhost, H., Dai, H., Eisenschlos, J. M., Arık, S. Ö., & Pfister, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.461 · Cited by 42
CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval
Guo, J., Cai, Y., Bi, K., Fan, Y., Chen, W., Zhang, R., & Cheng, X. (2024) · ACM Transactions on Information Systems · doi:10.1145/3678880 · Cited by 16
WTR: A Test Collection for Web Table Retrieval
Chen, Z., Zhang, S., & Davison, B. D. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463260 · Cited by 15
Modeling Context in Answer Sentence Selection Systems on a Latency Budget
Han, R., Soldaini, L., & Moschitti, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.261 · Cited by 14
CoMSum and SIBERT: A Dataset and Neural Model for Query-Based Multi-document Summarization
Kulkarni, S., Chammas, S., Zhu, W., Sha, F., & Ie, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_6 · Cited by 13
Boosting Search Engines with Interactive Agents
Adolphs, L., Börschinger, B., Buck, C., Huebscher, M. C., Ciaramita, M., Espeholt, L., Hofmann, T., Kilcher, Y., Rothe, S., Sessa, P. G., et al. (2021) · Transactions on machine learning research · doi:10.3929/ethz-b-000591593 · Cited by 24
RELiC: Retrieving Evidence for Literary Claims
Thai, K., Chang, Y., Krishna, K., & Iyyer, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.517 · Cited by 28
Salient Span Masking for Temporal Understanding
Cole, J. R., Chaudhary, A., Dhingra, B., & Talukdar, P. P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.222 · Cited by 22
Neural Knowledge Bank for Pretrained Transformers
Dai, D., Jiang, W., Dong, Q., Lyu, Y., & Sui, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_60 · Cited by 25
CFO: A Framework for Building Production NLP Systems
Chakravarti, R., Pendus, C., Sakrajda, A., Ferritto, A., Pan, L., Glass, M., Castelli, V., Murdock, J. W., Florian, R., Roukos, S., et al. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3006 · Cited by 12
Towards Automatic Generation of Questions from Long AnswersPreprint
Mishra, S. K., Goel, P., Sharma, A., Jagannatha, A., Jacobs, D. W., & Daumé, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05109 · Cited by 12
Zhang, Y., Dong, X., Chang, S., Yu, T., Shi, P., & Zhang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12634 · Cited by 27
Qi, P., Lee, H., Sido, O. T., & Manning, C. D. (2020) · arXiv (Cornell University) · Cited by 11
Theory and Practice of Algorithm EngineeringPreprint
Mendling, J., Depaire, B., & Leopold, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.10675 · Cited by 17
Proving Test Set Contamination in Black Box Language Models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2310.17623 · Cited by 11
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, É. (2021) · HAL · Cited by 10
A Study of Situational Reasoning for Traffic Understanding
Zhang, J., Ilievski, F., Ma, K., Kollaa, A., Francis, J., & Oltramari, A. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599246 · Cited by 12
Towards a small language model powered chain‐of‐reasoning for open‐domain question answering
Roh, J., Kim, M., & Bae, K. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0355 · Cited by 13
TWOLAR: A TWO-Step LLM-Augmented Distillation Method for Passage Reranking
Baldelli, D., Jiang, J., Aizawa, A., & Torroni, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56027-9_29 · Cited by 10
Building efficient and effective OpenQA systems for low-resource languages
Budur, E., Özçelik, R., Soylu, D., Khattab, O., Güngör, T., & Potts, C. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112243 · Cited by 13
Performance and sustainability of BERT derivatives in dyadic data
Escarda-Fernández, M., Eiras‐Franco, C., Cancela, B., Guijarro‐Berdiñas, B., & Alonso‐Betanzos, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125647 · Cited by 12
Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?
Ni, S., Bi, K., Yu, L., & Guo, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1710-4_10 · Cited by 19
Crafting the Path: Robust Query Rewriting for Information Retrieval
Baek, I., Lee, J., Yang, J., & Lee, H. (2025) · IEEE Access · doi:10.1109/access.2025.3538665 · Cited by 18
Wang, T., Jiao, X., Zhu, Y., Chen, Z., He, Y., Chu, X., Gao, J., Wang, Y., & Ma, L. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714640 · Cited by 105
HeterRAG: Heterogeneous Processing-in-Memory Acceleration for Retrieval-augmented Generation
Liu, C., Liu, H., Chen, D., Huang, Y., Zhang, Y., Xiao, W., Liao, X., & Jin, H. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731089 · Cited by 17
Improving Text Embedding Models with Positive-aware Hard-negative Mining
Moreira, G. D. S. P., Osmulski, R., Xu, M., Ak, R., Schifferer, B., & Oldridge, E. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761254 · Cited by 15
Generative Retrieval with Few-Shot Indexing
Askari, A., Meng, C., Aliannejadi, M., Ren, Z., Kanoulas, E., & Verberne, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_52 · Cited by 16
Less is More: Compact Clue Selection for Efficient Retrieval-Augmented Generation Reasoning
Zhang, Q., Zhang, H., Pang, L., Tong, Y., Zheng, H., & Zheng, Z. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792158 · Cited by 29
Ni, A., Gardner, M., & Dasigi, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.497 · Cited by 15
Machine Reading at Scale: A Search Engine for Scientific and Academic Research
Sousa, N., Oliveira, N. R. B. D., & Praça, I. (2022) · Systems · doi:10.3390/systems10020043 · Cited by 10
A Comparison of Transformer-Based Language Models on NLP Benchmarks
Greco, C. M., Tagarelli, A., & Zumpano, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08473-7_45 · Cited by 11
COVID-19 Claim Radar: A Structured Claim Extraction and Tracking System
Li, M., Reddy, R. G., Wang, Z., Chiang, Y.-S., Lai, T., Yu, P., Zhang, Z., & Ji, H. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.13 · Cited by 12
Dataset versus reality: Understanding model performance from the perspective of information need
Yu, M., & Sun, A. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24825 · Cited by 11
GLEN: Generative Retrieval via Lexical Index Learning
Lee, S., Choi, M., & Lee, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.477 · Cited by 41
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering
Zhong, V., Shi, W., Yih, W.-T., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.470 · Cited by 15
TriSampler: A Better Negative Sampling Principle for Dense Retrieval
Zhen, Y., Shao, Z., Dong, Y., & Tang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28779 · Cited by 15
Unsupervised Large Language Model Alignment for Information Retrieval via Contrastive Feedback
Dong, Q., Liu, Y., Ai, Q., Wu, Z., Li, H., Liu, Y., Wang, S., Yin, D., & Ma, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657689 · Cited by 17
Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation
Xu, Z., Chen, D., Kuang, J., Yi, Z., Li, Y., & Shen, Y. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657695 · Cited by 25
Explanatory argument extraction of correct answers in resident medical exams
Goenaga, I., Atutxa, A., Gojenola, K., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102985 · Cited by 17
Measuring Domain Portability and Error Propagation in Biomedical QA
Hosein, S., Andor, D., & McDonald, R. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43887-6_61 · Cited by 11
Latent Retrieval for Large-Scale Fact-Checking and Question Answering with NLI training
Samarinas, C., Hsu, W., & Lee, M. L. (2020) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai50040.2020.00147 · Cited by 10
Continual Domain Adaptation for Machine Reading Comprehension
Su, L., Guo, J., Zhang, R., Fan, Y., Lan, Y., & Cheng, X. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412047 · Cited by 10
On the Influence of Masking Policies in Intermediate Pre-training
Ye, Q., Li, B. Z., Wang, S., Bolte, B., Ma, H., Yih, W.-T., Ren, X., & Khabsa, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.573 · Cited by 11
WBI at MEDIQA 2021: Summarizing Consumer Health Questions with Generative Transformers
Sänger, M., Weber, L., & Leser, U. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.9 · Cited by 12
Robust Question Answering Through Sub-part Alignment
Chen, J., & Durrett, G. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.98 · Cited by 14
A Few More Examples May Be Worth Billions of Parameters
Kirstain, Y., Lewis, P., Riedel, S., & Levy, O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.72 · Cited by 35
Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering
Li, M., Li, M., Xiong, K., & Lin, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.26 · Cited by 21
On Synthetic Data for Back Translation
Xu, J., Ruan, Y., Bi, W., Huang, G., Shi, S., Chen, L., & Liu, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.32 · Cited by 12
Lewis, P., Oğuz, B., Xiong, W., Petroni, F., Yih, S., & Riedel, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.226 · Cited by 13
Generation of Coherent Multi-Sentence Texts with a Coherence Mechanism
Zhao, Q., Niu, J., Liu, X., He, W., & Tang, S. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101457 · Cited by 12
Context Generation Improves Open Domain Question Answering
Su, D., Patwary, M. A., Prabhumoye, S., Xu, P., Prenger, R., Shoeybi, M., Fung, P., Anandkumar, A., & Catanzaro, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.60 · Cited by 18
A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selection
Bonadiman, D., & Moschitti, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.457 · Cited by 11
Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder.Preprint
Lu, S., Xiong, C., He, D., Ke, G., Malik, W., Dou, Z., Bennett, P. N., Liu, T., & Overwijk, A. (2021) · arXiv (Cornell University) · Cited by 10
Efficient pre-training objectives for TransformersPreprint
Liello, L. D., Gabburo, M., & Moschitti, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09694 · Cited by 13
SpartQA: : A Textual Question Answering Benchmark for Spatial ReasoningPreprint
Mirzaee, R., Faghihi, H. R., Ning, Q., & Kordjmashidi, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05832 · Cited by 37
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning
Wang, J., Li, J., & Zhao, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.179 · Cited by 9
Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections
Ko, W.-J., Dalton, C., Simmons, M., Fisher, E., Durrett, G., & Li, J. J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.806 · Cited by 30
Thakur, N., Bonifacio, L., Fröbe, M., Bondarenko, A., Kamalloo, E., Potthast, M., Hagen, M., & Lin, J. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657861 · Cited by 29
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 15
SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
Xu, J., Pan, J., Zhou, Y., Chen, S., Li, J., Lian, Y., Wu, J., & Dai, G. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3730996 · Cited by 31
LLM-based Query Expansion Fails for Unfamiliar and Ambiguous Queries
Abe, K., Takeoka, K., Kato, M. P., & Oyamada, M. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730222 · Cited by 17
Efficient Re-ranking with Cross-encoders via Early Exit
Busolin, F., Lucchese, C., Nardini, F. M., Orlando, S., Perego, R., Trani, S., & Veneri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729962 · Cited by 14
Towards large language models with human-like episodic memory
Dong, C., Lu, Q., Norman, K. A., & Michelmann, S. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.06.016 · Cited by 28
CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks
Guo, J., Zhou, C., Zhang, R., Chen, J., de Rijke, M., Fan, Y., & Cheng, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3763233 · Cited by 14
Dynamic Batch Processing with FlexiDecode Scheduler for Efficient LLM Inference in IIoT
Jia, X., Gu, B., Chen, J.-J., Gao, L.-X., Pang, W., Lv, G., Qu, Y., & Cui, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020025 · Cited by 6
Song, J., Ashktorab, Z., Pan, Q., Dugan, C., Geyer, W., & Malone, T. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757486 · Cited by 11
Mucciaccia, S. S., Paixão, T. M., Mutz, F. W., Souza, A. F. D., Badue, C. S., & Oliveira-Santos, T. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5801 · Cited by 11
Leveraging historical information to boost retrieval-augmented generation in conversations
Mo, F., Gao, Y., Wu, Z., Liu, X., Chen, P., Li, Z., Wang, Z., Li, X., Jiang, M., & Nie, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104449 · Cited by 13
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
Huang, K., Wu, H., Shi, Z., Zou, H., Yu, M., & Shi, Q. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772239 · Cited by 35
Secure Retrieval-Augmented Generation Against Poisoning Attacks
Cheng, Z., Sun, J., Gao, A., Quan, Y., Liu, Z., Hu, X. T., & Fang, M. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402647 · Cited by 23
Cross-layer Attention Sharing for Pre-trained Large Language Models
Mu, Y., Wu, Y., Fan, Y., Wang, C. Q., Li, H., Zeng, J., He, Q., Yang, M., Meng, F., Zhou, J., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.616 · Cited by 27
XCMRC: Evaluating Cross-Lingual Machine Reading Comprehension
Liu, P., Deng, Y., Zhu, C., & Hu, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_43 · Cited by 9
FarsNewsQA: a deep learning-based question answering system for the Persian news articles
Kazemi, A., Zojaji, Z., Malverdi, M., Mozafari, J., Ebrahimi, F., Abadani, N., Varasteh, M. R., & Nematbakhsh, M. A. (2023) · Information Retrieval · doi:10.1007/s10791-023-09417-2 · Cited by 10
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Won’t Get Fooled Again: Answering Questions with False Premises
Hu, S., Luo, Y., Wang, H., Cheng, X., Liu, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.309 · Cited by 75
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 33
RFiD: Towards Rational Fusion-in-Decoder for Open-Domain Question Answering
Wang, C., Yu, H., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.155 · Cited by 16
Natural Language Deduction with Incomplete Information
Sprague, Z., Bostrom, K., Chaudhuri, S., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.564 · Cited by 26
Automated extraction of domain knowledge in the dairy industry
Zhu, J., Lacroix, R., & Wade, K. M. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108330 · Cited by 11
A Comprehensive Survey On Efficient Transformers
Elouargui, Y., Zyate, M., Sassioui, A., Chergui, M., Kamili, M. E., & Ouzzif, M. (2023) · International Conference on Wireless Networks and Mobile Communications (WINCOM) · doi:10.1109/wincom59760.2023.10322921 · Cited by 9
Expand, Highlight, Generate: RL-driven Document Generation for Passage Reranking
Askari, A., Aliannejadi, M., Meng, C., Kanoulas, E., & Verberne, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.623 · Cited by 25
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks
Sun, K., Qi, P., Zhang, Y., Liu, L., Wang, W. Y., & Huang, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.887 · Cited by 22
Chen, Z., Sun, X., Jiao, X., Lian, F., Kang, Z., Wang, D., & Xu, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30087 · Cited by 21
Hoque, M. N., Mahfuz, A. A., Kindi, M. S., & Hassan, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642007 · Cited by 17
Fairness Definitions in Language Models Explained
Yin, Z., Wang, Z., Palikhe, A., & Zhang, W. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70063 · Cited by 27
Survey of Large Language Models for Answering Questions Across Various Fields
Shailendra, P., Ghosh, R. C., Kumar, R., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717078 · Cited by 24
Online Safety Analysis for LLMs: A Benchmark, an Assessment, and a Path Forward
Xie, X., Song, J., Zhou, Z., Huang, Y., Song, D., & Ma, L. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3603547 · Cited by 17
Open Korean Corpora: A Practical Report
Cho, W. I., Moon, S., & Song, Y. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.12 · Cited by 13
Shinoda, K., Sugawara, S., & Aizawa, A. (2021) · doi:10.18653/v1/2021.acl-srw.21 · Cited by 10
KerasBERT: Modeling the Keras Language
Shorten, C., & Khoshgoftaar, T. M. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00041 · Cited by 9
Can Question Rewriting Help Conversational Question Answering?
Ishii, E., Xu, Y., Cahyawijaya, S., & Wilie, B. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.13 · Cited by 11
Metadata
- Authors
- Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., & Petrov, S.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Kwiatkowski, Tom
- ORCID
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2019-11
- Volume
- 7
- Pages
- 453-466
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2912924812
- Cited by
- 2,069
- References
- 34
- Citations by year
- 2026: 262 · 2025: 322 · 2024: 232 · 2023: 364 · 2022: 303 · 2021: 335 · 2020: 193 · 2019: 58
- FWCI
- 101.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, benchmark dataset, Google search queries, annotation variability, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.