The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Open accessAbstract
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples.TriviaQA includes 95K questionanswer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions.We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers.We also present two baseline algorithms: a featurebased classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension.Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that Trivi-aQA is a challenging testbed that is worth significant future study. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017;Yang et al., 2018;Thorne et al., 2018;Dinan et al., 2019;Fan et al., 2019;Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become saturated by recent language models.”
From BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these benchmarks primarily feature single-hop questions, which typically do not require multi-step reasoning or iterative retrieval.”
From BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“On TriviaQA (Joshi et al., 2017) we show results for both the filtered (previously used in retrieval and open-book work) and unfiltered set (previously used in large language model evaluations). In both cases, Chinchilla substantially out performs Gopher .”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Joshi et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [24].”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measured
In the citing paper
“LLMs perform better on nearly all datasets, such as on NaturalQuestions [62], WebQuestions [10], and TriviaQA [56].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 5,022
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,333
Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., & Weld, D. S. (2011) · Cited by 878
Self-citation
Research works printed but not linked: 7
- Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Lan- guage Processing and Computational Natural Lan- guage Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1290-1301. http://www.aclweb.org/anthology/D12-1118.
- Danqi Chen, Jason Bolton, and Christopher D. Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Pro- ceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2358-2367. http://www.aclweb.org/anthology/P16-1223.
- Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gim- pel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2230-2235. https://aclweb.org/anthology/D16- 1241.
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, San- dro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse con- text. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 1525- 1534. http://www.aclweb.org/anthology/P16-1144.
- Panupong Pasupat and Percy Liang. 2015. Com- positional semantic parsing on semi-structured ta- bles. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Confer- ence on Natural Language Processing of the Asian Federation of Natural Language Process- ing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pages 1470-1480. http://aclweb.org/anthology/P/P15/P15-1142.pdf.
- Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Beijing, China, pages 700-706. http://www.aclweb.org/anthology/P15-2115.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierar- chical attention networks for document classifica- tion. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin- guistics, San Diego, California, pages 1480-1489. http://www.aclweb.org/anthology/N16-1174.
Cited by
Sorted by citations
Items 201–300 of 1,088
Open Question Answering over Tables and TextPreprint
Chen, W., Chang, M.-W., Schlinger, E., Wang, W. Y., & Cohen, W. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10439 · Cited by 151
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., d'Autume, C. D. M., Kocisky, T., Ruder, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01951 · Cited by 90
A Review on Question Generation from Natural Language Text
Zhang, R., Guo, J., Chen, L., Fan, Y., & Cheng, X. (2021) · ACM Transactions on Information Systems · doi:10.1145/3468889 · Cited by 102
Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E. J., & Fung, P. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.14 · Cited by 108
Li, J., Liu, M., Kan, M., Zheng, Z., Wang, Z., Lei, W., Liu, T., & Qin, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.238 · Cited by 96
LearningQ: A Large-Scale Dataset for Educational Question Generation
Chen, G., Yang, J., Hauff, C., & Houben, G.-J. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.14987 · Cited by 83
Jiang, Y., & Bansal, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1262 · Cited by 92
Understanding Dataset Design Choices for Multi-hop Reasoning
Chen, J., & Durrett, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1405 · Cited by 82
Coarse-to-Fine Query Focused Multi-Document Summarization
Xu, Y., & Lapata, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.296 · Cited by 86
Oğuz, B., Chen, X., Karpukhin, V., Peshterliev, S., Okhonko, D., Schlichtkrull, M., Gupta, S., Mehdad, Y., & Yih, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.115 · Cited by 89
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Zhao, T., Yan, Z., Cao, Y., & Li, Z. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/546 · Cited by 78
Episodic Memory in Lifelong Language Learning
d'Autume, C. D. M., Ruder, S., Kong, L., & Yogatama, D. (2019) · arXiv · doi:10.48550/arxiv.1906.01076 · Cited by 95
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1170 · Cited by 78
CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
Duan, X., Wang, B., Wang, Z., Ma, W., Cui, Y., Wu, D., Wang, S., Liu, T., Huo, T., Hu, Z., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_36 · Cited by 79
Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 94
Few-Shot Question Answering by Pretraining Span Selection
Ram, O., Kirstain, Y., Berant, J., Globerson, A., & Levy, O. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.239 · Cited by 95
ASQA: Factoid Questions Meet Long-Form Answers
Stelmakh, I., Luan, Y., Dhingra, B., & Chang, M.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.566 · Cited by 176
Knowledge Guided Text Retrieval and Reading for Open Domain Question AnsweringPreprint
Min, S., Chen, D., Zettlemoyer, L., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03868 · Cited by 111
Self-citationExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 132
Understanding Emergent Abilities of Language Models from the Loss Perspective
Du, Z., Zeng, A., Dong, Y., & Tang, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1683 · Cited by 76
Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation
Elangovan, A., He, J., & Verspoor, K. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.113 · Cited by 75
Algorithms, data, and platforms: the diverse challenges of governing AI
Nitzberg, M., & Zysman, J. (2022) · Journal of European Public Policy · doi:10.1080/13501763.2022.2096668 · Cited by 89
Do Large Language Models Know What They Don’t Know?
Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., & Huang, X. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.551 · Cited by 75
R$^3$: Reinforced Reader-Ranker for Open-Domain Question AnsweringPreprint
Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G. J., Zhou, B., & Jiang, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00023 · Cited by 75
Question answering by reasoning across documents with graph convolutional networks
De Cao, N., Aziz, W., & Titov, I. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1240 · Cited by 54
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 158
The Effect of Natural Distribution Shift on Question Answering Models
Miller, J., Krauth, K., Recht, B., & Schmidt, L. (2020) · arXiv · doi:10.48550/arxiv.2004.14444 · Cited by 74
Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., & Ji, H. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.15 · Cited by 74
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73
Multi-style Generative Reading Comprehension
Nishida, K., Saito, I., Nishida, K., Shinoda, K., Otsuka, A., Asano, H., & Tomita, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1220 · Cited by 81
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00494 · Cited by 72
Relevance-guided Supervision for OpenQA with ColBERT
Khattab, O., Potts, C., & Zaharia, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00405 · Cited by 81
ScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 86
Shuster, K., Komeili, M., Adolphs, L., Roller, S., Szlam, A. D., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.27 · Cited by 118
Reducing conversational agents' overconfidence through linguistic calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2020) · arXiv · doi:10.48550/arxiv.2012.14983 · Cited by 72
Learning Dense Representations of Phrases at Scale
Lee, J., Sung, M., Kang, J., & Chen, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.518 · Cited by 105
Improving Passage Retrieval with Zero-Shot Question Generation
Sachan, D., Lewis, M., Joshi, M., Aghajanyan, A., Yih, W.-T., Pineau, J., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.249 · Cited by 90
Self-citationGeneralizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5827 · Cited by 76
A Survey of Text Watermarking in the Era of Large Language Models
Liu, A., Pan, L., Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3691626 · Cited by 138
Neural Speed Reading via Skim-RNNPreprint
Seo, M., Min, S., Farhadi, A., & Hajishirzi, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02085 · Cited by 77
SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Efimov, P., Chertok, A., Boytsov, L., & Braslavski, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58219-7_1 · Cited by 74
Open-Domain Question Answering
Chen, D., & Yih, W.-T. (2020) · doi:10.18653/v1/2020.acl-tutorials.8 · Cited by 67
Universal Information Extraction as Unified Semantic Matching
Lou, J., Lu, Y., Dai, D., Jia, W., Lin, H., Han, X., Sun, L., & Wu, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26563 · Cited by 87
Natural language processing in finance: A survey
Du, K., Zhao, Y., Mao, R., Xing, F., & Cambria, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102755 · Cited by 92
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 73
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
Sachan, D. S., Patwary, M. A., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., & Catanzaro, B. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.519 · Cited by 88
Dynamic Integration of Background Knowledge in Neural NLU SystemsPreprint
Weissenborn, D., Kočiský, T., & Dyer, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02596 · Cited by 77
REPLUG: Retrieval-augmented black-box language models
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2023) · arXiv · doi:10.48550/arxiv.2301.12652 · Cited by 65
Self-citationLogic-Guided Data Augmentation and Regularization for Consistent Question Answering
Asai, A., & Hajishirzi, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.499 · Cited by 81
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,339
Generative Question Answering: Learning to Answer the Whole Question.
Lewis, M., & Fan, A. (2018) · International Conference on Learning Representations · Cited by 60
Recent Trends in Deep Learning Based Open-Domain Textual Question Answering Systems
Huang, Z., Xu, S., Hu, M., Wang, X., Qiu, J., Fu, Y., Zhao, Y., Peng, Y., & Wang, C. (2020) · IEEE Access · doi:10.1109/access.2020.2988903 · Cited by 69
Question Answering with Long Multiple-Span Answers
Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., & Reddy, C. K. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.342 · Cited by 68
Lee, D. B., Lee, S., Jeong, W. T., Kim, D., & Hwang, S. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.20 · Cited by 69
QA-RAG: Exploring LLM Reliance on External Knowledge
Mansurova, A., Mansurova, A., & Nugumanova, A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8090115 · Cited by 67
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, S., Liu, X., & Jin, X. (2026) · ACM Transactions on Computer Systems · doi:10.1145/3768628 · Cited by 61
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Jin, C., Zhang, Z., Jiang, X., Liu, F., Liu, X., Liu, X., & Jin, X. (2024) · arXiv · doi:10.48550/arxiv.2404.12457 · Cited by 61
To Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 68
Explanations for CommonsenseQA: New Dataset and Models
Aggarwal, S., Mandowara, D., Agrawal, V., Khandelwal, D., Singla, P., & Garg, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.238 · Cited by 85
KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering
Yu, D., Zhu, C., Fang, Y., Yu, W., Wang, S., Xu, Y., Ren, X., Yang, Y., & Zeng, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.340 · Cited by 72
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60
Efficient Passage Retrieval with Hashing for Open-domain Question Answering
Yamada, I., Asai, A., & Hajishirzi, H. (2021) · doi:10.18653/v1/2021.acl-short.123 · Cited by 88
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117
“As a first step toward investigating this question, we apply ITI—using the activation shift directions and hyperparameters learned from TruthfulQA—to two different data sets that relate to real-world truth: Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
MethodSimple and Effective Semi-Supervised Question Answering
Dhingra, B., Danish, D., & Rajagopal, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2092 · Cited by 58
FriendsQA: Open-Domain Question Answering on TV Show Transcripts
Yang, Z., & Choi, J. D. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5923 · Cited by 60
Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Zheng, B., Wen, H., Liang, Y., Duan, N., Che, W., Jiang, D., Zhou, M., & Liu, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.599 · Cited by 55
MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00595 · Cited by 87
MEMEN: Multi-layer Embedding with Memory Networks for Machine ComprehensionPreprint
Pan, B., Li, H., Zhao, Z., Cao, B., Cai, D., & He, X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09098 · Cited by 58
QuAC : Question Answering in ContextPreprint
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07036 · Cited by 209
Self-citationCRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
“Datasets and Metrics We experiment with three datasets: AmbigNQ (Min et al., 2020), an enhanced version of Natural Question (Kwiatkowski et al., 2019) that employs multi-reference annotations to resolve ambiguity, along with TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018).”
BackgroundInfluentialCommonsense Properties from Query Logs and Question Answering Forums
Romero, J., Razniewski, S., Pal, K., Pan, J. Z., Sakhadeo, A., & Weikum, G. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357955 · Cited by 57
Careful Selection of Knowledge to Solve Open Book Question Answering
Banerjee, P., Pal, K. K., Mitra, A., & Baral, C. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1615 · Cited by 56
Greco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 87
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., & Chang, M.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.925 · Cited by 184
CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
Chen, J., Zhang, R., Guo, J., Liu, Y., Fan, Y., & Cheng, X. (2022) · Proceedings of the 31st ACM International Conference on Information & Knowledge Management · doi:10.1145/3511808.3557271 · Cited by 57
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
Yatskar, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1241 · Cited by 60
Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Hu, M., Peng, Y., Huang, Z., & Li, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1221 · Cited by 59
Coarse-grain Fine-grain Coattention Network for Multi-evidence Question AnsweringPreprint
Zhong, V., Xiong, C., Keskar, N. S., & Socher, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00603 · Cited by 65
Cao, Y. H., Fang, M., & Tao, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04969 · Cited by 56
Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
Jiang, Y., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1455 · Cited by 59
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?
Wang, C., Liu, P., & Zhang, Y. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.251 · Cited by 55
SKILL: Structured Knowledge Infusion for Large Language Models
Moiseev, F., Dong, Z., Alfonseca, E., & Jaggi, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.113 · Cited by 71
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 67
Multilingual Extractive Reading Comprehension by Runtime Machine TranslationPreprint
Asai, A., Eriguchi, A., Hashimoto, K., & Tsuruoka, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03275 · Cited by 59
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question AnsweringPreprint
Sachan, D. S., Reddy, S., Hamilton, W. L., Dyer, C., & Yogatama, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05346 · Cited by 71
Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., & Wei, H. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736569 · Cited by 53
“Uncertainty quantification (UQ) is often evaluated as a binary clas-sification task, the rationale being that high uncertainty shouldcor-respond to low expected accuracy.”
Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs
(376528), X. L., (20096703), S. P., (20096706), R. R., (20096709), A. A., (3824542), Y. W., & Weikum, G. (2019) · Figshare · Cited by 53
Interactive Question Answering Systems: Literature Review
Biancofiore, G. M., Deldjoo, Y., Noia, T. D., Sciascio, E. D., & Narducci, F. (2024) · ACM Computing Surveys · doi:10.1145/3657631 · Cited by 61
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 700
“We also use TriviaQA (Joshi et al., 2017) as a closed-book QA problem (in which the model must answer a question without access to a supporting paragraph).”
MethodArabic machine reading comprehension on the Holy Qur’an using CL-AraBERT
Malhas, R., & Elsayed, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103068 · Cited by 64
Look at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 80
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 81
Reinforced Mnemonic Reader for Machine ComprehensionPreprint
Hu, M., Peng, Y., & Qiu, X. (2017) · arXiv (Cornell University) · Cited by 54
Question Directed Graph Attention Network for Numerical Reasoning over Text
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.549 · Cited by 56
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
Retrieval-augmented Generation across Heterogeneous Knowledge
Yu, W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.7 · Cited by 62
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 62
Improved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News
Deka, P., Jurek, A., & Deepak, P. (2022) · Journal of Data Intelligence · doi:10.26421/jdi3.4-5 · Cited by 58
Pilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 75
Metadata
- Authors
- Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1601-1611
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2612431505
- Cited by
- 1,690
- References
- 21
- Citations by year
- 2026: 141 · 2025: 205 · 2024: 157 · 2023: 220 · 2022: 204 · 2021: 259 · 2020: 198 · 2019: 190 · 2018: 97 · 2017: 19
- FWCI
- 61.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- IEEE DataPort (cc-by)
- Funders
- Tencent
- Defense Advanced Research Projects Agency (FA8750-13-2-0019, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- TriviaQA, reading comprehension, distant supervision, trivia questions, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
