TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Open accessAbstract
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study. Data and code available at -- http://nlp.cs.washington.edu/triviaqa/
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing benchmarks like NaturalQuestions, TriviaQA, and ELI5 focus on specific aspects but lack comprehensive assessment.
“Existing benchmarks like NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and ELI5 (Fan et al., 2019) focus on specific aspects but lack comprehensive assessment.”
From Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024) · cites this paper as Joshi et al., 2017
TriviaQA emphasizes factual correctness.
“NaturalQuestions tests retrieval precision, TriviaQA emphasizes factual correctness, and ELI5 targets explainability without rigorous multi-hop reasoning evaluation.”
From Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024) · cites this paper as Joshi et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 33
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on free- base from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18- 21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Spe- cial Interest Group of the ACL. pages 1533-1544. http://aclweb.org/anthology/D/D13/D13-1160.pdf.
- Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple ques- tion answering with memory networks. CoRR abs/1506.02075. https://arxiv.org/abs/1506.02075. · arXiv
- Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Lan- guage Processing and Computational Natural Lan- guage Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1290-1301. http://www.aclweb.org/anthology/D12-1118.
- Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In Proceedings of the 51st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computa- tional Linguistics, Sofia, Bulgaria, pages 423-433. http://www.aclweb.org/anthology/P13-1042.
- Danqi Chen, Jason Bolton, and Christopher D. Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Pro- ceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2358-2367. http://www.aclweb.org/anthology/P16-1223.
- Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset aug- mented with context from a search engine. CoRR https://arxiv.org/abs/1704.05179. · arXiv
- Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, USA, KDD '14, pages 1156-1165. https://doi.org/10.1145/2623330.2623677. · DOI
- Paolo Ferragina and Ugo Scaiella. 2010. Tagme: On-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM International Conference on Informa- tion and Knowledge Management. ACM, New York, NY, USA, CIKM '10, pages 1625-1628. https://doi.org/10.1145/1871437.1871689. · DOI
- David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyan- pur, Adam Lally, J. William Murdock, Eric Ny- berg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building watson: An overview of the deepqa project. AI MAGAZINE 31(3):59-79.
- He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. 2016. Opponent modeling in deep reinforcement learning. In Maria Flo- rina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Confer- ence on Machine Learning. PMLR, New York, New York, USA, volume 48 of Proceedings of Machine Learning Research, pages 1804-1813. http://proceedings.mlr.press/v48/he16.html.
- Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teach- ing machines to read and comprehend. In Ad- vances in Neural Information Processing Systems. http://arxiv.org/abs/1506.03340. · arXiv
- Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representa- tions. CoRR https://arxiv.org/abs/1511.02301. · arXiv
- Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceed- ings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin- guistics, Portland, Oregon, USA, pages 541-550. http://www.aclweb.org/anthology/P11-1055.
- Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP). Association for Computa- tional Linguistics, Doha, Qatar, pages 633-644. http://www.aclweb.org/anthology/D14-1070.
- Mandar Joshi, Uma Sawant, and Soumen Chakrabarti. 2014. Knowledge graph and corpus driven segmen- tation and answer inference for telegraphic entity- seeking queries. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computa- tional Linguistics, Doha, Qatar, pages 1104-1114. http://www.aclweb.org/anthology/D14-1117.
- Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. CoRR https://arxiv.org/abs/1704.04683. · arXiv
- Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. CoRR https://arxiv.org/abs/1607.06275. · arXiv
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop in Ad- vances in Neural Information Processing Systems. https://arxiv.org/pdf/1611.09268.pdf. · arXiv
- Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gim- pel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2230-2235. https://aclweb.org/anthology/D16- 1241.
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, San- dro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse con- text. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 1525- 1534. http://www.aclweb.org/anthology/P16-1144.
- Panupong Pasupat and Percy Liang. 2015. Com- positional semantic parsing on semi-structured ta- bles. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Confer- ence on Natural Language Processing of the Asian Federation of Natural Language Process- ing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pages 1470-1480. http://aclweb.org/anthology/P/P15/P15-1142.pdf.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguistics, Austin, Texas, pages 2383-2392. https://aclweb.org/anthology/D16-1264.
- Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguis- tics, Seattle, Washington, USA, pages 193-203. http://www.aclweb.org/anthology/D13-1020.
- Sebastian Riedel, Limin Yao, and Andrew Mc- Callum. 2010. Modeling relations and their mentions without labeled text. In Proceedings of the 2010 European Conference on Ma- chine Learning and Knowledge Discovery in Databases: Part III. Springer-Verlag, Berlin, Heidelberg, ECML PKDD'10, pages 148-163. http://dl.acm.org/citation.cfm?id=1889788.1889799.
- Uma Sawant and Soumen Chakrabarti. 2013. Learn- ing joint query interpretation and response rank- ing. In Proceedings of the 22Nd International Conference on World Wide Web. ACM, New York, NY, USA, WWW '13, pages 1099-1110. https://doi.org/10.1145/2488388.2488484. · DOI
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the International Conference on Learning Represen- tations (ICLR). https://arxiv.org/abs/1611.01603. · arXiv
- Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bach- man, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. CoRR https://arxiv.org/abs/1611.09830. · arXiv
- Ellen M. Voorhees and Dawn M. Tice. 2000. Build- ing a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and De- velopment in Information Retrieval. ACM, New York, NY, USA, SIGIR '00, pages 200-207. https://doi.org/10.1145/345508.345577. · DOI
- Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Beijing, China, pages 700-706. http://www.aclweb.org/anthology/P15-2115.
- Qiang Wu, Christopher J. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for infor- mation retrieval measures. Inf. Retr. 13(3):254-270. https://doi.org/10.1007/s10791-009-9112-1. · DOI
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, at- tend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning. https://arxiv.org/abs/1502.03044. · arXiv
- Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2013-2018. http://aclweb.org/anthology/D15-1237.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierar- chical attention networks for document classifica- tion. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin- guistics, San Diego, California, pages 1480-1489. http://www.aclweb.org/anthology/N16-1174.
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 132 · see citing works
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117 · see citing works
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 52 · see citing works
Effective Long-Context Scaling of Foundation Models
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16039 · Cited by 9 · see citing works
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.01658 · Cited by 5 · see citing works
CiteME: Can Language Models Accurately Cite Scientific Claims?
Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., & Bethge, M. (2024) · arXiv · doi:10.48550/arxiv.2407.12861 · Cited by 4
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3 · see citing works
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Krishna, S., Krishna, K., Mohananey, A., Schwarcz, S., Stambler, A., Upadhyay, S., & Faruqui, M. (2024) · arXiv · doi:10.48550/arxiv.2409.12941 · Cited by 1 · see citing works
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 1 · see citing works
Self-citationConditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Zhang, H., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.226 · Cited by 3 · see citing works
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Zhou, P., Leon, B., Ying, X., Zhang, C., Shao, Y., Ye, Q., Chong, D., Jin, Z., Xie, C., Cao, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.19314
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
- Authors
- Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- Pages
- 1601-1611
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 1705.03551
- OpenAlex
- W2612431505
- Cited by
- 1,650
- References
- 0
- Citations by year
- 2026: 133 · 2025: 205 · 2024: 157 · 2023: 218 · 2022: 193 · 2021: 255 · 2020: 197 · 2019: 186 · 2018: 89 · 2017: 17
- FWCI
- 61.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- IEEE DataPort (cc-by)
- Funders
- Tencent
- Defense Advanced Research Projects Agency (FA8750-13-2-0019, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- TriviaQA, reading comprehension, distant supervision, trivia questions, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
