The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Open accessAbstract
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples.TriviaQA includes 95K questionanswer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions.We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers.We also present two baseline algorithms: a featurebased classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension.Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that Trivi-aQA is a challenging testbed that is worth significant future study. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017;Yang et al., 2018;Thorne et al., 2018;Dinan et al., 2019;Fan et al., 2019;Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become saturated by recent language models.”
From BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these benchmarks primarily feature single-hop questions, which typically do not require multi-step reasoning or iterative retrieval.”
From BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“On TriviaQA (Joshi et al., 2017) we show results for both the filtered (previously used in retrieval and open-book work) and unfiltered set (previously used in large language model evaluations). In both cases, Chinchilla substantially out performs Gopher .”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Joshi et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [24].”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measured
In the citing paper
“LLMs perform better on nearly all datasets, such as on NaturalQuestions [62], WebQuestions [10], and TriviaQA [56].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 5,022
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,333
Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., & Weld, D. S. (2011) · Cited by 878
Self-citation
Research works printed but not linked: 7
- Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Lan- guage Processing and Computational Natural Lan- guage Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1290-1301. http://www.aclweb.org/anthology/D12-1118.
- Danqi Chen, Jason Bolton, and Christopher D. Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Pro- ceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2358-2367. http://www.aclweb.org/anthology/P16-1223.
- Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gim- pel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2230-2235. https://aclweb.org/anthology/D16- 1241.
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, San- dro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse con- text. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 1525- 1534. http://www.aclweb.org/anthology/P16-1144.
- Panupong Pasupat and Percy Liang. 2015. Com- positional semantic parsing on semi-structured ta- bles. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Confer- ence on Natural Language Processing of the Asian Federation of Natural Language Process- ing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pages 1470-1480. http://aclweb.org/anthology/P/P15/P15-1142.pdf.
- Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Beijing, China, pages 700-706. http://www.aclweb.org/anthology/P15-2115.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierar- chical attention networks for document classifica- tion. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin- guistics, San Diego, California, pages 1480-1489. http://www.aclweb.org/anthology/N16-1174.
Cited by
Sorted by citations
Items 1,001–1,088 of 1,088
Weakly-Supervised Question Answering with Effective Rank and Weighted Loss over Candidates
Qin, H., Zhu, J., & Shen, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449919 · Cited by 4
Representing Words in Vector Space and Beyond
Wang, B., Buccio, E. D., & Melucci, M. (2019) · STEAM-H: Science, Technology, Engineering, Agriculture, Mathematics & Health · doi:10.1007/978-3-030-25913-6_5 · Cited by 4
Semantics-Aware Inferential Network for Natural Language Understanding
Zhang, S., Zhao, H., Zhou, J., Zhou, X., & Zhou, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17697 · Cited by 4
FastFusionNet: New State-of-the-Art for DAWNBench SQuADPreprint
Wu, F., Li, B., Wang, L., Lao, N., Blitzer, J. C., & Weinberger, K. Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11291 · Cited by 5
Towards a Better Metric for Evaluating Question Generation SystemsPreprint
Nema, P., & Khapra, M. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10192 · Cited by 4
Reasoning-Driven Question-Answering for Natural Language UnderstandingPreprint
Khashabi, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04926 · Cited by 5
Self-Assembling Modular Networks for Interpretable Multi-Hop ReasoningPreprint
Jiang, Y., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05803 · Cited by 12
Generating Fact Checking BriefsPreprint
Fan, A., Piktus, A., Petroni, F., Wenzek, G., Saeidi, M., Vlachos, A., Bordes, A., & Riedel, S. (2020) · arXiv (Cornell University) · doi:10.17863/cam.78760 · Cited by 4
Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?Preprint
Wang, C., Liu, P., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01561 · Cited by 4
Bian, N., Han, X., Chen, B., Lin, H., He, B., & Sun, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.08582 · Cited by 5
Knowledge Graph Question Answering via SPARQL Silhouette GenerationPreprint
Purkayastha, S., Dana, S., Garg, D., Khandelwal, D., & Bhargav, G. P. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09475 · Cited by 7
FQuAD2.0: French Question Answering and knowing that you know nothingPreprint
Heinrich, Q., Viaud, G., & Belblidia, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13209 · Cited by 7
COVIDRead: A Large-scale Question Answering Dataset on COVID-19Preprint
Saikh, T., Sahoo, S. K., Ekbal, A., & Bhattacharyya, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09321 · Cited by 6
Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
Xia, T., & Zhang, S. Q. (2025) · Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture · doi:10.1145/3725843.3756071 · Cited by 3
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
DeepSeek-AI, Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.19348 · Cited by 4
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Liu, A., Green, J., Patel, K., Meng, R., Su, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.06600 · Cited by 3
“Traditional benchmarks such as NaturalQuestions [27] and TriviaQA [28] have significantly contributed to evaluating retrieval and retrieval-augmented generation systems [2, 15, 29].”
A Brief Review on Question Answering System: Techniques, Challenges and Future Directions
Kumar, R., Singh, G. B., Shailendra, P., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717204 · Cited by 3
VAP3: Variation-Aware Prompt Performance Prediction
Arabzadeh, N., & Ebrahim Bagheri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730264 · Cited by 3
“Constructed from HotpotQA and TriviaQA benchmarks [39, 63], PromptSET comprises a diverse collection of short-form question-answer pairs.”
Less LLM, More Documents: Searching for Improved RAG
Ning, J., Kong, Y., Long, Y., & Callan, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_38 · Cited by 9
Lee, G., Kim, S., Lee, D., Goh, K., & Kim, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108900 · Cited by 4
Lite Unified Modeling for Discriminative Reading Comprehension
Zhao, Y., Zhao, H., Shen, L., & Zhao, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.594 · Cited by 4
Two-Step Question Retrieval for Open-Domain QA
Seonwoo, Y., Son, J., Jin, J., Lee, S., Kim, J., Ha, J.-W., & Oh, A. H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.117 · Cited by 3
Deng, Z., Zhu, Y., Qi, Q., Witbrock, M., & Riddle, P. (2022) · doi:10.18653/v1/2022.dlg4nlp-1.8 · Cited by 3
RobustQA: Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering
Han, R., Qi, P., Zhang, Y., Liu, L., Burger, J., Wang, W. Y., Huang, Z., Xiang, B., & Roth, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.263 · Cited by 5
Named Entity Recognition Based on Boundary Enhanced for Chinese Electronic Medical Records
Chen, X., Liu, T., Tang, G., & Liu, Y. (2023) · doi:10.1109/ictech58362.2023.00025 · Cited by 3
Auto311: A Confidence-Guided Automated System for Non-emergency Calls
Chen, Z., Sun, X., Li, Y., & Ma, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30199 · Cited by 5
Weight Averaging and re-adjustment ensemble for QRCD
Aftab, E., & Malik, M. K. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102037 · Cited by 3
CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
Feng, Y., Rahman, S., Feng, A., Chen, V. S., & Kandogan, E. (2024) · doi:10.1145/3665601.3669846 · Cited by 5
Question Generation Capabilities of “Small" Large Language Models
Berger, J., Koß, J., Stamatakis, M., Hoppe, A., Ewerth, R., & Wartena, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_18 · Cited by 4
SyncIntellects: Orchestrating LLM Inference with Progressive Prediction and QoS-Friendly Control
Lin, X., Zhang, Z., Yue, P., Li, H., Zhang, J., Fan, B., Su, H., & Gong, X. (2024) · International Workshop on Quality of Service · doi:10.1109/iwqos61813.2024.10682949 · Cited by 3
GARAG: A General adaptive question-answering system based on RAG
Wei, Z., Huang, D., Zhang, J., Song, C., Zhang, S., Zhang, J., Li, Z., Jiang, K., Li, R., & Duan, Q. (2024) · doi:10.1145/3695080.3695156 · Cited by 4
Park, S.-I., & Lee, J.-Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00724 · Cited by 9
CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation
He, H., Liu, Q., Xu, L., Shivade, C., Zhang, Y., Srinivasan, S., & Kirchhoff, K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34575 · Cited by 8
A Few‐Shot Learning Approach for a Multilingual Agro‐Information Question Answering System
Banda, F. E., Marivate, V. N., & Nabende, J. N. (2025) · Applied AI Letters · doi:10.1002/ail2.122 · Cited by 4
SensorQA: A Question Answering Benchmark for Daily-Life Monitoring
Reichman, B., Yu, X., Hu, L., Truxal, J., Jain, A., Chandrupatla, R., Rosing, T., & Heck, L. P. (2025) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/3715014.3722074 · Cited by 6
Summary-Oriented Question Generation for Informational Queries
Yin, X., Zhou, L., Small, K., & May, J. (2021) · doi:10.18653/v1/2021.dialdoc-1.11 · Cited by 4
Aksu, T., Liu, Z., Kan, M., & Chen, N. F. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.14 · Cited by 3
Yu, S., & Lu, Y. (2021) · Bridging human and machine: future education with intelligence · doi:10.1007/978-981-16-2770-5_8 · Cited by 3
A Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU
Li, H., Choi, J., & Ahn, J. H. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00019 · Cited by 3
Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
Castillo-Bolado, D., Davidson, J., Gray, F., & Rosa, M. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1347 · Cited by 3
MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources
Lee, D., Satya Prakash, C., FitzGerald, J., & Lehmann, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.953 · Cited by 3
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
Cai, Z., Zhang, Y., Gao, B., Liu, Y., Li, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Hu, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.02069 · Cited by 2
“It includes 17 datasets covering tasks such as single-document QA [20, 6], multi-document QA [33, 16], summarization [17, 35, 10], few-shot learning [21, 13, 19], synthetic, and code generation [14, 24].”
BackgroundQuantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2
“However, for some benchmarks, such as HellaSwag and TriviaQA, the difference is minimal, likely since these benchmarks saturate earlier in training.”
InfluentialBrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Sharifymoghaddam, S., Liu, A., Green, J., Patel, K., Meng, R., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1023 · Cited by 2
“Traditional benchmarks such as NaturalQues-tions (Kwiatkowski et al., 2019) and Trivi-aQA (Joshi et al., 2017) have significantly contributed to evaluating retrieval and retrieval-augmented generation systems (Lewis et al., 2020; Karpukhin et al., 2020; Lin et al., 2024).”
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Wei, J., Sun, Z., Papay, S., McKinney, S., Han, J., Fulford, I., Chung, H. W., Passos, A. T., Fedus, W., & Glaese, A. (2025) · arXiv · doi:10.48550/arxiv.2504.12516 · Cited by 2
“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017; Yang et al., 2018; Thorne et al., 2018; Dinan et al., 2019; Fan et al., 2019; Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become…”
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Murty, S., Manning, C., Shaw, P., Joshi, M., & Lee, K. (2024) · arXiv · doi:10.48550/arxiv.2403.08140 · Cited by 2
Self-citationOLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00656 · Cited by 2
Self-citationMemento: Fine-tuning LLM Agents without Fine-tuning LLMs
Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.16153 · Cited by 2
“…(Zheng et al., 2025), which draws from seven open-domain QA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQ) (Joshi et al., 2017), HotpotQA (Yang et al., 2018), 2Wiki (Ho et al., 2020), MusiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2022), and PopQA…”
InfluentialMiMo: Unlocking the Reasoning Potential of Language Model - From Pretraining to Posttraining
Xiaomi, L.-C., :, Xia, B., Shen, B., Cici, Zhu, D., Zhang, D., Wang, G., Zhang, H., Liu, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.07608 · Cited by 1
“TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019).”
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2
Naser, M. (2026) · arXiv · doi:10.48550/arxiv.2603.03299 · Cited by 1
“From the lens of benchmarking, notable benchmarks such as TruthfulQA [16], TriviaQA [17], and others, evaluate factual accuracy across diverse question types, but none include a citation-specific evaluation component.”
Berges, V.-P., Oğuz, B., Haziza, D., Yih, W.-T., Zettlemoyer, L., & Ghosh, G. (2024) · arXiv · doi:10.48550/arxiv.2412.09764 · Cited by 46
“Our evaluations cover factual question answering (NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017)), multi-hop question answering (HotpotQA (Yang et al., 2018)), scientific and common sense world knowledge (MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019),…”
Self-citationSimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Haas, L., Yona, G., D'Antonio, G., Goldshtein, S., & Das, D. (2025) · arXiv · doi:10.48550/arxiv.2509.07968 · Cited by 1
“SimpleQA (Wei et al., 2024a) was created as a challenging factuality benchmark in response to older short-answer factuality benchmarks like TriviaQA (Joshi et al., 2017) and Natural Questions (Kwiatkowski et al., 2019) becoming saturated.”
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1
“Thakur et al. (2025) studied whether LLMs can judge correctness on 400 short-answer questions from Triv-iaQA (Joshi et al., 2017).”
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1
“The dataset and evaluation landscape has progressed from standardized, static evaluations [22, 28, 23, 70] toward more challenging settings such as deep reasoning [35, 30, 68], coding [27, 26], and open-ended dialogue under crowd-sourced evaluation [10, 70].”
Mamba-3: Improved Sequence Modeling using State Space Principles
Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., & Gu, A. (2026) · arXiv · doi:10.48550/arxiv.2603.15569 · Cited by 1
“…( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐵 , U ) C = contract ( 𝐷𝑁𝑅, 𝐷 → 𝑁𝑅 )( W 𝐶 , U ) , Intermediate output Y ′ is obtained by the residual function 𝜙 , Y ′ ← 𝜙 ( Y 𝑡 , Z ) , where 𝜙 ( Y 𝑡 , Z ) : = Y ⊙ SiLU ( Z 𝑡 ) in Finally, the layer output O ∈ R is computed via the following down projections:”
MiMo-V2-Flash Technical Report
Core Team, Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.02780 · Cited by 1
“We evaluate MiMo-V2-Flash-Base on a series of benchmarks, encompassing various capabilities: (1) General language understanding and reasoning, including BBH (Suzgun et al., 2023), MMLU (Hendrycks et al., 2021a), MMLU-Redux (Gema et al., 2024), MMLU-Pro (Wang et al., 2024), DROP (Dua et al., 2019), ARC (Clark et al., 2…”
Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
Du, Y., Huang, W., Zheng, D., Wang, Z., Montella, S., Lapata, M., Wong, K.-F., & Pan, J. Z. (2025) · arXiv · doi:10.48550/arxiv.2505.00675 · Cited by 1
Gated Delta Networks: Improving Mamba2 with Delta Rule
Yang, S., Kautz, J., & Hatamizadeh, A. (2024) · arXiv · doi:10.48550/arxiv.2412.06464 · Cited by 1
TaSR-RAG: Taxonomy-guided Structured Reasoning for Retrieval-Augmented Generation
Sun, J., Xie, Y., Shi, J., Wang, S., & Han, J. (2026) · arXiv · doi:10.48550/arxiv.2603.09341
“For single-step retrieval, we utilize Natural Questions (NQ) [8], TriviaQA [7], and PopQA [14].”
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“The suite includes SWDE [46] for structured relation extraction from HTML, FDA [47] for key-value retrieval from PDFs, and question-answering datasets including SQuAD [48], TriviaQA [49], DROP [50], and Natural Questions [51].”
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
WideSearch: Benchmarking Agentic Broad Info-Seeking
Wong, R., Wang, J., Zhao, J., Chen, L., Gao, Y., Zhang, L., Zhou, X., Wang, Z., Xiang, K., Zhang, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07999
“Early benchmarks such as Natural Questions [11] and TriviaQA [10] established a foundation for question answering, but often tested information that could be retrieved with a single query or was already contained within a model’s parametric knowledge.”
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Zhou, P., Leon, B., Ying, X., Zhang, C., Shao, Y., Ye, Q., Chong, D., Jin, Z., Xie, C., Cao, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.19314
“To assess retrieval capabilities, a variety of widely used English benchmarks have been proposed, including TriviaQA Joshi et al. (2017), HotpotQA Yang et al. (2018), FEVER Thorne et al. (2018), KILT Petroni et al. (2020), and GAIA Mialon et al. (2023).”
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
“The task we consider departs significantly from traditional question answering and RAG-based evaluations [22– 24, 26, 28, 43, 52], requiring a carefully designed and holistic evaluation framework.”
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
“MMLU [23], ARC [11], BoolQ [10], CSQA [59], OBQA [39], PiQA [4], SocialIQA [53], HellaSwag [69], WinoGrande [52], DROP [16], CoQA [48], Jeopardy [61], NaturalQs [28], SQuAD [47], TriviaQA [26], MedMCQA [43], MMLU Pro [64], AGI Eval [70], GPQA [49] Math.”
Wanner, M., Azzopardi, L., Thomas, P., Dan, S., Van Durme, B., & Craswell, N. (2025) · arXiv · doi:10.48550/arxiv.2510.07083
“These consisted of commonly used datasets focused on: entities (Min et al., 2023; Zhao et al., 2024), QA (Yang et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017), and complex reasoning (Su et al., 2024).”
EvolKV: Evolutionary KV Cache Compression for LLM Inference
Yu, B., & Chai, Y. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.88
Mitigating Context Interference for Reliable and Efficient Search Agents
Xue, B., Wu, B., Qiao, S., Wang, S., Wang, R., Du, Y., Wang, H., Pan, J. Z., Yilmaz, E., Wong, K.-F., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.160
“Single-hop QA includes: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023).”
DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking
Lan, T., Zhu, B., Jia, Q., Ren, J., Li, H., Wang, L., Xu, Z., Luo, W., & Zhang, K. (2025) · arXiv · doi:10.48550/arxiv.2510.20168
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Tian, C., Wang, J., Zhao, Q., Chen, K., Liu, J., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2025) · arXiv · doi:10.48550/arxiv.2507.17634
“For base model, tasks are grouped into several categories, such as: (a) General Knowledge/Reasoning (e.g., ARC (Bhakthavatsalam et al., 2021), AGIEval (Zhong et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), WorldSense (Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al.…”
InfluentialInstruction Following by Principled Boosting Attention of Large Language Models
Guardieiro, V., Khare, A., Stein, A., & Wong, E. (2025) · arXiv · doi:10.48550/arxiv.2506.13734
Conformity in Large Language Models
Zhu, X., Zhang, C., Stafford, T., Collier, N., & Vlachos, A. (2024) · arXiv · doi:10.48550/arxiv.2410.12428
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
MatKV: Trading Compute for Flash Storage in LLM Inference
Shin, K.-W., Park, J. H., Oh, M., Jo, Y., Do, J., & Lee, S.-W. (2026) · 2026 IEEE 42nd International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00071
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448
Long Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · arXiv · doi:10.48550/arxiv.1903.00161 · Cited by 238
An Explanation of In-context Learning as Implicit Bayesian Inference
Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2111.02080 · Cited by 138
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, E. (2020) · arXiv · doi:10.48550/arxiv.2012.04584 · Cited by 102
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021) · arXiv · doi:10.48550/arxiv.2104.08663 · Cited by 45
A Dataset for Answering Time-Sensitive Questions
Chen, W., Wang, X., & Wang, W. Y. (2021) · arXiv · doi:10.48550/arxiv.2108.06314 · Cited by 40
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
Gonzalez, A. V., Bansal, G., Fan, A., Jia, R., Mehdad, Y., & Iyer, S. (2020) · arXiv · doi:10.48550/arxiv.2012.15075 · Cited by 7
Metadata
- Authors
- Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1601-1611
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2612431505
- Cited by
- 1,690
- References
- 21
- Citations by year
- 2026: 141 · 2025: 205 · 2024: 157 · 2023: 220 · 2022: 204 · 2021: 259 · 2020: 198 · 2019: 190 · 2018: 97 · 2017: 19
- FWCI
- 61.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- IEEE DataPort (cc-by)
- Funders
- Tencent
- Defense Advanced Research Projects Agency (FA8750-13-2-0019, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- TriviaQA, reading comprehension, distant supervision, trivia questions, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
