The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Open accessAbstract
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples.TriviaQA includes 95K questionanswer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions.We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers.We also present two baseline algorithms: a featurebased classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension.Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that Trivi-aQA is a challenging testbed that is worth significant future study. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017;Yang et al., 2018;Thorne et al., 2018;Dinan et al., 2019;Fan et al., 2019;Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become saturated by recent language models.”
From BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these benchmarks primarily feature single-hop questions, which typically do not require multi-step reasoning or iterative retrieval.”
From BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“On TriviaQA (Joshi et al., 2017) we show results for both the filtered (previously used in retrieval and open-book work) and unfiltered set (previously used in large language model evaluations). In both cases, Chinchilla substantially out performs Gopher .”
From Training Compute-Optimal Large Language Models (2022) · cites this paper as Joshi et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [24].”
From Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) · cites this paper as [24]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“One reason to focus on imitative falsehoods is that they are less likely to be covered by existing question-answering benchmarks (Clark et al., 2018; Kwiatkowski et al., 2019; Joshi et al., 2017; Hendrycks et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Joshi et al., 2017
Experimental — accuracy not yet measured
In the citing paper
“LLMs perform better on nearly all datasets, such as on NaturalQuestions [62], WebQuestions [10], and TriviaQA [56].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 5,022
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Building Watson: An Overview of the DeepQA Project
Ferrucci, D. A., Brown, E. W., Chu‐Carroll, J., Fan, J., Gondek, D. C., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J. M., et al. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2303 · Cited by 1,523
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,333
Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations
Hoffmann, R., Zhang, C., Ling, X., Zettlemoyer, L., & Weld, D. S. (2011) · Cited by 878
Self-citation
Research works printed but not linked: 7
- Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daumé III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Con- ference on Empirical Methods in Natural Lan- guage Processing and Computational Natural Lan- guage Learning. Association for Computational Linguistics, Jeju Island, Korea, pages 1290-1301. http://www.aclweb.org/anthology/D12-1118.
- Danqi Chen, Jason Bolton, and Christopher D. Man- ning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Pro- ceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2358-2367. http://www.aclweb.org/anthology/P16-1223.
- Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gim- pel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2230-2235. https://aclweb.org/anthology/D16- 1241.
- Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, San- dro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse con- text. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 1525- 1534. http://www.aclweb.org/anthology/P16-1144.
- Panupong Pasupat and Percy Liang. 2015. Com- positional semantic parsing on semi-structured ta- bles. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Confer- ence on Natural Language Processing of the Asian Federation of Natural Language Process- ing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pages 1470-1480. http://aclweb.org/anthology/P/P15/P15-1142.pdf.
- Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Beijing, China, pages 700-706. http://www.aclweb.org/anthology/P15-2115.
- Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierar- chical attention networks for document classifica- tion. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Lin- guistics, San Diego, California, pages 1480-1489. http://www.aclweb.org/anthology/N16-1174.
Cited by
Sorted by citations
Items 101–200 of 1,088
Coreferential Reasoning Learning for Language Representation
Ye, D., Lin, Y., Du, J., Liu, Z., Li, P., Sun, M., & Liu, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.582 · Cited by 181
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
Harvesting Paragraph-level Question-Answer Pairs from Wikipedia
Du, X., & Cardie, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1177 · Cited by 168
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187
Symbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303
“• NLG: TriviaQA [45], Natural Questions [51], Web Questions [6].”
BackgroundCondenser: a Pre-training Architecture for Dense Retrieval
Gao, L., & Callan, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.75 · Cited by 256
Transformer models used for text-based question answering systems
Nassiri, K., & Akhloufi, M. A. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04052-8 · Cited by 194
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
Evaluating Retrieval Quality in Retrieval-Augmented Generation
Salemi, A., & Zamani, H. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657957 · Cited by 259
Blockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 247
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 213
Neural Machine Reading Comprehension: Methods and Trends
Liu, S., Zhang, X., Zhang, S., Wang, H., & Zhang, W. (2019) · Applied Sciences · doi:10.3390/app9183698 · Cited by 160
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 172
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1226 · Cited by 155
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Hu, M., Wei, F., Peng, Y., Huang, Z., Yang, N., & Li, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016529 · Cited by 147
Open-Retrieval Conversational Question Answering
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., & Iyyer, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401110 · Cited by 163
Neural Arabic Question Answering
Mozannar, H., Maamary, E., Hajal, K. E., & Hajj, H. M. (2019) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/w19-4612 · Cited by 174
Efficient and Robust Question Answering from Minimal Context over Documents
Min, S., Zhong, V., Socher, R., & Xiong, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1160 · Cited by 150
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., & Gardner, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1606 · Cited by 173
Retrieving and Reading: A Comprehensive Survey on Open-domain Question AnsweringPreprint
Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., & Chua, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00774 · Cited by 343
Neural entity linking: A survey of models based on deep learning
Sevgili, Ö., Shelmanov, A., Arkhipov, M., Panchenko, A., & Biemann, C. (2022) · Semantic Web · doi:10.3233/sw-222986 · Cited by 202
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., & Riedel, S. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00415 · Cited by 171
Learning and Evaluating General Linguistic IntelligencePreprint
Yogatama, D., d’Autume, C. D. M., Connor, J. T., Kočiský, T., Chrzanowski, M., Kong, L., Lazaridou, A., Wang, L., Yu, L., Dyer, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11373 · Cited by 212
Generation-Augmented Retrieval for Open-Domain Question Answering
Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.316 · Cited by 146
TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer
Zhou, M., Xu, W., Kang, J., & Rosing, T. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00082 · Cited by 151
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.86 · Cited by 145
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Lu, J., Clark, C., Zellers, R., Mottaghi, R., & Kembhavi, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08916 · Cited by 145
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145
Denoising Distantly Supervised Open-Domain Question Answering
Lin, Y., Ji, H., Liu, Z., & Sun, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1161 · Cited by 143
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question AnsweringPreprint
Wang, S., Yu, M., Jiang, J., Zhang, W., Guo, X., Chang, S., Wang, Z., Klinger, T., Tesauro, G. J., & Campbell, M. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05116 · Cited by 154
A Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 144
Self-citationInterpretation of Natural Language Rules in Conversational Machine Reading
Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., & Riedel, S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1233 · Cited by 141
JEC-QA: A Legal-Domain Question Answering Dataset
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6519 · Cited by 166
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1416 · Cited by 138
Self-citationA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 328
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Mari Ostendorf, E., Yih, W.-T., Smith, N. A., Zettlemoyer, L., & Yu, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.71 · Cited by 190
Self-citationSharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
“Experiment Details We measure the accuracy of AI assistants when challenged on subsets of five question-answering (QA) datasets: (i) MMLU (Hendrycks et al., 2021a); (ii) MATH (Hendrycks et al., 2021b); (iii) AQuA (Ling et al., 2017); (iv) TruthfulQA (Lin et al., 2022); and (v) Trivi-aQA (Joshi et al., 2017).”
Improving Machine Reading Comprehension with General Reading Strategies
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1270 · Cited by 134
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
Nie, Y., Wang, S., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1258 · Cited by 132
Selective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.503 · Cited by 229
Errudite: Scalable, Reproducible, and Testable Error Analysis
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1073 · Cited by 137
Self-citationEntities as Experts: Sparse Memory Access with Entity Supervision
Févry, T., Soares, L. B., FitzGerald, N., Choi, E., & Kwiatkowski, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.400 · Cited by 158
Self-citationTrick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Wallace, E., Rodríguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. L. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00279 · Cited by 139
Training Question Answering Models From Synthetic Data
Puri, R., Spring, R., Shoeybi, M., Patwary, M. A., & Catanzaro, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.468 · Cited by 161
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672274 · Cited by 118
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
Yagcioglu, S., Erdem, A., Erdem, E., & İkizler-Cinbiş, N. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1166 · Cited by 161
Automatic question generation and answer assessment: a survey
Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-021-00151-1 · Cited by 123
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2022) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 145
Large-scale Cloze Test Dataset Created by Teachers
Xie, Q., Lai, G., Dai, Z., & Hovy, E. H. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1257 · Cited by 144
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
Lin, B. Y., He, C., Ze, Z., Wang, H., Hua, Y., Dupuy, C., Gupta, R., Soltanolkotabi, M., Ren, X., & Avestimehr, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.13 · Cited by 138
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 268
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
Longpre, S., Lu, Y., & Daiber, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00433 · Cited by 150
Shi, J., Cao, S., Hou, L., Li, J., & Zhang, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.341 · Cited by 123
Simplified Data Wrangling with ir_datasets
MacAvaney, S., Yates, A., Feldman, S., Downey, D., Cohan, A., & Goharian, N. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463254 · Cited by 127
Semantic Models for the First-Stage Retrieval: A Comprehensive Review
Guo, J., Cai, Y., Fan, Y., Sun, F., Zhang, R., & Cheng, X. (2022) · ACM Transactions on Information Systems · doi:10.1145/3486250 · Cited by 153
Transforming Question Answering Datasets Into Natural Language Inference DatasetsPreprint
Demszky, D., Guu, K., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02922 · Cited by 187
Efficient One-Pass End-to-End Entity Linking for Questions
Li, B. Z., Min, S., Iyer, S. V., Mehdad, Y., & Yih, W.-T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.522 · Cited by 138
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Ostermann, S., Roth, M., Modi, A., Thater, S., & Pinkal, M. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1119 · Cited by 116
Improving the Robustness of Question Answering Systems to Question Paraphrasing
Gan, W. C., & Ng, H. T. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1610 · Cited by 117
FeTaQA: Free-form Table Question Answering
Nan, L., Hsieh, C., Mao, Z., Lin, X., Verma, N., Zhang, R., Kryściński, W., Schoelkopf, H., Kong, R., Tang, X., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00446 · Cited by 150
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112
A Watermark for Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2301.10226 · Cited by 194
DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension
Saha, A., Aralikatte, R., Khapra, M. M., & Sankaranarayanan, K. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1156 · Cited by 125
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.330 · Cited by 123
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesPreprint
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05002 · Cited by 109
Self-citationDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2019) · now publishers, Inc. eBooks · doi:10.1561/9781680835533 · Cited by 125
NumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108
KLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 193
Towards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Zhang, S., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1253 · Cited by 121
What Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1453 · Cited by 111
Dalvi, B. B., Huang, L., Tandon, N., Yih, W.-T., & Clark, P. E. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1144 · Cited by 134
Attentive History Selection for Conversational Question Answering
Qu, C., Yang, L., Qiu, M., Zhang, Y., Chen, C., Croft, W. B., & Iyyer, M. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357905 · Cited by 115
RealFormer: Transformer Likes Residual Attention
He, R., Ravula, A., Kanagal, B., & Ainslie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.81 · Cited by 133
Beat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102
Qin, Y., Lin, Y., Takanobu, R., Liu, Z., Li, P., Ji, H., Huang, M., Sun, M., & Zhou, J. (2021) · doi:10.18653/v1/2021.acl-long.260 · Cited by 125
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1500 · Cited by 116
Automatic Judgment Prediction via Legal Reading Comprehension
Long, S., Tu, C., Liu, Z., & Sun, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32381-3_45 · Cited by 110
A Survey on Machine Reading Comprehension—Tasks, Evaluation Metrics and Benchmark Datasets
Zeng, C., Li, S., Li, Q., Hu, J., & Hu, J. (2020) · Applied Sciences · doi:10.3390/app10217640 · Cited by 117
Re2G: Retrieve, Rerank, Generate
Glass, M., Rossiello, G., Chowdhury, M. F. M., Naik, A., Cai, P., & Gliozzo, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.194 · Cited by 186
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101
“Short-form generations tasks include two open-domain question answering (QA) datasets, PopQA (Mallen et al., 2023) and TriviaQA-unfiltered (Joshi et al., 2017), where systems need to answer arbitrary questions about factual knowledge.”
BackgroundPretrained Transformers for Text Ranking: BERT and BeyondPreprint
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101
Answering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1261 · Cited by 106
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“Closed-book question answering datasets including TriviaQA (Joshi et al., 2017) and Natu-ralQuestions (Kwiatkowski et al., 2019).”
MethodInfluentialMulti-Passage Machine Reading Comprehension with Cross-Passage Answer Verification
Wang, Y., Liu, K., Liu, J., He, W. P., Lyu, Y., Wu, H. R., Li, S., & Wang, H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1178 · Cited by 103
Simple Entity-Centric Questions Challenge Dense Retrievers
Sciavolino, C., Zhong, Z., Lee, J., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.496 · Cited by 98
Neural Models for Key Phrase Extraction and Question Generation
Subramanian, S., Wang, T., Yuan, X., Zhang, S., Trischler, A., & Bengio, Y. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2609 · Cited by 100
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 160
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018) · arXiv · doi:10.48550/arxiv.1809.09600 · Cited by 97
Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs
Fan, A., Gardent, C., Braud, C., & Bordes, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1428 · Cited by 101
MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 101
A Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 108
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction
Nishida, K., Nishida, K., Nagata, M., Otsuka, A., Saito, I., Asano, H., & Tomita, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1225 · Cited by 105
Multi-Hop Paragraph Retrieval for Open-Domain Question Answering
Feldman, Y., & El‐Yaniv, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1222 · Cited by 103
From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectPreprint
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01958 · Cited by 92
Knowledge Graph-Guided Retrieval Augmented Generation
Zhu, X., Xie, Y., Liu, Y., Li, Y., & Hu, W. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.449 · Cited by 91
Metadata
- Authors
- Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1601-1611
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2612431505
- Cited by
- 1,690
- References
- 21
- Citations by year
- 2026: 141 · 2025: 205 · 2024: 157 · 2023: 220 · 2022: 204 · 2021: 259 · 2020: 198 · 2019: 190 · 2018: 97 · 2017: 19
- FWCI
- 61.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- IEEE DataPort (cc-by)
- Funders
- Tencent
- Defense Advanced Research Projects Agency (FA8750-13-2-0019, FA8750)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- TriviaQA, reading comprehension, distant supervision, trivia questions, SQuAD
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
