CoChat

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Open access
Rogers, Anna, Gardner, Matt, Augenstein, Isabelle · 2023 · ACM Computing Surveys
Cited by 156

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Abstract

Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with more than 80 new datasets appearing in the past 2 years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of “skills” that question answering/reading comprehension systems are supposed to acquire and propose a new taxonomy. The supplementary materials survey the current multilingual resources and monolingual resources for languages other than English, and we discuss the implications of overfocusing on English. The study is aimed at both practitioners looking for pointers to the wealth of existing data and at researchers working on new resources.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3560260

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 254

  1. 10.18653/v1/D19-6130 · DOI
  2. Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2019. ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters. In Proceedings of the 17th Annual Conference of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19). 307–317. https://aclweb.org/anthology/papers/N/N19/N19-1027/.
  3. 10.18653/v1/N19-1194 · DOI
  4. 10.1162/tacl_a_00471 · DOI · Paper page
  5. 10.18653/v1/2021.emnlp-main.164 · DOI
  6. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. 1998. Dialogue Acts in Verbmobil 2. Technical Report. Verbmobil.
  7. 10.18653/v1/2020.acl-main.421 · DOI · Paper page
  8. 10.18653/v1/2021.acl-long.118 · DOI
  9. Akari Asai, Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2018. Multilingual extractive reading comprehension by runtime machine translation. arXiv:1809.03275 [CS] (2018). http://arxiv.org/abs/1809.03275. · arXiv
  10. Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020. XOR QA: Cross-lingual open-retrieval question answering. arXiv:2010.11856 [CS] (2020). http://arxiv.org/abs/2010.11856. · arXiv
  11. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: A corpus for adding memory to goal-oriented dialogue systems. arXiv:1704.00057 [CS] (2017). http://arxiv.org/abs/1704.00057. · arXiv
  12. 10.18653/v1/2020.acl-main.656 · DOI · Paper page
  13. 10.18653/v1/D19-1475 · DOI · Paper page
  14. 10.1002/9781118784235.eelt0369 · doi:10.1002/9781118784235.eelt0369
  15. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv:1611.09268 [CS] (2016). http://arxiv.org/abs/1611.09268. · arXiv
  16. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2017. Embracing data abundance: BookTest dataset for reading comprehension. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17). https://openreview.net/pdf?id=H1U4mhVFe.
  17. Somnath Banerjee, Sudip Kumar Naskar, and Paolo Rosso. 2016. The first cross-script code-mixed question answering corpus. In Proceedings of the Workshop on Modeling, Learning, and Mining for Cross/Multilinguality (MultiLingMine’16) Co-located with the 2016 European Conference on Information Retrieval (ECIR’16). 1–10. 56–65. http://ceur-ws.org/Vol-1589/MultiLingMine6.pdf.
  18. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’13). 1533–1544. https://www.aclweb.org/anthology/D13-1160.
  19. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 1499–1510.
  20. Yevgeni Berzak Jonathan Malmaud and Roger Levy. 2020. STARC: Structured annotations for reading comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20) . 5726–5735. https://www.aclweb.org/anthology/2020.acl-main.507. · DOI
  21. Chandra Bhagavatula Ronan Le Bras Chaitanya Malaviya Keisuke Sakaguchi Ari Holtzman Hannah Rashkin Doug Downey Wen-Tau Yih and Yejin Choi. 2019. Abductive commonsense reasoning. In Proceedings of the 7th International Conference on Learning Representations (ICLR’19) . https://openreview.net/forum?id=Byg1v1HKDB.
  22. 10.18653/v1/2020.emnlp-main.703 · DOI · Paper page
  23. 10.1609/aaai.v34i05.6239 · DOI · Paper page
  24. 10.18653/v1/2020.emnlp-main.442 · DOI
  25. Simon Blackburn. 2008. Inference. Retrieved September 16 2022 from https://www.oxfordreference.com/view/10.1093/acref/9780199541430.001.0001/acref-9780199541430. · doi:10.1093/acref/9780199541430.001.0001/acref-9780199541430
  26. Simon Blackburn. 2008. Reasoning. Retrieved September 16 2022 from https://www.oxfordreference.com/view/10.1093/acref/9780199541430.001.0001/acref-9780199541430. · doi:10.1093/acref/9780199541430.001.0001/acref-9780199541430
  27. Jordan Boyd-Graber. 2019. What question answering can learn from trivia nerds. arXiv:1910.14464 [CS] (2019). http://arxiv.org/abs/1910.14464. · arXiv
  28. 10.1007/978-3-319-94042-7_9 · DOI
  29. B. Barla Cambazoglu, Mark Sanderson, Falk Scholer, and Bruce Croft. 2020. A review of public datasets in question answering research. ACM SIGIR Forum 54, 2 (2020), 23. http://www.sigir.org/wp-content/uploads/2020/12/p07.pdf.
  30. Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, and Eneko Agirre. 2020. DoQA-accessing domain-specific FAQs via conversational QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 7302–7314. https://aclanthology.org/2020.acl-main.652/. · DOI
  31. 10.18653/v1/2022.acl-long.422 · DOI · Paper page
  32. Casimiro Pio Carrino, Marta R. Costa-Jussà, and José A. R. Fonollosa. 2019. Automatic Spanish translation of the SQuAD dataset for multilingual question answering. arXiv:1912.05200 [CS] (2019). http://arxiv.org/abs/1912.05200. · arXiv
  33. Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, et al. 2020. The TechQA dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 1269–1278. https://www.aclweb.org/anthology/2020.acl-main.117. · DOI
  34. 10.18653/v1/W18-3204 · DOI
  35. Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R. Selvaraju, Dhruv Batra, and Devi Parikh. 2017. Counting everyday objects in everyday scenes. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 1135–1144. https://openaccess.thecvf.com/content_cvpr_2017/html/Chattopadhyay_Counting_Everyday_Objects_CVPR_2017_paper.html.
  36. 10.18653/v1/2021.acl-long.345 · DOI · Paper page
  37. 10.18653/v1/D19-5817 · DOI · Paper page
  38. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20). 6521–6532. https://www.aclweb.org/anthology/2020.emnlp-main.528.
  39. 10.18653/v1/2020.acl-tutorials.8 · DOI
  40. 10.18653/v1/2020.findings-emnlp.91 · DOI · Paper page
  41. 10.18653/v1/2021.emnlp-main.343 · DOI
  42. 10.1145/371578.371581 · DOI · Paper page
  43. Minseok Cho, Reinald Kim Amplayo, Seung-Won Hwang, and Jonghyuck Park. 2018. Adversarial TableQA: Attention supervision for question answering on tables. In Proceedings of Machine Learning Research. 391–406. http://proceedings.mlr.press/v95/cho18a/cho18a.pdf.
  44. 10.1162/tacl_a_00377 · DOI · Paper page
  45. Sagnik Ray Choudhury Anna Rogers and Isabelle Augenstein. 2022. Machine reading fast and slow: When do models “Understand” language? In Proceedings of the 29th International Conference on Computational Linguistics . International Committee on Computational Linguistics 78–93. https://aclanthology.org/2022.coling-1.8.
  46. 10.1145/3357384.3358016 · DOI
  47. 10.18653/v1/2021.emnlp-main.493 · DOI
  48. 10.18653/v1/P18-1078 · DOI · Paper page
  49. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 17th Annual Conference of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19). 2924–2936. https://aclweb.org/anthology/papers/N/N19/N19-1300/.
  50. 10.1162/tacl_a_00317 · DOI · Paper page
  51. Anthony Colas, Seokhwan Kim, Franck Dernoncourt, Siddhesh Gupte, Zhe Wang, and Doo Soon Kim. 2020. TutorialVQA: Question answering dataset for tutorial videos. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’20). 5450–5455. https://www.aclweb.org/anthology/2020.lrec-1.670.
  52. Tarcísio Souza Costa, Simon Gottschalk, and Elena Demidova. 2020. Event-QA: A dataset for event-centric question answering over knowledge graphs. arXiv:2004.11861 [CS] (2020). http://arxiv.org/abs/2004.11861. · arXiv
  53. 10.3233/IA-190018 · DOI
  54. 10.18653/v1/D19-1600 · DOI
  55. Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018. Dataset for the first evaluation on chinese machine reading comprehension. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18). https://www.aclweb.org/anthology/L18-1431.
  56. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. In Proceedings of the International Conference on Computational Linguistics (COLING’16). 1777–1786. https://www.aclweb.org/anthology/C16-1167.
  57. 10.18653/v1/2020.coling-main.589 · DOI
  58. 10.18653/v1/D19-1606 · DOI · Paper page
  59. Martin d’Hoffschmidt, Maxime Vidal, Wacim Belblidia, and Tom Brendlé. 2020. FQuAD: French question answering dataset. arXiv:2002.06071 [CS] (2020). http://arxiv.org/abs/2002.06071. · arXiv
  60. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. arXiv:1811.01241 [CS] (2018). http://arxiv.org/abs/1811.01241. · arXiv
  61. 10.3115/v1/P15-2114 · DOI
  62. 10.18653/v1/D19-5820 · DOI
  63. 10.1007/978-3-319-73618-1_86 · DOI
  64. Jesse Dunietz, Greg Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and Dave Ferrucci. 2020. To test machine comprehension, start by defining comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 7839–7859. https://www.aclweb.org/anthology/2020.acl-main.701. · DOI · Paper page
  65. 10.18653/v1/2021.emnlp-main.693 · DOI
  66. 10.1007/978-3-030-58219-7_1 · DOI
  67. 10.18653/v1/D19-1605 · DOI · Paper page
  68. Mihail Eric and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. arXiv:1705.05414 [CS] (2017). http://arxiv.org/abs/1705.05414. · arXiv
  69. 10.18653/v1/D18-1091 · DOI · Paper page
  70. 10.1109/TASLP.2020.3010650 · DOI
  71. 10.18653/v1/2020.coling-main.570 · DOI
  72. 10.18653/v1/2020.emnlp-main.86 · DOI · Paper page
  73. Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. 2020. A dataset and baselines for visual question answering on art. arXiv:2008.12520 [CS] (2020). http://arxiv.org/abs/2008.12520. · arXiv
  74. 10.18653/v1/2020.findings-emnlp.117 · DOI · Paper page
  75. 10.48550/arXiv.1909.11291 · DOI
  76. 10.18653/v1/2021.emnlp-main.135 · DOI · Paper page
  77. 10.1609/aaai.v34i05.6282 · DOI · Paper page
  78. 10.18653/v1/D19-1456 · DOI · Paper page
  79. Taisia Glushkova, Alexey Machnev, Alena Fenogenova, Tatiana Shavrina, Ekaterina Artemova, and Dmitry I. Ignatov. 2020. DaNetQA: A yes/no question answering dataset for the russian language. arXiv:2010.02605 [CS] (2020). http://arxiv.org/abs/2010.02605. · arXiv
  80. Ana Valeria González, Anna Rogers, and Anders Søgaard. 2021. On the interaction of belief bias and explanations. In Findings of ACL-IJCNLP’21. 2930–2942. https://aclanthology.org/2021.findings-acl.259.
  81. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 Task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics (*SEM’12). 394–398. https://aclweb.org/anthology/papers/S/S12/S12-1052/.
  82. Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. IQA: Visual question answering in interactive environments. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 4089–4098. · DOI · Paper page
  83. 10.1145/3442381.3449992 · DOI
  84. Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2020. MultiReQA: A cross-domain evaluation for retrieval question answering models. arXiv:2005.02507 [CS] (2020). http://arxiv.org/abs/2005.02507. · arXiv
  85. Shangmin Guo, Kang Liu, Shizhu He, Cao Liu, Jun Zhao, and Zhuoyu Wei. 2017. IJCNLP-2017 Task 5: Multi-choice question answering in examinations. In Proceedings of the IJCNLP’17, Shared Tasks. 34–40. https://www.aclweb.org/anthology/I17-4005.
  86. Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018. MMQA: A multi-domain multi-lingual question-answering framework for English and Hindi. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18). https://www.aclweb.org/anthology/L18-1440.
  87. 10.24963/ijcai.2019/694 · DOI
  88. 10.18653/v1/W18-3205 · DOI
  89. 10.18653/v1/2021.emnlp-main.597 · DOI
  90. Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W. Bruce Croft. 2019. ANTIQUE: A non-factoid question answering benchmark. arXiv:1905.08957 [CS] (2019). http://arxiv.org/abs/1905.08957. · arXiv
  91. Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward automated fact-checking: Detecting check-worthy factual claims by ClaimBuster. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’17). ACM, New York, NY, 1803–1812. http://dblp.uni-trier.de/db/conf/kdd/kdd2017.html#HassanALT17.
  92. 10.18653/v1/W18-2605 · DOI · Paper page
  93. Nancy Hedberg, Juan M. Sosa, and Lorna Fadden. 2004. Meanings and configurations of questions in English. In Proceedings of the International Conference on Speech Prosody. 309–312. https://www.isca-speech.org/archive/sp2004/papers/sp04_309.pdf.
  94. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24–27, 1990. https://www.aclweb.org/anthology/H90-1021.
  95. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS’15). 1693–1701. http://dl.acm.org/citation.cfm?id=2969239.2969428.
  96. 10.18653/v1/P16-1145 · DOI · Paper page
  97. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv:1511.02301 [CS] (2015). http://arxiv.org/abs/1511.02301. · arXiv
  98. Andrea Horbach, Itziar Aldabe, Marie Bexte, Oier Lopez de Lacalle, and Montse Maritxalar. 2020. Linguistic appropriateness and pedagogic usefulness of reading comprehension questions. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’20). 1753–1762. https://www.aclweb.org/anthology/2020.lrec-1.217.
  99. 10.18653/v1/D19-1243 · DOI · Paper page
  100. Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). 6700–6709. https://openaccess.thecvf.com/content_CVPR_2019/html/Hudson_GQA_A_New_Dataset_for_Real-World_Visual_Reasoning_and_Compositional_CVPR_2019_paper.html.
  101. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 2758–2766. https://openaccess.thecvf.com/content_cvpr_2017/html/Jang_TGIF-QA_Toward_Spatio-Temporal_CVPR_2017_paper.html.
  102. Dalton Jeffrey, Xiong Chenyan, and Callan Jamie. 2019. CAsT 2019: The conversational assistance track overview. In Proceedings of the Text REtrival Conference (TREC’19).
  103. 10.1145/3184558.3191536 · DOI · Paper page
  104. 10.1145/3269206.3269247 · DOI · Paper page
  105. Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. FreebaseQA: A new factoid QA data set matching trivia-style question-answer pairs with Freebase. In Proceedings of the 17th Annual Conference of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19). 318–323. https://aclweb.org/anthology/papers/N/N19/N19-1028/.
  106. 10.18653/v1/2022.acl-long.443 · DOI
  107. 10.18653/v1/2022.naacl-industry.34 · DOI
  108. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In Proceedings of the International Conference on Learning Representations(ICLR’19). https://openreview.net/forum?id=Sklgs0NFvr.
  109. Rachel Keraron, Guillaume Lancrenon, Mathilde Bras, Frédéric Allary, Gilles Moyse, Thomas Scialom, Edmundo-Pavel Soriano-Morales, and Jacopo Staiano. 2020. Project PIAF: Building a native French question-answering dataset. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’20). 5481–5490. https://www.aclweb.org/anthology/2020.lrec-1.673.
  110. 10.18653/v1/N18-1023 · DOI · Paper page
  111. Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017. DeepStory: Video story QA by deep embedded memory networks. In Proceedings of the 2017 International Joint Conference on Artificial Intelligence (IJCAI’17). https://openreview.net/forum?id=ryZczSz_bS.
  112. Seokhwan Kim Luis Ferdinando D’Haro Rafael E. Banchs Matthew Henderson Jason Willisams and Koichiro Yoshino. 2016. Dialog State Tracking Challenge 5 Handbook v.3.1. Retrieved September 16 2022 from http://workshop.colips.org/dstc5/.
  113. 10.18653/v1/2020.acl-main.502 · DOI
  114. Vladislav Korablinov and Pavel Braslavski. 2020. RuBQ: A Russian dataset for question answering over wikidata. arXiv:2005.10659 [CS] (2020). http://arxiv.org/abs/2005.10659. · arXiv
  115. 10.3115/v1/P14-1026 · DOI · Paper page
  116. 10.1109/SLT.2018.8639505 · DOI
  117. Kyungjae Lee, Kyoungho Yoon, Sunghyun Park, and Seung-Won Hwang. 2018. Semi-supervised training data generation for multilingual question answering. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18). https://www.aclweb.org/anthology/L18-1437.
  118. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the 13th International Conference on Principles of Knowledge Representation and Reasoning. 552–561.
  119. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 7315–7330. https://www.aclweb.org/anthology/2020.acl-main.653/. · DOI · Paper page
  120. Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-Yi Lee. 2018. Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension. arXiv:1804.00320 [CS] (2018). http://arxiv.org/abs/1804.00320. · arXiv
  121. 10.18653/v1/2022.naacl-main.90 · DOI
  122. Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, and Bing Qin. 2020. Molweni: A challenge multiparty dialogues-based machine reading comprehension dataset with discourse structure. arXiv:2004.05080 [CS] (2020). http://arxiv.org/abs/2004.05080. · arXiv
  123. 10.18653/v1/2021.emnlp-main.698 · DOI
  124. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv:1607.06275 [CS] (2016). http://arxiv.org/abs/1607.06275. · arXiv
  125. 10.18653/v1/P19-1129 · DOI · Paper page
  126. 10.18653/v1/2022.acl-long.290 · DOI
  127. Yichan Liang, Jianheng Li, and Jian Yin. 2019. A new multi-choice reading comprehension dataset for curriculum learning. In Proceedings of the 11th Asian Conference on Machine Learning. 742–757. http://proceedings.mlr.press/v101/liang19a.html.
  128. Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. KorQuAD1.0: Korean QA dataset for machine reading comprehension. arXiv:1909.07005 [CS] (2019). http://arxiv.org/abs/1909.07005. · arXiv
  129. 10.18653/v1/2020.emnlp-main.557 · DOI · Paper page
  130. 10.18653/v1/D19-5808 · DOI · Paper page
  131. 10.1007/978-3-319-10602-1_48 · DOI · Paper page
  132. 10.24963/ijcai.2020/501 · DOI · Paper page
  133. 10.18653/v1/P19-1227 · DOI
  134. 10.1007/978-3-030-32233-5_43 · DOI
  135. 10.18653/v1/D17-1086 · DOI
  136. Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. arXiv:2007.15207 [CS] (2020). http://arxiv.org/abs/2007.15207. · arXiv
  137. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv:1506.08909 [CS] (2015). http://arxiv.org/abs/1506.08909. · arXiv
  138. 10.18653/v1/N18-1185 · DOI
  139. Leigh-Ann MacFarlane and Genviève Boulet. 2017. Multiple-choice tests can support deep learning! Proceedings of the Atlantic Universities’ Teaching Showcase 21 (2017), 61–66. https://ojs.library.dal.ca/auts/article/view/8430.
  140. Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. arXiv:1611.07810 [CS] (2017). http://arxiv.org/abs/1611.07810. · arXiv
  141. Cheryl L. Marcham, Treasa M. Turnbeaugh, Susan Gould, and Joel T. Nadler. 2018. Developing certification exam questions: More deliberate than you may think. Professional Safety 63, 5 (May 2018), 44–49. https://onepetro.org/PS/article/63/05/44/33528/Developing-Certification-Exam-Questions-More.
  142. Michał Marcinczuk, Marcin Ptak, Adam Radziszewski, and Maciej Piasecki. 2013. Open dataset for development of Polish question answering systems. In Proceedings of the 6th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics. https://www.researchgate.net/profile/Maciej-Piasecki/publication/272685856_Open_dataset_for_development_of_Polish_Question_Answering_systems.
  143. 10.18653/v1/2022.findings-acl.177 · DOI · Paper page
  144. 10.1145/2872427.2883044 · DOI · Paper page
  145. John McCarthy and Patrick Hayes. 1969. Some philosophical problems from the standpoint of artificial intelligence. In Machine Intelligence 4, B. Meltzer and Donald Michie (Eds.). Edinburgh University Press, 463–502.
  146. Tom McCoy, Junghyun Min, and Tal Linzen. 2019. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv:1911.02969 [CS] (2019). http://arxiv.org/abs/1911.02969. · arXiv
  147. 10.18653/v1/P19-1334 · DOI · Paper page
  148. 10.1016/S0079-7421(09)51009-2 · DOI · Paper page
  149. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’18). 2381–2391. http://aclweb.org/anthology/D18-1260. · DOI · Paper page
  150. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. arXiv:2004.10645 [CS] (2020). http://arxiv.org/abs/2004.10645. · arXiv
  151. 10.18653/v1/2021.naacl-main.364 · DOI
  152. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. 2020. Towards question format independent numerical reasoning: A set of prerequisite tasks. arXiv preprint arXiv:2005.08516 (2020). https://arxiv.org/abs/2005.08516. · arXiv
  153. Ashutosh Modi Tatjana Anikina Simon Ostermann and Manfred Pinkal. 2016. InScript: Narrative texts annotated with script information. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’16) . 3485–3493. https://www.aclweb.org/anthology/L16-1555.
  154. Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020. COVID-QA: A question answering dataset for COVID-19. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL’20. https://www.aclweb.org/anthology/2020.nlpcovid19-acl.18.
  155. Nasrin Mostafazadeh, Michael Roth, Nathanael Chambers, and Annie Louis. 2017. LSDSem 2017 shared task: The story Cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential, and Discourse-Level Semantics. 46–51. http://www.aclweb.org/anthology/W17-0900. · DOI · Paper page
  156. 10.18653/v1/W19-4612 · DOI
  157. Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. 2017. MarioQA: Answering questions by watching gameplay videos. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV’17). http://arxiv.org/abs/1612.01669. · arXiv
  158. Preslav Nakov, Doris Hoogeveen, Lluís Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 Task 3: Community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval’17). 27–48. http://www.aclweb.org/anthology/S17-2003. · DOI · Paper page
  159. Preslav Nakov Lluís Màrquez Walid Magdy Alessandro Moschitti Jim Glass and Bilal Randeree. 2015. SemEval-2015 Task 3: Answer selection in community question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval’15) . 269–281. · DOI
  160. Preslav Nakov Lluís Màrquez Alessandro Moschitti Walid Magdy Hamdy Mubarak abed Alhakim Freihat Jim Glass and Bilal Randeree. 2016. SemEval-2016 Task 3: Community question answering. 525–545. · DOI · Paper page
  161. 10.18653/v1/2020.coling-main.233 · DOI
  162. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. arXiv:2005.00242 [CS] (2020). http://arxiv.org/abs/2005.00242. · arXiv
  163. Kazumasa Omura, Daisuke Kawahara, and Sadao Kurohashi. 2020. A method for building a commonsense inference dataset based on basic events. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20). 2450–2460. https://www.aclweb.org/anthology/2020.emnlp-main.192.
  164. 10.18653/v1/D16-1241 · DOI · Paper page
  165. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. MCScript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18). https://www.aclweb.org/anthology/L18-1564.
  166. 10.18653/v1/S18-1119 · DOI
  167. 10.3115/v1/P15-1142 · DOI · Paper page
  168. Alkesh Patel, Akanksha Bindal, Hadas Kotek, Christopher Klein, and Jason Williams. 2020. Generating natural questions from images for multimodal assistants. arXiv:2012.03678 [CS] (2020). http://arxiv.org/abs/2012.03678. · arXiv
  169. 10.1007/978-3-319-11382-1_23 · doi:10.1007/978-3-319-11382-1_23
  170. Anselmo Peñas, Christina Unger, Georgios Paliouras, and Ioannis Kakadiaris. 2015. Overview of the CLEF question answering track 2015. In Experimental IR Meets Multilinguality, Multimodality, and Interaction.Lecture Notes in Computer Science, Vol. 9283. Springer, 539–544. · DOI
  171. Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing MANtIS: A novel multi-domain information seeking dialogues dataset. arXiv:1912.04639 [CS] (2019). http://arxiv.org/abs/1912.04639. · arXiv
  172. 10.18653/v1/D19-1460 · DOI · Paper page
  173. 10.18653/v1/2022.findings-acl.196 · DOI
  174. Eric Price. 2014. The NIPS Experiment. Retrieved September 16 2022 from http://blog.mrtz.org/2014/12/15/the-nips-experiment.html.
  175. Boyu Qiu, Xu Chen, Jungang Xu, and Yingfei Sun. 2019. A survey on neural machine reading comprehension. arXiv:1906.03824 [CS] (2019). http://arxiv.org/abs/1906.03824. · arXiv
  176. 10.1145/3209978.3210124 · DOI
  177. Filip Radlinski Krisztian Balog Bill Byrne and Karthik Krishnamoorthi. 2019. Coached conversational preference elicitation: A case study in understanding movie preferences. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue . https://research.google/pubs/pub48414/. · DOI
  178. 10.1145/2740908.2743006 · doi:10.1145/2740908.2743006
  179. Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. 2020. Explaining and improving model behavior with k nearest neighbor representations. arXiv:2010.09030 [CS] (2020). http://arxiv.org/abs/2010.09030. · arXiv
  180. 10.18653/v1/2020.coling-main.603 · DOI · Paper page
  181. 10.18653/v1/P18-1043 · DOI · Paper page
  182. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’13). 193–203.
  183. 10.18653/v1/2021.emnlp-main.758 · DOI
  184. 10.18653/v1/2020.emnlp-main.655 · DOI
  185. 10.48550/arXiv.1904.04792 · DOI
  186. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Proceedings of the AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. 6.
  187. Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Findings of EMNLP’20. 1256–1262. https://www.aclweb.org/anthology/2020.findings-emnlp.112/.
  188. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to AI complete question answering: A set of prerequisite real tasks. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20). 8722–8731. https://aaai.org/ojs/index.php/AAAI/article/view/6398.
  189. Uma Roy, Noah Constant, Rami Al-Rfou, Aditya Barua, Aaron Phillips, and Yinfei Yang. 2020. LAReQA: Language-agnostic answer retrieval from a multilingual pool. arXiv:2004.05484 [CS] (2020). http://arxiv.org/abs/2004.05484. · arXiv
  190. Sebastian Ruder and Si Avirup. 2021. Multi-domain multilingual question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’21).
  191. 10.18653/v1/2020.findings-emnlp.418 · DOI
  192. Barbara Rychalska, Dominika Basaj, Anna Wróblewska, and Przemyslaw Biecek. 2018. Does it care what you asked? Understanding importance of verbs in deep learning QA system. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP. 322–324. http://aclweb.org/anthology/W18-5436.
  193. 10.3115/v1/P15-1024 · DOI · Paper page
  194. 10.18653/v1/D19-1454 · DOI · Paper page
  195. Viktor Schlegel, Goran Nenadic, and Riza Batista-Navarro. 2020. Beyond leaderboards: A survey of methods for revealing weaknesses in natural language inference data and models. arXiv:2005.14709 [CS] (2020). http://arxiv.org/abs/2005.14709. · arXiv
  196. Viktor Schlegel, Marco Valentino, André Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020. A framework for evaluation of machine reading comprehension gold standards. In Proceedings of the Language Resources and Evaluation Conference. http://arxiv.org/abs/2003.04642. · arXiv
  197. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2015. A survey of available corpora for building data-driven dialogue systems. arXiv:1512.05742 [CS, STAT] (2015). http://arxiv.org/abs/1512.05742. · arXiv
  198. Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2019. DRCD: A Chinese machine reading comprehension dataset. arXiv:1806.00920 [CS] (2019). http://arxiv.org/abs/1806.00920. · arXiv
  199. 10.18653/v1/D15-1135 · DOI · Paper page
  200. Hideyuki Shibuki, Kotaro Sakamoto, Yoshionobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y. Itakura, Di Wang, Tatsunori Mori, and Noriko Kando. 2014. Overview of the NTCIR-11 QA-lab task. In Proceedings of the 11th NTCIR Conference. 518–529. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings11/pdf/NTCIR/OVERVIEW/01-NTCIR11-OV-QALAB-ShibukiH.pdf.

200 of 254 shown

Cited by

See all 156

Metadata

Authors
Rogers, A., Gardner, M., & Augenstein, I.
Venue
ACM Computing Surveys
Published
2023-02-02
Volume
55
Issue
10
Pages
1-45
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W3184751871
Cited by
156
References
74
FWCI
11.31
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. https://app.cochat.ai/papers/10.1145/3560260
The paper
Rogers, A., Gardner, M., & Augenstein, I. (2023). QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. ACM Computing Surveys. https://doi.org/10.1145/3560260

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.