CoChat
Sign Up
PreprintCited by 21Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

Modern Question Answering Datasets and Benchmarks: A Survey

Open access
Wang, Zhen·2022·arXiv
DOI 10.48550/arxiv.2206.15030arXiv 2206.15030
Open this paper in CoChat — free accountOpen PDF

Abstract

Question Answering (QA) is one of the most important natural language processing (NLP) tasks. It aims using NLP technologies to generate a corresponding answer to a given question based on the massive unstructured corpus. With the development of deep learning, more and more challenging QA datasets are being proposed, and lots of new methods for solving them are also emerging. In this paper, we investigate influential QA datasets that have been released in the era of deep learning. Specifically, we begin with introducing two of the most common QA tasks - textual question answer and visual question answering - separately, covering the most representative datasets, and then give some current challenges of QA research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.15030

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Of 41 NLP Q&A datasets reviewed by Wang (2022), only BioASQ aligns with scienceQ&A.

    “Of 41 NLP Q&A datasets reviewed by Wang (2022), only BioASQ aligns with scienceQ&A.”

    From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Wang, 2022

    Section 2.3 (page unavailable) reviews BioASQ as a biomedical question-answering dataset, supporting its inclusion in Wang’s survey. The same section also discusses PubMedQA and QASPER; it does not assess which datasets align with scienceQ&A. A search for “41” did not substantiate the claimed dataset count. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • FinQA: A Dataset of Numerical Reasoning over Financial Data

    Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258 · cited 5× · see citing works

    “Compare to previous multi-hop QA only focuses on textual data, FinQA (Chen et al., 2021), which extracted from financial reports, includes more kinds of information, such as table and number, and therefore some specific numerical operation is needed.”
    Influential

Flagged first, then by citations

Research works printed but not linked: 73

  1. Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019. Tallyqa: Answering complex counting ques- tions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8076-8084.
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question an- swering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
  3. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7439.
  4. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075. · arXiv
  5. Yingshan Chang, Mridu Narang, Hisami Suzuki, Gui- hong Cao, Jianfeng Gao, and Yonatan Bisk. 2021. Webqa: Multihop and multimodal qa. arXiv preprint arXiv:2109.00590. · arXiv
  6. Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. Codah: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP, pages 63-69.
  7. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036. · arXiv
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. · arXiv
  9. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. · arXiv
  10. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Co- han, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and an- swers anchored in research papers. arXiv preprint arXiv:2105.03011. · arXiv
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805. · arXiv
  12. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question an- swering by search and reading. arXiv preprint arXiv:1707.03904. · arXiv
  13. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark re- quiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161. · arXiv
  14. Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2019a. From two graphs to n ques- tions: A vqa dataset for compositional reason- ing on vision and commonsense. arXiv preprint arXiv:1908.02962. · arXiv
  15. Shen Gao, Zhaochun Ren, Yihong Zhao, Dongyan Zhao, Dawei Yin, and Rui Yan. 2019b. Product- aware answer generation in e-commerce question- answering. In Proceedings of the Twelfth ACM Inter- national Conference on Web Search and Data Min- ing, pages 429-437.
  16. Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. 2020a. Knowit vqa: Answering knowledge-based questions about videos. In Pro- ceedings of the AAAI Conference on Artificial Intel- ligence, volume 34, pages 10826-10834.
  17. Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. 2020b. A dataset and baselines for visual question answering on art. In European Conference on Computer Vision, pages 92-108. Springer.
  18. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of common- sense causal reasoning. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main con- ference and the shared task, and Volume 2: Proceed- ings of the Sixth International Workshop on Seman- tic Evaluation (SemEval 2012), pages 394-398.
  19. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image under- standing in visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913.
  20. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 3608-3617.
  21. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representa- tions. arXiv preprint arXiv:1511.02301. · arXiv
  22. Lynette Hirschman and Robert Gaizauskas. 2001. Nat- ural language question answering: the view from here. natural language engineering, 7(4):275-300.
  23. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceed- ings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 6700-6709.
  24. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146. · arXiv
  25. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
  26. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. arXiv preprint arXiv:1705.03551. · arXiv
  27. Kushal Kafle and Christopher Kanan. 2017. An analy- sis of visual question answering algorithms. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 1965-1973.
  28. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking be- yond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 252-262.
  29. Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017. Deepstory: Video story qa by deep embedded memory networks. arXiv preprint arXiv:1707.00836. · arXiv
  30. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
  31. Xiang Kong, Varun Gangal, and Eduard Hovy. 2020. Scde: sentence cloze dataset with high quality distractors from examinations. arXiv preprint arXiv:2004.12934. · arXiv
  32. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision, 123(1):32- 73.
  33. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a bench- mark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
  34. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. · arXiv
  35. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. Tvqa: Localized, compositional video ques- tion answering. arXiv preprint arXiv:1809.01696. · arXiv
  36. Hugo Liu and Push Singh. 2004. Conceptnet-a practi- cal commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
  37. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading compre- hension with logical reasoning. arXiv preprint arXiv:2007.08124. · arXiv
  38. Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. 2018. R-vqa: learning visual rela- tion facts with semantic attention for visual ques- tion answering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1880-1889.
  39. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion, pages 3195-3204.
  40. Minesh Mathew, Dimosthenis Karatzas, and CV Jawa- har. 2021. Docvqa: A dataset for vqa on docu- ment images. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 2200-2209.
  41. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? a new dataset for open book question answer- ing. arXiv preprint arXiv:1809.02789. · arXiv
  42. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPS.
  43. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for squad. arXiv preprint arXiv:1806.03822. · arXiv
  44. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. · arXiv
  45. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
  46. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 193- 203.
  47. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical rea- soning abilities of neural models. arXiv preprint arXiv:1904.01557. · arXiv
  48. Abigail See, Peter J Liu, and Christopher D Man- ning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. · arXiv
  49. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 8876-8884.
  50. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion, pages 8317-8326.
  51. Haitian Sun, William W Cohen, and Ruslan Salakhutdi- nov. 2021. Conditionalqa: A complex reading com- prehension dataset with conditional answers. arXiv preprint arXiv:2110.06884. · arXiv
  52. Simon Šuster and Walter Daelemans. 2018. Clicr: A dataset of clinical case reports for machine reading comprehension. arXiv preprint arXiv:1803.09720. · arXiv
  53. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv:1803.06643. · arXiv
  54. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937. · arXiv
  55. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631-4640.
  56. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Ka- heer Suleman. 2016. Newsqa: A machine compre- hension dataset. arXiv preprint arXiv:1611.09830. · arXiv
  57. Alexander Trott, Caiming Xiong, and Richard Socher. 2017. Interpretable counting for visual question an- swering. arXiv preprint arXiv:1712.08697. · arXiv
  58. George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the bioasq large- scale biomedical semantic indexing and question an- swering competition. BMC bioinformatics, 16(1):1- 28.
  59. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Fvqa: Fact- based visual question answering. IEEE transac- tions on pattern analysis and machine intelligence, 40(10):2413-2427.
  60. Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2015. Explicit knowledge-based reasoning for visual question an- swering. arXiv preprint arXiv:1511.02570. · arXiv
  61. Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. · arXiv
  62. Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulka- rni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Tweetqa: A social media focused question answering dataset. arXiv preprint arXiv:1907.06292. · arXiv
  63. Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Na- zli Ikizler-Cinbis. 2018. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint arXiv:1809.00812. · arXiv
  64. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 confer- ence on empirical methods in natural language pro- cessing, pages 2013-2018.
  65. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint arXiv:1809.09600. · arXiv
  66. Jinyoung Yeo, Gyeongbok Lee, Gengyu Wang, Seung- taek Choi, Hyunsouk Cho, Reinald Kim Amplayo, and Seung-won Hwang. 2018. Visual choice of plausible alternatives: An evaluation of image-based commonsense causal reasoning. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  67. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenen- baum. 2019. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442. · arXiv
  68. Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206.
  69. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326. · arXiv
  70. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8807-8817.
  71. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Vi- sual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6720-6731.
  72. Lucia Zheng, Neel Guha, Brandon R Anderson, Pe- ter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset. arXiv preprint arXiv:2104.08671. · arXiv
  73. Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei- Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004.

Cited by

Sorted by citations

Authors
Wang, Z.
Affiliations
Delft University of Technology (NL)
Corresponding author
Wang, Zhen
ORCID
Wang, Zhen
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.15030
arXiv
2206.15030
OpenAlex
W4283773758
Cited by
21
References
1
Citations by year
2026: 1 · 2025: 5 · 2024: 4 · 2023: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
question answering, question answering dataset, question answering benchmark, visual question answering, deep learning, natural language processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review