The DOI matches its Crossref record; 22 of the 24 references it cites were found in a registry or an index.
NewsQA: A Machine Comprehension Dataset
Open accessAbstract
We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Existing datasets vary in size, difficulty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al.”
BackgroundGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“Word embeddings are initialized using GloVe vectors (Pennington et al., 2014) pre-trained on the 840-billion Common Crawl corpus.”
BackgroundMethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“BLEU is a precision-based metric popular in machine translation that uses a weighted average of variable length phrase matches ( n -grams) against the reference sentence (Papineni et al., 2002).”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 3×
“Existing datasets vary in size, difficulty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al., 2013) are too small for training data-intensive deep…”
BackgroundMethodInfluentialCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2×
“ase matches (n-grams) against the reference sentence (Papineni et al., 2002). CIDEr was designed to correlate better with human judgements of sentence similarity, and uses tf-idf scores over n-grams (Vedantam et al., 2015). As given in Table 4, humans averaged 0.694 F1 on NewsQA. The human EM scores are relatively low…”
MethodInfluentialOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,377
Exact solutions to the nonlinear dynamics of learning in deep linear neural networksPreprint
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 1,010
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 5×
“Like Richardson et al. (2013) we want to encourage reasoning in comprehension models.”
BackgroundInfluential
Research works printed but not linked: 3
- NewsQA: A Machine Comprehension Dataset https://aclanthology.org/W17-2623/ (Trischler et al., RepL4NLP 2017)
- Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A Machine Comprehension Dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
- ACL materials are Copyright © 1963–2026 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.
Cited by
Sorted by citations
Items 101–200 of 708
ScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 86
“There has been ongoing research on MRC on various textual documents like Wikipedia articles [37,59], news articles [7,47], etc.”
BackgroundInfluentialGeneralizing Question Answering System with Pre-trained Language Model Fine-tuning
Su, D., Xu, Y., Winata, G. I., Xu, P., Kim, H., Liu, Z., & Fung, P. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5827 · Cited by 76
“Dataset passages cover different topics, such as movies (Saha et al., 2018), news (Trischler et al., 2017), and biomedicine (Tsatsaronis et al., 2012).”
BackgroundMethodInfluentialGraph Neural Networks for Natural Language Processing: A SurveyPreprint
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06090 · Cited by 70
Hierarchical Graph Network for Multi-hop Question Answering
Fang, Y., Sun, S., Gan, Z., Pillai, R., Wang, S., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.710 · Cited by 176
A Vietnamese Dataset for Evaluating Machine Reading Comprehension
Nguyen, K. V., Nguyen, D.-V., Nguyen, T. H. A., & Nguyen, N. L.-T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.233 · Cited by 77
Universal Information Extraction as Unified Semantic Matching
Lou, J., Lu, Y., Dai, D., Jia, W., Lin, H., Han, X., Sun, L., & Wu, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26563 · Cited by 87
“2019), NewsQA (Trischler et al. 2017), SQuAD (Rajpurkar et al.”
TWEETQA: A Social Media Focused Question Answering Dataset
Xiong, W., Wu, J., Wang, H., Kulkarni, V., Yu, M., Chang, S., Guo, X., & Wang, W. Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1496 · Cited by 79
“Such efforts give the rise of large-scale human-annotated RC datasets, many of which are quite popular in the community such as SQuAD (Rajpurkar et al., 2016), MS MARCO (Nguyen et al., 2016), NewsQA (Trischler et al., 2016).”
BackgroundFQuAD: French Question Answering Dataset
d’Hoffschmidt, M., Belblidia, W., Heinrich, Q., Brendlé, T., & Vidal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.107 · Cited by 102
Influence Functions in Deep Learning Are FragilePreprint
Basu, S., Pope, P., & Feizi, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14651 · Cited by 163
“In NLP, deep learning models are often used for machine comprehension [31], summarization [32] and question-answering [33] tasks, amongst others.”
UNQOVERing Stereotyping Biases via Underspecified Questions
Li, T., Khashabi, D., Khot, T., Sabharwal, A., & Srikumar, V. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.311 · Cited by 124
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
Sood, E., Tannert, S., Frassinelli, D., Bulling, A., & Vu, N. T. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.2 · Cited by 66
To Test Machine Comprehension, Start by Defining Comprehension
Dunietz, J., Burnham, G., Bharadwaj, A., Rambow, O., Chu‐Carroll, J., & Ferrucci, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.701 · Cited by 68
Document Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 95
“However, [57, 89] pose only extractive questions where the answer follows the context on which the question is defined as in other question answering benchmarks [78, 92, 42].”
BackgroundFriendsQA: Open-Domain Question Answering on TV Show Transcripts
Yang, Z., & Choi, J. D. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5923 · Cited by 60
“, 2016) use Wikipedia, whereas NewsQA (Trischler et al., 2017) use CNN articles as evidence passages.”
MethodDocument Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension
Zheng, B., Wen, H., Liang, Y., Duan, N., Che, W., Jiang, D., Zhou, M., & Liu, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.599 · Cited by 55
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
Yatskar, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1241 · Cited by 60
“With the development of large scale benchmarks and sufficiently simple evaluations (Trischler et al., 2016; Nguyen et al., 2016; Hermann et al., 2015) progress has been rapid.”
BackgroundCapturing Greater Context for Question Generation
Tuan, L. A., Shah, D., & Barzilay, R. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i05.6440 · Cited by 59
On the Importance of Diversity in Question Generation for QA
Sultan, M. A., Chandel, S., Astudillo, R. F., & Castelli, V. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.500 · Cited by 62
“To find out if the same holds in a more challenging out-of-domain setup, we perform a zero-shot application (i.e., with no further fine-tuning) of four of the above SQ u AD -trained QG models to N ews QA , a reading comprehension dataset of CNN news articles (Trischler et al., 2017).”
MethodCoarse-grain Fine-grain Coattention Network for Multi-evidence Question AnsweringPreprint
Zhong, V., Xiong, C., Keskar, N. S., & Socher, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00603 · Cited by 65
“QA tasks span a variety of sources such as Wikipedia (Yang et al., 2015; Rajpurkar et al., 2016; 2018; Hewlett et al., 2016; Joshi et al., 2017; Welbl et al., 2018), news articles (Hermann et al., 2015; Trischler et al., 2017), books (Richardson et al., 2013), and trivia (Iyyer et al., 2014).”
BackgroundAdversarial Domain Adaptation for Machine Reading Comprehension
Wang, H., Gan, Z., Liu, X., Liu, J., Gao, J., & Wang, H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1254 · Cited by 65
“The proposed approach is validated on a set of popular benchmarks, including SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al., 2016), and MS MARCO (Bajaj et al., 2016), using state-of-the-art MRC models including SAN (Liu et al., 2018) and BiDAF (Seo et al., 2017).”
BackgroundMethodTwo-Stage Synthesis Networks for Transfer Learning in Machine Comprehension
Golub, D., Huang, P.-S., He, X., & Deng, L. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1087 · Cited by 54
“…on the NewsQA dataset (Trischler et al., 2016), which consists of CNN/Daily Mail articles, we improve the performance of the SQuAD baseline from 39.0% to 46.6% F1 and approach results of previously published work of Trischler et al. (2016) (50.0% F1), without use of labeled data in the new domain.”
BackgroundMethodResultInfluentialLook at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 80
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
Zhao, T., Lu, X., & Lee, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.47 · Cited by 81
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 62
“NewsQA (Trischler et al., 2017) is made up of 100K questions on 10K CNN articles.”
MethodInfluentialSource-Free Unsupervised Domain Adaptation: Current research and future directions
Zhang, N., Lü, J., Li, K., Fang, Z., & Zhang, G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126921 · Cited by 52
Pilault, J., Elhattami, A., & Pal, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09139 · Cited by 75
Movie Plot Analysis via Turning Point Identification
Papalampidi, P., Keller, F., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1180 · Cited by 54
“…Answering (QA) tasks are transitioning from investigating single short and clean articles or queries (Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2016) to large scale datasets that consist of complex stories (Tapaswi et al., 2016; Frermann et al., 2018; Koˇcisk`y et al.,…”
DoQA - Accessing Domain-Specific FAQs via Conversational QA
Campos, J. A., Otegi, A., Soroa, A., Deriu, J., Cieliebak, M., & Agirre, E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.652 · Cited by 59
Deep learning based question answering system in Bengali
Mayeesha, T. T., Sarwar, A. M., & Rahman, R. M. (2020) · Journal of Information and Telecommunication · doi:10.1080/24751839.2020.1833136 · Cited by 54
“For English language domain speci fi c QA datasets like NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017) are also available.”
MethodExploiting Explicit Paths for Multi-hop Reading Comprehension
Kundu, S., Khot, T., Sabharwal, A., & Clark, P. E. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1263 · Cited by 50
“Many reading comprehension (RC) datasets (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) have been proposed recently to evaluate a system’s ability to answer a question from a given text passage.”
Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
Xu, Y., Liu, X., Shen, Y., Liu, J., & Gao, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1271 · Cited by 60
“For example, on NewsQA (Trischler et al., 2017), our model surpassed human performance by 13.4 (46.5 vs 59.9) and 3.2 (72.6 vs 69.4) absolute points in terms of exact match and F1.”
BackgroundMethodInfluentialEQG-RACE: Examination-Type Question Generation
Jia, X., Zhou, W., Sun, X., & Wu, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17553 · Cited by 48
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Ye, D., Lin, Y., Huang, Y., & Sun, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.463 · Cited by 68
DUMA: Reading Comprehension With Transposition Thinking
Zhu, P., Zhang, Z., Zhao, H., & Li, X. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3138683 · Cited by 60
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 57
“The first is extractive QA (Ra-jpurkar et al., 2016, 2018; Trischler et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019), where the answer to each answerable question appears as a span of tokens in the passage.”
BackgroundInfluentialODSQA: Open-Domain Spoken Question Answering Dataset
Lee, C., Wang, S.-M., Chang, H.-C., & Lee, H.-Y. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639505 · Cited by 48
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world ApplicationsPreprint
He, W. P., Liu, K., Liu, J., Lyu, Y., Zhao, S., Xiao, X., Liu, Y., Wang, Y., Wu, H. R., She, Q., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05073 · Cited by 85
“A series of neural network models, such as Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2016), R-net (Wang et al., 2017), have been proposed, achieving promising results on a variety of MRC evaluation tasks.”
BackgroundInfluentialMachine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 63
“NewsQA (Trischler et al. 2017) Ex 120K news article C blindly created questions SearchQA (Dunn et al.”
BackgroundInfluentialAmazonQA: A Review-Based Question Answering Task
Gupta, M., Kulkarni, N., Chanda, R., Rayasam, A., & Lipton, Z. C. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/694 · Cited by 52
“NewsQA [Trischler et al., 2017] Crowd-Sourced CNN Span of words 100K 10K SearchQA [Dunn et al.”
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 61
Machine Reading Comprehension Using Structural Knowledge Graph-aware Network
Qiu, D., Zhang, Y., Feng, X., Liao, X., Jiang, W., Lyu, Y., Liu, K., & Zhao, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1602 · Cited by 44
“The ability of utilizing external knowledge is of great signifi-cance in an MRC system (Rajpurkar et al., 2016; Trischler et al., 2017).”
BackgroundBeat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00338 · Cited by 130
Quiz-Style Question Generation for News Stories
Lelkes, A. D., Tran, V. Q., & Yu, C. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449892 · Cited by 49
The Turking Test: Can Language Models Understand Instructions?Preprint
Efrat, A., & Levy, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11982 · Cited by 109
“The instructions I are modeled after the original instructions of NewsQA (Trischler et al., 2016), but replace visual HTML cues (e.g “Ask a question here.” from Figure 1) with purely textual prompts (blue), as without them the model’s accuracy was 0% on every turking task.”
MethodLi, J., Liu, M., Zheng, Z., Zhang, H., Qin, B., Kan, M., & Liu, T. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533364 · Cited by 42
Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
Gong, H., Shen, Y., Yu, D., Chen, J., & Yu, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.603 · Cited by 42
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Sinha, K., Sodhani, S., Dong, J., Pineau, J., & Hamilton, W. L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1458 · Cited by 110
“This includes the SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), and MCTest (Richardson et al., 2013) benchmarks that focus on factual questions; the SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) benchmarks for sentence understanding; and the bABI tasks (We-ston…”
BackgroundMethodInfluentialHsu, T.-Y., Liu, C.-L., & Lee, H.-Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1607 · Cited by 48
“…al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018) have nourished the development of numerous powerful and diverse RC models (Seo et al., 2016; Hu et al., 2018; Wang et al., 2017; Clark and Gardner, 2018; Huang et al., 2017).”
BackgroundLeaf: Multiple-Choice Question Generation
Vachev, K., Hardalov, M., Karadzhov, G., Georgiev, G., Koychev, I., & Nakov, P. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-99739-7_41 · Cited by 45
Answering Open-Domain Questions of Varying Reasoning Steps from Text
Qi, P., Lee, H., Sido, O. T., & Manning, C. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.292 · Cited by 65
“Aside from Wikipedia, researchers have also used news articles (Trischler et al., 2016) and search results from the web (Dunn et al.”
BackgroundMethodInfluentialEnglish Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 47
“5(F1) 3 what 3 NewsQA (Trischler et al., 2017) 120k CNN CRW ] 69.”
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00955 · Cited by 39
“…(Hendrycks et al., 2020)) Extractive SQuAD 1.1 (wikipedia (Rajpurkar et al., 2016)), SQuAD 2 (Wikipedia (Rajpurkar et al., 2018)), NewsQA (news (Trischler et al., 2017)), Quoref (wikipedia (Dasigi et al., 2019)), ROPES (situation understanding (Lin et al., 2019)) Table 2: Datasets used in this…”
BackgroundInfluentialEnd-to-End Video Question-Answer Generation With Generator-Pretester Network
Su, H.-T., Chang, C.-H., Shen, P.-W., Wang, Y.-S., Chang, Y.-L., Chang, Y.-C., Cheng, P.-J., & Hsu, W. H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3051277 · Cited by 38
Generative Language Models for Paragraph-Level Question Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.42 · Cited by 41
A dataset and baselines for sequential open-domain question answering
Elgohary, A., Zhao, C., & Boyd-Graber, J. L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1134 · Cited by 42
“Several question answering datasets have been proposed (Berant et al., 2013; Joshi et al., 2017; Trischler et al., 2017; Rajpurkar et al., 2018, inter alia).”
BackgroundMCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants
Ostermann, S., Roth, M., & Pinkal, M. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1012 · Cited by 43
“First, NewsQA (Trischler et al., 2017) is a dataset of newswire texts from CNN with questions and answers written by crowdsourcing workers.”
A review of public datasets in question answering research
Cambazoglu, B. B., Sanderson, M., Scholer, F., & Croft, B. W. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483389 · Cited by 36
“The NewsQA dataset [Trischler et al., 2017] was released by Microsoft in 2017 with the goal of creating a more natural and challenging dataset than the previous ones.”
MethodKALA: Knowledge-Augmented Language Model Adaptation
Kang, M., Baek, J., & Hwang, S. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.379 · Cited by 44
Proceedings of the Fourth Arabic Natural Language Processing Workshop
El-Hajj, W., Belguith, L. H., Bougares, F., Magdy, W., Zitouni, I., Tomeh, N., & El Hajj, M. (2019) · doi:10.18653/v1/w19-46 · Cited by 37
Yang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
SubjQA: A Dataset for Subjectivity and Review Comprehension
Bjerva, J., Bhutani, N., Golshan, B., Tan, W.-C., & Augenstein, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.442 · Cited by 45
Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space
Liu, D., Gong, Y., Fu, J., Yan, Y., Chen, J., Lv, J., Duan, N., & Zhou, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.467 · Cited by 37
Unsupervised Domain Adaptation on Reading Comprehension
Cao, Y. H., Fang, M., Yu, B., & Zhou, J. T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6245 · Cited by 38
New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles
Nguyen, K. V., Huynh, T. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3527631 · Cited by 45
“, 2016) and the NewsQA corpus (Trischler et al., 2016), but it is not clear these state-of-the-art models will obtain similar performance with corpora in different languages.”
BackgroundMethodInfluentialContext-Aware Answer Extraction in Question Answering
Seonwoo, Y., Kim, J., Ha, J.-W., & Oh, A. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.189 · Cited by 40
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering
Lipping, S., Sudarsanam, P., Drossos, K., & Virtanen, T. I. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909680 · Cited by 34
Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
Sugawara, S., Kido, Y., Yokono, H., & Aizawa, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1075 · Cited by 35
“As summarized in Table 2, the annotation was performed on six existing RC datasets: QA4MRE (Sutcliffe et al., 2013), MCTest (Richardson et al., 2013), SQuAD (Rajpurkar et al., 2016), Who-didWhat (Onishi et al., 2016), MS MARCO (Nguyen et al., 2016), and NewsQA (Trischler et al., 2016).”
MethodInfluentialComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1027 · Cited by 34
“Reading comprehension (RC) is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kociský et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
BackgroundTextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
Yang, Z., Cui, Y., Chen, Z., Che, W., Liu, T., Wang, S., & Hu, G. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.2 · Cited by 38
Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Lyu, C., Shang, L., Graham, Y., Foster, J., Jiang, X., & Liu, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.340 · Cited by 37
Structural Embedding of Syntactic Trees for Machine Comprehension
Liu, R., Hu, J., Wei, W., Yang, Z., & Nyberg, E. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1085 · Cited by 39
“Machine comprehension on question answering is a problem that requires machines to generate answers by locating a phase inside a context paragraph [14, 17], which is especially challenging since the algorithm has to come up with an answer that matches exactly with what human would do.”
BackgroundRead before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.61 · Cited by 37
Conditional Generative Adversarial Networks for Commonsense Machine Comprehension
Wang, B., Liu, K., & Zhao, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/576 · Cited by 32
“Some MC datasets and evaluations have been proposed focusing on this type of task, for example, CNN/Daily Mail [Hermann et al. , 2015] and Children Book Test [Hill et al. , 2015] where one must fill a noun in a cloze style sentence based on the context; Or SQuAD [Rajpurkar et al. , 2016] and NewsQA [Trischler et al. ,…”
BackgroundInfluentialA Question-Focused Multi-Factor Attention Network for Question Answering
Kundu, S., & Ng, H. T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12065 · Cited by 33
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Yue, X., Jimenez Gutierrez, B., & Sun, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.410 · Cited by 33
Inquisitive Question Generation for High Level Text Comprehension
Ko, W.-J., Chen, T.-Y., Huang, Y., Durrett, G., & Li, J. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.530 · Cited by 39
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.158 · Cited by 54
KI-MAG: A knowledge-infused abstractive question answering system in medical domain
Zafar, A., Sahoo, S. K., Bhardawaj, H., Das, A., & Ekbal, A. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127141 · Cited by 35
Temporal Blind Spots in Large Language Models
Wallat, J., Jatowt, A., & Anand, A. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635818 · Cited by 37
“NewsQA [57] is a machine reading comprehension dataset containing 119 𝐾 text span answers created based on a collection of CNN news articles published over nine years (2007 - 2015).”
BackgroundInteractive Language Learning by Question Answering
Yuan, X., Côté, M.-A., Fu, J., Lin, Z., Pal, C., Bengio, Y., & Trischler, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1280 · Cited by 36
“datasets (Rajpurkar et al., 2016; Trischler et al., 2016; Nguyen et al., 2016; Reddy et al., 2018; Yang et al., 2018) and a host of neural models that address them (Wang and Jiang, 2016; Wang et al.”
BackgroundSelf-citationNguyen, K. V., Tran, K. V., Luu, S. T., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2020) · IEEE Access · doi:10.1109/access.2020.3035701 · Cited by 32
“Recently, there has been a significant increase in the construction of extractive MRC datasets with formal written texts such as SQuAD [2], CNN/Daily Mail [1], CBT [28], NewsQA [29], TriviaQA [31], WIKI-HOP [32], DRCD [37], and CMRC2018 [38].”
BackgroundMixQG: Neural Question Generation with Mixed Answer Types
Murakhovs’ka, L., Wu, C.-S., Laban, P., Niu, T., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.111 · Cited by 31
Review on Neural Question Generation for Education Purposes
Faraby, S. A., Adiwijaya, A., & Romadhony, A. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00374-x · Cited by 44
“The availability of datasets is a crucial prerequisite for producing neural-based QG. Datasets that were widely used during the early days of NQG research were specifi-cally aimed at improving QA systems, such as SQuAD, MS Marco (Bajaj et al., 2016), and NewsQA (Trischler et al., 2016).”
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Li, Z., Wang, W., Dong, L., Wei, F., & Xu, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.600 · Cited by 38
QuASE: Question-Answer Driven Sentence Encoding
He, H., Qiang, N., & Roth, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.772 · Cited by 31
“We extensively compare 6 different choices of A : TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), SQuAD (Ra-jpurkar et al., 2016), relation extraction (RE) dataset in QA format (QA-RE for short) (Levy et al., 2017), Large QA-SRL (FitzGerald et al., 2018), and QAMR (Michael et al., 2017).”
MethodInfluentialMultilingual Transfer Learning for QA using Translation as Data Augmentation
Bornea, M. A., Pan, L., Rosenthal, S., Florian, R., & Sil, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17491 · Cited by 33
Nishida, K., Saito, I., Otsuka, A., Asano, H., & Tomita, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271702 · Cited by 45
“ntained in one relevant passage. Currently, SQuADfull is the only large-scale MRS dataset that can be used to bothtrain and evaluate extractive RC models in the same domainsand question styles.NewsQA [45] canbeused as the training data for extractive RC models, while its questions are too dependent on the correspondin…”
BackgroundDual Co-Matching Network for Multi-choice Reading ComprehensionPreprint
Zhang, S., Zhao, H., Wu, Y., Zhang, Z., Zhou, X., & Zhou, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09381 · Cited by 31
“, 2016) and NewsQA (Trischler et al., 2017) where the expected answer is usually in the form of a short span from the given passage, answer in multi-choice MRC is non-extractive and may not appear in the original passage, which allows rich types of questions such as commonsense reasoning and passage summarization, as…”
BackgroundMethodWhy Machine Reading Comprehension Models Learn Shortcuts?
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.85 · Cited by 32
Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text
Laban, P., Schnabel, T., Bennett, P. N., & Hearst, M. A. (2021) · doi:10.18653/v1/2021.acl-long.498 · Cited by 32
“Questions were generated via a GPT2 question generation model fine-tuned on the NewsQA dataset (Trischler et al., 2017).”
MethodAn Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering
Longpre, S., Lu, Y., Tu, Z., & DuBois, C. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5829 · Cited by 40
“3 NewsQA (Trischler et al., 2017) 74K 214K 49.”
Systematic Error Analysis of the Stanford Question Answering Dataset
Rondeau, M.-A., & Hazen, T. J. (2018) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/w18-2602 · Cited by 30
“As such, we recommend the use of datasets where question creation is independent of the passage, such as: • MSMarco (Nguyen et al., 2016) • NarrativeQA (Kocisk´y et al., 2017) • NewsQA (Trischler et al., 2017) • SearchQA (Dunn et al., 2017) • TriviaQA (Joshi et al., 2017)”
MethodBioMRC: A Dataset for Biomedical Machine Reading Comprehension
Pappas, D., Stavropoulos, P., Androutsopoulos, I., & McDonald, R. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.15 · Cited by 36
Natural Test Generation for Precise Testing of Question Answering Software
Shen, Q., Chen, J., Zhang, J. M., Wang, H., Liu, S., & Tian, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556953 · Cited by 36
EduQG: A Multi-Format Multiple-Choice Dataset for the Educational Domain
Hadifar, A., Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2023) · IEEE Access · doi:10.1109/access.2023.3248790 · Cited by 35
“researches exist in other domains such as NewsQA [30] which is a QA dataset on news articles, QUASAR-S [31] which is a collection of cloze questions constructed from”
BackgroundR-Trans: RNN Transformer Network for Chinese Machine Reading Comprehension
Liu, S., Zhang, S., Zhang, X., & Wang, H. (2019) · IEEE Access · doi:10.1109/access.2019.2901547 · Cited by 30
“They can be roughly divided into four categories: multiple choice (McTest [9], ARC [10], RACE [11]), cloze-style (CNN/Daily News [12], CBT [13], CLOTH [14]), extractive (SQuAD [15], NewsQA [16], QuAC [17]) and abstractive (MS MARCO [18], NarrativeQA [19]).”
BackgroundLi, K., Xian, X., Wang, J., & Yu, N. (2018) · Applied Surface Science · doi:10.1016/j.apsusc.2018.11.214 · Cited by 30
“l and high school Chinese students. This corpus allows types of passages to be more various. In contrast to one fixed style for the whole dataset, such as news for CNN & Daily Mail [24] and NewsQA [80], fictional stories for CBT [25] and MCTest [66], almost all kinds of passages can be found in RACE. As a multiple c…”
BackgroundResultProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 33
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 32
“Example datasets include N EWS QA (Trischler et al., 2016) 4 , Q U AC (Choi et al., 2018) and N ATURAL Q UESTIONS (Kwiatkowski et al., 2019).”
MethodSynthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering
Riabi, A., Scialom, T., Keraron, R., Sagot, B., Seddah, D., & Staiano, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.562 · Cited by 44
“Signifi-cant progress has recently been enabled by the use of large pre-trained language models (Devlin et al., 2019; Raffel et al., 2020), which reach human-level performances on several publicly available benchmarks, such as SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017).”
Method
Metadata
- Authors
- Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K.
- Venue
- Proceedings of the 2nd Workshop on Representation Learning for NLP
- Published
- 2017
- Pages
- 191-200
- OpenAlex
- W2557764419
- Cited by
- 706
- References
- 24
- Citations by year
- 2026: 18 · 2025: 26 · 2024: 39 · 2023: 96 · 2022: 110 · 2021: 134 · 2020: 138 · 2019: 87 · 2018: 41 · 2017: 16
- FWCI
- 35.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Representation Learning for NLP (RepL4NLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- machine reading comprehension, question answering, news articles, exploratory questions, reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
