Cited by 708Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 22 of the 24 references it cites were found in a registry or an index.

NewsQA: A Machine Comprehension Dataset

Open access

Trischler, Adam, Wang, Tong, Yuan, Xingdi, Harris, Justin, Sordoni, Alessandro, Bachman, Philip, Suleman, Kaheer·2017·Proceedings of the 2nd Workshop on Representation Learning for NLP

DOI 10.18653/v1/w17-2623

Open this paper in CoChat — free account

Abstract

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.

Integrity signals

Registry

Crossref · 10.18653/v1/w17-2623

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Existing datasets vary in size, difficulty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al.”
Background
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“Word embeddings are initialized using GloVe vectors (Pennington et al., 2014) pre-trained on the 840-billion Common Crawl corpus.”
BackgroundMethod
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“BLEU is a precision-based metric popular in machine translation that uses a weighted average of variable length phrase matches ( n -grams) against the reference sentence (Papineni et al., 2002).”
Method
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 3×
“Existing datasets vary in size, difﬁculty, and collection methodology; however, as pointed out by Rajpurkar et al. (2016), most suffer from one of two shortcomings: those that are designed explicitly to test comprehension (Richardson et al., 2013) are too small for training data-intensive deep…”
BackgroundMethodInfluential
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2×
“ase matches (n-grams) against the reference sentence (Papineni et al., 2002). CIDEr was designed to correlate better with human judgements of sentence similarity, and uses tf-idf scores over n-grams (Vedantam et al., 2015). As given in Table 4, humans averaged 0.694 F1 on NewsQA. The human EM scores are relatively low…”
MethodInfluential
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,377
Exact solutions to the nonlinear dynamics of learning in deep linear neural networksPreprint
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 1,010
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · cited 5×
“Like Richardson et al. (2013) we want to encourage reasoning in comprehension models.”
BackgroundInfluential

Show all 24 references

Research works printed but not linked: 3

NewsQA: A Machine Comprehension Dataset https://aclanthology.org/W17-2623/ (Trischler et al., RepL4NLP 2017)
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A Machine Comprehension Dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada. Association for Computational Linguistics.
ACL materials are Copyright © 1963–2026 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License.

Sorted by citations

Items 201–300 of 708

Semantic matching based legal information retrieval system for COVID-19 pandemic
Zhu, J., Wu, J., Luo, X., & Liu, J. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09354-x · Cited by 30
Pitfalls of Static Language Modelling.Preprint
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Giménez, M., d’Autume, C. D. M., Ruder, S., Yogatama, D., et al. (2021) · arXiv (Cornell University) · Cited by 32
Document Modeling with External Attention for Sentence Extraction
Narayan, S., Cardenas, R., Papasarantopoulos, N., Cohen, S. B., Lapata, M., Yu, J., & Chang, Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1188 · Cited by 25
Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading ComprehensionPreprint
Nishida, K., Saito, I., Otsuka, A., Asano, H., & Tomita, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10628 · Cited by 27
“ntained in one relevant passage. Currently, SQuADfull is the only large-scale MRS dataset that can be used to bothtrain and evaluate extractive RC models in the same domainsand question styles.NewsQA [45] canbeused as the training data for extractive RC models, while its questions are too dependent on the correspondin…”
Background
Learning Typed Entailment Graphs with Global Soft Constraints
Hosseini, M. J., Chambers, N., Reddy, S., Holt, X., Cohen, S. B., Johnson, M., & Steedman, M. J. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00250 · Cited by 26
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
Li, C., & Choi, J. D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.505 · Cited by 25
Graph Neural Networks for Natural Language Processing: A Survey
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000096 · Cited by 417
Dual Multi-head Co-attention for Multi-choice Reading Comprehension.Preprint
Zhu, P., Zhao, H., & Li, X. (2020) · arXiv (Cornell University) · Cited by 26
Conversations with Documents. An Exploration of Document-Centered AssistancePreprint
ter Hoeve, M., Sim, R., Nouri, E., Fourney, A., de Rijke, M., & White, R. W. (2020) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2002.00747 · Cited by 26
“A lot of progress has been made in the area, driven by the successful application of deep learning architectures and the increase of large scale datasets [e.g., 10, 15, 17, 19, 26– 28, 32, 39, 40].”
Background
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01352 · Cited by 26
“, 2018) ✓ 32,747 newsqa NewsQA (Trischler et al., 2017) ✓ 74,160 pubmed qa PubMedQA (Jin et al.”
ElimiNet: A Model for Eliminating Options for Reading Comprehension with Multiple Choice Questions
Parikh, S., Sai, A. B., Nema, P., & Khapra, M. M. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/594 · Cited by 25
“For example, SQuAD [Ra-jpurkar et al. , 2016], TriviaQA [Joshi et al. , 2017], NewsQA [Trischler et al. , 2016], MS MARCO [Nguyen et al. , 2016], NarrativeQA [Kocisk´y et al. , 2017], etc. contain { passage, question, answer } where the answer matches a span of the passage or it has to be generated.”
Background
What’s The Latest? A Question-driven News Chatbot
Laban, P., Canny, J., & Hearst, M. A. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.43 · Cited by 27
“NewsQA (Trischler et al., 2017) collected a dataset by having a crowdworker read the summary of a news article and ask a follow-up question.”
Knowing More About Questions Can Help: Improving Calibration in Question Answering
Zhang, S., Gong, C., & Choi, E. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.172 · Cited by 43
Reinforced Multi-task Approach for Multi-hop Question Generation
Gupta, D. K., Chauhan, H., Akella, R. T., Ekbal, A., & Bhattacharyya, P. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.249 · Cited by 25
Learning to Ask More: Semi-Autoregressive Sequential Question Generation under Dual-Graph Interaction
Chai, Z., & Wan, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.21 · Cited by 25
“As the reverse task of QA, QG is often performed on existing QA datasets, e.g., SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), etc.”
Method
Transformer-Based Extractive Social Media Question Answering on TweetQA
Butt, S., Ashraf, N., Fahim, H., Sidorov, G., & Gelbukh, A. F. (2021) · Computación y Sistemas · doi:10.13053/cys-25-1-3897 · Cited by 25
Proceedings of the 2nd Workshop on Machine Reading for Question Answering
(2019) · doi:10.18653/v1/d19-58 · Cited by 26
Asking questions like educational experts: automatically generating question-answer pairs on real-world examination data
Qu, F., Jia, X., & Wu, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.202 · Cited by 30
“Tang et al. (2017) regard QA and QG as dual tasks and explicitly leverage their probabilistic correlation to guide the training process of both QA and QG. Wang et al. (2017) design a generative machine that encodes the document and generates a question (answer) given an answer (question).”
World Knowledge for Reading Comprehension: Rare Entity Prediction with Hierarchical LSTMs Using External Descriptions
Long, T., Bengio, E., Lowe, R., Cheung, J. C. K., & Precup, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1086 · Cited by 24
“The original dataset was found to have a low ceiling for machine improvement (Chen et al., 2016); thus, alternative datasets have been proposed that consist of more difﬁcult questions (Trischler et al., 2016; Rajpurkar et al., 2016).”
Background
A Nil-Aware Answer Extraction Framework for Question Answering
Kundu, S., & Ng, H. T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1456 · Cited by 21
“We use the NewsQA dataset with nil questions (Trischler et al., 2017) 17) To compute the overall EM and F1 scores, we consider Nil as correct for the questions which do not have any valid answers.”
BackgroundMethodInfluential
What’s Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering
Khot, T., Sabharwal, A., & Clark, P. E. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1281 · Cited by 24
“While several RC datasets (Ra-jpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) can be addressed with single sentence understanding, newer datasets (Weston et al., 2015; Welbl et al., 2018; Khashabi et al., 2018; Yang et al., 2018) speciﬁcally target multi-hop reasoning.”
Background
CAiRE-COVID: A Question Answering and Multi-Document Summarization System for COVID-19 Research.
Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E. J., & Fung, P. (2020) · Europe PMC (PubMed Central) · Cited by 24
A Turkish Question Answering System Based on Deep Learning Neural Networks
GEMİRTER, C. B., & Goularas, D. (2021) · Journal of Intelligent Systems Theory and Applications · doi:10.38016/jista.815823 · Cited by 25
“is the NewsQA data set (Trischler et al., 2016) translated also automatically in Turkish having also a similar structure with the SQuAD data set.”
Method
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
Reddy, R. G., Rui, X., Li, M., Lin, X., Wen, H., Cho, J., Huang, L., Bansal, M., Sil, A., Chang, S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21370 · Cited by 28
MVP: Multi-task Supervised Pre-training for Natural Language Generation
Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.558 · Cited by 24
Multi-Turn Dialogue Reading Comprehension With Pivot Turns and Knowledge
Zhang, Z., Li, J., & Zhao, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3058616 · Cited by 24
“Although the dialogue comprehension shares a similar form with the standard reading comprehension tasks [31], [38], [55], [58], [59], understanding multi-turn dialogues is much more complex for two key challenges.”
Background
A Comparative Study of Transformer-Based Language Models on Extractive Question AnsweringPreprint
Pearce, K., Zhan, T., Komanduri, A., & Zhan, J. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03142 · Cited by 53
“[7] 2) SQuAD 2.0 : One major ﬂaw in the SQuAD dataset is the focus on questions that are guaranteed to have an answer within the context, making models use context and type-matching heuristics.”
MethodInfluential
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
Jia, R., Lewis, M., & Zettlemoyer, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.59 · Cited by 29
Ensemble learning-based approach for improving generalization capability of machine reading comprehension systems
Baradaran, R., & Amirkhani, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.095 · Cited by 25
NewsPod: Automatic and Interactive News Podcasts
Laban, P., Ye, E., Korlakunta, S., Canny, J., & Hearst, M. A. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511147 · Cited by 26
Sentiment enhanced answer generation and information fusing for product-related question answering
Du, Y., Jin, X., Yan, R., & Yan, J. (2023) · Information Sciences · doi:10.1016/j.ins.2023.01.098 · Cited by 23
Learning When Not to Answer: a Ternary Reward Structure for Reinforcement Learning Based Question Answering
Godin, F., Kumar, A., & Mittal, A. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-2016 · Cited by 23
“Recently, popular textual QA datasets have been extended with not-answerable questions (Trischler et al., 2017; Rajpurkar et al., 2018).”
Background
Self-Supervised Test-Time Learning for Reading Comprehension
Banerjee, P., Gokhale, T., & Baral, C. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.95 · Cited by 30
Quizbowl: The Case for Incremental Question AnsweringPreprint
Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 47
“More recent datasets following this framework include WikiQA (Yang et al., 2015), SQuAD (Rajpurkar et al., 2016), WikiReading (Hewlett et al., 2016), ms marco (Nguyen et al., 2016), and NewsQA (Trischler et al., 2017).”
Method
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase ClustersPreprint
Abujabal, A., Roy, R. S., Yahya, M., & Weikum, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09528 · Cited by 33
“Reading comprehension (RC) is a recently introduced task, where the goal is to answer a question from a given textual paragraph (Kociský et al., 2017; Lai et al., 2017; Rajpurkar et al., 2016; Trischler et al., 2017; Yang et al., 2015).”
Background
Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation ModelPreprint
Hsu, T.-Y., Liu, C.-L., & Lee, H.-Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09587 · Cited by 22
“…al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018) have nourished the development of numerous powerful and diverse RC models (Seo et al., 2016; Hu et al., 2018; Wang et al., 2017; Clark and Gardner, 2018; Huang et al., 2017).”
Background
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
(2021) · doi:10.18653/v1/2021.acl-srw · Cited by 22
Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance
Su, D., Yu, T., & Fung, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.275 · Cited by 26
MGRC: An End-to-End Multigranularity Reading Comprehension Model for Question Answering
Liu, Q., Geng, X., Huang, H., Qin, T., Lü, J., & Jiang, D. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3107029 · Cited by 23
“2) NewsQA [30]: It is an MRC dataset built on CNN news articles.”
RoR: Read-over-Read for Long Document Machine Reading Comprehension
Zhao, J., Bao, J., Wang, Y., Zhou, Y., Wu, Y., He, X., & Zhou, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.160 · Cited by 22
Block-Skim: Efficient Question Answering for Transformer
Guan, Y., Li, Z., Lin, Z., Zhu, Y., Leng, J., & Guo, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21316 · Cited by 27
A Deep Reinforcement Learning Based Multi-Step Coarse to Fine Question Answering (MSCQA) System
Wang, Y., & Jin, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017224 · Cited by 21
“During the past several years, lots of progresses have been made on building deep learning based QA models to answer questions from large scale datasets including unstructured documents (Hermann et al. 2015; Hill et al. 2015; Onishi et al. 2016; Trischler et al. 2016; Nguyen et al. 2016).”
Background
NeurQuRI: Neural Question Requirement Inspector for Answerability Prediction in Machine Reading Comprehension
Back, S., Chinthakindi, S. C., Kedia, A., Lee, H., & Choo, J. (2020) · International Conference on Learning Representations · Cited by 20
FQuAD: French Question Answering DatasetPreprint
d’Hoffschmidt, M., Belblidia, W., Brendlé, T., Heinrich, Q., & Vidal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06071 · Cited by 21
“0 [5] En 150k NewsQA [9] En 100k+ CoQA [6] En 127k+ QuAC [10] En 98k+ HotpotQA [11] En 113k+ SQuAD-es Es 87595 FQuAD Fr 25k+ Table 1: Benchmark of existing Reading Compre-hension datasets, including FQuAD.”
Method
Contextualized Representations Using Textual Encyclopedic KnowledgePreprint
Joshi, M., Lee, K., Luan, Y., & Toutanova, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12006 · Cited by 31
Automatic Question Answering for Medical MCQs: Can It Go Further than Information Retrieval?
Ha, L. A., & Yaneva, V. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_049 · Cited by 20
“This has driven the development of large-scale question-answering data sets such as SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), WikiMovies benchmark (Chen et al., 2017), Triv-iaQA (Joshi et al., 2017) (to name a few), as well as the organisation of workshops such as the Machine Reading for Questio…”
Method
Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
Hardalov, M., Koychev, I. K., & Nakov, P. (2019) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-056-4_053 · Cited by 24
“The growing interest in machine reading comprehension (MRC) has led to the release of various datasets for both extractive (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019) and nonextractive (Richardson et al.”
BackgroundInfluential
FFCI: A Framework for Interpretable Automatic Evaluation of Summarization
Koto, F., Baldwin, T., & Lau, J. H. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13167 · Cited by 25
Frame-based Neural Network for Machine Reading Comprehension
Guo, S., Guan, Y., Tan, H., Li, R., & Li, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106889 · Cited by 20
“This is different from relatively easy extractive MRC datasets such as SQuAD [21] and NewsQA [22], which require a model to extract an answer span to a question from reference passage.”
Question Answering Survey: Directions, Challenges, Datasets, Evaluation MatricesPreprint
Pandya, H. A., & Bhatt, B. (2021) · Preprints.org · doi:10.20944/preprints202112.0136.v1 · Cited by 20
Automatically Generating Cause-and-Effect Questions from Passages
Stasaski, K., Rathod, M., Tu, T., Xiao, Y., & Hearst, M. A. (2021) · Workshop on Innovative Use of NLP for Building Educational Applications · Cited by 19
ReasonBERT: Pre-trained to Reason with Distant Supervision
Deng, X., Su, Y., Lees, A., Wu, Y., Yu, C., & Sun, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.494 · Cited by 27
“, 2016), NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al.”
Method
Cross-Domain Slot Filling as Machine Reading Comprehension: A New Perspective
Liu, J., Yu, M., Chen, Y., & Xu, J. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3140559 · Cited by 19
PersianQuAD: The Native Question Answering Dataset for the Persian Language
Kazemi, A., Mozafari, J., & Nematbakhsh, M. A. (2022) · IEEE Access · doi:10.1109/access.2022.3157289 · Cited by 25
“NewsQA [13] is a QA dataset based on the CNN news articles, with over 100,000 question-answer pairs.”
BackgroundMethod
STARC: Structured Annotations for Reading Comprehension
Berzak, Y., Malmaud, J., & Lévy, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.507 · Cited by 22
Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?
Tang, Y., Ng, H. T., & Tung, A. K. H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.283 · Cited by 32
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 22
“Answering Styles An answer can be (i) chosen from a text span of the given document ( answer extraction ) (NewsQA (Trischler et al., 2017)), (ii) chosen from a candidate set of answers ( multiple choice ) (MCTest (Richardson et al., 2013)), or (iii) generated as a free-form text ( description )…”
Method
Single-dataset Experts for Multi-dataset Question Answering
Friedman, D., Dodge, B., & Chen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.495 · Cited by 22
Comparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18
“6 – NewsQA (Trischler et al., 2017) 76,568 4,343 4,293 EM 57.”
Method
PathReasoner: Explainable reasoning paths for commonsense question answering
Zhan, X., Huang, Y., Dong, X. M., Cao, Q., & Liang, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107612 · Cited by 21
Improving Spoken Question Answering Using Contextualized Word Representation
Su, D., & Fung, P. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053979 · Cited by 16
“The detailed descriptions for each of them can be found in [14, 15, 16, 17, 18].”
Background
Diversify Question Generation with Continuous Content Selectors and Question Type Modeling
Wang, Z., Rao, S., Zhang, J., Qin, Z., Tian, G., & Wang, Y.-X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.194 · Cited by 16
“We can observe the similar phenomenon that our method performs better with regard to the diversity metrics from the performance on NewsQA in Table 2.”
MethodInfluential
NLQuAD: A Non-Factoid Long Question Answering Data Set
Soleimani, A., Monz, C., & Worring, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.106 · Cited by 17
“NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), NarrativeQA (Koˇcisk´y et al., 2018), and DuoRC (Saha et al., 2018) fall into this category and their documents are extracted from news articles, stories, and movie plots, respectively.”
PQuAD: A Persian question answering dataset
Darvishi, K., Shahbodaghkhan, N., Abbasiantaeb, Z., & Momtazi, S. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101486 · Cited by 22
“NewsQA (Trischler et al., 2017) is a large-scale dataset which includes 119,633 questions gathered by human workers from 12,744 CNN’s news articles.”
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 28
QAmeleon : Multilingual QA with Only 5 Examples
Agrawal, P., Alberti, C., Huot, F., Maynez, J., Ma, J., Ruder, S., Ganchev, K., Das, D., & Lapata, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00625 · Cited by 31
“, 2020), and the availability of high-quality benchmarks (Rajpurkar et al., 2016; Trischler et al., 2017; Kwiatkowski et al., 2019).”
Background
Conversational Question Answering in Low Resource Scenarios: A Dataset and Case Study for Basque.
Otegi, A., González-Agirre, A., Campos, J. A., Soroa, A., & Agirre, E. (2020) · Language Resources and Evaluation · Cited by 18
Dynamic Sampling Strategies for Multi-Task Reading Comprehension
Gottumukkala, A., Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.86 · Cited by 16
“Setup The eight reading comprehension tasks are from the ORB benchmark (Dua et al., 2019b): DROP (Dua et al., 2019a), DuoRC (Saha et al., 2018), NarrativeQA (Koˇcisky et al., 2017), NewsQA (Trischler et al., 2017), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), SQuAD (Rajpurkar et al., 2016 2.0 (Rajpurkar et…”
MethodInfluential
Target-oriented Fine-tuning for Zero-Resource Named Entity Recognition
Zhang, Y., Meng, F., Chen, Y., Xu, J., & Zhou, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.140 · Cited by 16
Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation
Laban, P., Wu, C.-S., Murakhovs’ka, L., Liu, W., & Xiong, C. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.9 · Cited by 20
ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading ComprehensionPreprint
Dua, D., Gottumukkala, A., Talmor, A., Singh, S., & Gardner, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12598 · Cited by 16
“NewsQA (Trischler et al., 2017) dataset focuses on paraphrased questions with predicate-argument structure understanding.”
Background
DQI: Measuring Data Quality in NLPPreprint
Mishra, S. R., Arunkumar, A., Sachdeva, B., Bryan, C., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00816 · Cited by 38
“Answer: 93 Possible answers: United Airlines Flight 93, Flight 93 Here, multiple choices have the correct span of 93 (Trischler et al., 2017).”
Background
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and modelsPreprint
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14709 · Cited by 15
Answering Science Exam Questions Using Query Reformulation with Background Knowledge
Musa, R., Wang, X., Fokoue, A., Mattei, N., Chang, M., Kapanipathi, P., Makni, B., Talamadupula, K., & Witbrock, M. (2019) · Automated Knowledge Base Construction · Cited by 15
Unsupervised Domain Adaptation via Contrastive Adversarial Domain Mixup: A Case Study on COVID-19
Zeng, H., Yue, Z., Shang, L., Zhang, Y., & Wang, D. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3354419 · Cited by 17
ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
Wang, B., Yao, T., Zhang, Q., Xu, J., & Wang, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6450 · Cited by 15
“NewsQA (Trischler et al. 2017) is based on CNN/Daily Mail, the answers and questions are generated by different people to solicit exploratory questions that require reasoning.”
Method
Improving question answering performance using knowledge distillation and active learning
Boreshban, Y., Mirbostani, S. M., Ghassem-Sani, G., Mirroshandel, S. A., & Amiriparian, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106137 · Cited by 22
Review and Arrange: Curriculum Learning for Natural Language Understanding
Zhang, L., Mao, Z., Xu, B., Wang, Q., & Zhang, Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3121986 · Cited by 16
Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices
Pandya, H. A., & Bhatt, B. (2021) · Journal of Xidian University · doi:10.37896/jxu15.4/014 · Cited by 42
“For 1000 randomly selected examples of NewsQA and SQuAD, to identify the reasoning category, manual labeling is applied in one of the five categories as 1)Word Matching, 2) Paraphrasing, 3) Inference, 4)Synthesis, 5)Ambiguous/Insufficient.”
Chains-BERT: A High-Performance Semi-Supervised and Contrastive Learning-Based Automatic Question-and-Answering Model for Agricultural Scenarios
Huang, Y., Liu, J., & Lv, C. (2023) · Applied Sciences · doi:10.3390/app13052924 · Cited by 15
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension.Preprint
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · arXiv (Cornell University) · Cited by 15
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches.Preprint
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2021) · arXiv (Cornell University) · Cited by 17
Question Generation With Doubly Adversarial Nets
Bao, J., Gong, Y., Duan, N., Zhou, M., & Zhao, T. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2859777 · Cited by 14
“We conduct experiments on two existing datasets, SQuAD [26] and NewsQA [27], which was originally created for the MRC task.”
Method
Undersensitivity in Neural Reading Comprehension
Welbl, J., Minervini, P., Bartolo, M., Stenetorp, P., & Riedel, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.103 · Cited by 16
An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension
Liu, X., Liu, K., Li, X., Su, J., Ge, Y., Wang, B., & Luo, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/525 · Cited by 15
“(cid:15) N EWS QA [Trischler et al. , 2017].”
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
Kang, M., Han, M., & Hwang, S. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.493 · Cited by 15
Constrained BERT BiLSTM CRF for understanding multi-sentence entity-seeking questions
Contractor, D., Patra, B., Mausam, M., & Singla, P. (2020) · Natural Language Engineering · doi:10.1017/s1351324920000017 · Cited by 15
“Reading comprehension tasks (Trischler et al. 2016; Joshi et al. 2017; Trivedi et al. 2017; Rajpurkar et al. 2018; Yang et al. 2018; Dua et al. 2019) require answers to be generated from unstructured text also only return answers for relatively simple (single-sentence) questions.”
VLSP 2021-ViMRC Challenge: Vietnamese Machine Reading Comprehension
Nguyen, K. V., Tran, S. Q., Nguyen, L. T., Huynh, T. V., Luu, S. T., & Nguyen, N. L.-T. (2022) · VNU Journal of Science Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.340 · Cited by 14
“…al., 2018; Fisch et al., 2019; Zheng et al., 2021) and benchmark corpora (Richardson et al., 2013; Rajpurkar et al., 2016, 2018; Joshi et al., 2017; Trischler et al., 2017; Koˇciský et al., 2018; Lai et al., 2017; Reddy et al., 2019) have attracted a range of researchers from academia and industry.”
BackgroundMethodInfluential
Learning to Generate Task-Specific Adapters from Task Description
Ye, Q., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.82 · Cited by 26
“, 2019) and NewsQA (Trischler et al., 2017), and use these as out-of-domain test examples.”
BackgroundMethodInfluential
Frustratingly Easy Natural Question AnsweringPreprint
Pan, L., Chakravarti, R., Ferritto, A., Glass, M., Gliozzo, A., Roukos, S., Florian, R., & Sil, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05286 · Cited by 16
Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural NetworksPreprint
Hazen, T. J., Dhuliawala, S., & Boies, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02655 · Cited by 20
“Since the release of the SQuAD dataset, there have been a variety of new data sets released covering different use cases and text data sources including NewsQA [10], SearchQA [11], TriviaQA [12], and MS-MARCO [13].”
MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
Xu, C., Pei, J., Wu, H., Liu, Y., & Li, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.330 · Cited by 14
SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning
Zheng, B., Yang, X., Ruan, Y.-P., Ling, Z.-H., Liu, Q., Wei, S., & Zhu, X. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.4 · Cited by 15
Weakly-Supervised Open-Retrieval Conversational Question Answering
Qu, C., Liu, Y., Chen, C., Croft, W. B., Krishna, K., & Iyyer, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_35 · Cited by 13
Improving paragraph-level question generation with extended answer network and uncertainty-aware beam search
Zeng, H., Zhi, Z., Liu, J., & Wei, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.026 · Cited by 14
Asking questions on handwritten document collections
Mathew, M., Gómez, L., Karatzas, D., & Jawahar, C. V. (2021) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-021-00383-3 · Cited by 16
“where an answer is not generated, but returned as a span (a sequence of contiguous text tokens) of the given context [47, 56].”
Background
CodeQA: A Question Answering Dataset for Source Code Comprehension
Liu, C., & Wan, X. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.223 · Cited by 18
FedQAS: Privacy-Aware Machine Reading Comprehension with Federated Learning
Ait‐Mlouk, A., Alawadi, S., Salman Zubair, T. O. O. R., & Hellander, A. (2022) · Applied Sciences · doi:10.3390/app12063130 · Cited by 13
LIQUID: A Framework for List Question Answering Dataset Generation
Lee, S., Kim, H., & Kang, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26529 · Cited by 17
QAScore—An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Ji, T., Lyu, C., Jones, G., Zhou, L., & Graham, Y. (2022) · Entropy · doi:10.3390/e24111514 · Cited by 12
“Although the aforementioned prevailing automatic metrics mentioned above are widely employed for QG evaluation, criticism of n -gram overlap-based metrics’ ability to accurately and comprehensively evaluate the quality has also been highlighted (Yuan et al., 2017).”

Authors: Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K.
ORCID: Trischler, Adam
Wang, Tong
Yuan, Xingdi
Venue: Proceedings of the 2nd Workshop on Representation Learning for NLP
Publisher: Association for Computational Linguistics
Published: 2017
Pages: 191-200
DOI: 10.18653/v1/w17-2623
OpenAlex: W2557764419
Cited by: 706
References: 24
Citations by year: 2026: 18 · 2025: 26 · 2024: 39 · 2023: 96 · 2022: 110 · 2021: 134 · 2020: 138 · 2019: 87 · 2018: 41 · 2017: 16
FWCI: 35.17
Type: conference paper
Language: en
Open access: hybrid
doi.org
Licence: cc-by
Locations: Workshop on Representation Learning for NLP (RepL4NLP) (publishedVersion, cc-by)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: machine reading comprehension, question answering, news articles, exploratory questions, reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

NewsQA: A Machine Comprehension Dataset

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks